Unified Shared Memory Sectiononeapi_afu.json to table 5-7board_hw.tcl in table 5-10(NUMBER_OF_DMA_CHANNELS, DATA_WIDTH, MAX_BURST_SIZE, KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE) and parameters.tclacl_quartus_report.txt in section 5.3.5Extracting oneAPI Binary Informtion usingaocl bineditCommand Unified Shared Memory Sectiononeapi_afu.json to table 5-7board_hw.tcl in table 5-10(NUMBER_OF_DMA_CHANNELS, DATA_WIDTH, MAX_BURST_SIZE, KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE) and parameters.tclacl_quartus_report.txt in section 5.3.5 Extracting oneAPI Binary Informtion usingaocl bineditCommand
Open FPGA Stack (OFS): OFS is an open-source hardware and software framework that reduces the development time for creating your custom platform. Provided within this framework are reference designs targeting different Altera FPGA devices with upstreamed drivers and management software tools.
The reference shells, called FPGA Interface Manager (FIMs), provide an integrated, timing closed design with the most common interfaces for host attach applications. After selecting your starting shell, you can add or subtract interfaces depending on your application requirements. Then leverage the build scripts, RTL, unit tests, Universal Verification Methodology (UVM) environment, software and drivers for this reference shell as a starting point for your own FPGA platform solution.
OFS currently targets Stratix\u00ae 10 and Agilex\u00ae 7 FPGA Device Families. To find out more about Altera FPGAs, visit the Stratix 10 and Agilex 7 pages at Intel.com.
"},{"location":"#how-can-i-start-using-ofs","title":"How Can I Start Using OFS?","text":"If you are interested in a production card that uses OFS for your workload application development or for full deployment, please refer to the OFS Board Catalog.
If you are an FPGA developer, refer to our FPGA Developer Journey Guide: Open FPGA Stack to understand the OFS development flow as well as the reference shells, tools and development boards you can use to gest started. FPGA Developers interested in oneAPI should reference this guide as well.
If you are a software developer, refer to our Software Tab for driver and user space tool development resources.
If you are an application developer, preview our overview video on how OFS can help you develop FPGA-based workloads and review one of the AFU Developer Guides to find the OFS resources available for creating your own application workload.
Beyond the resources we have on this site, you can navigate to the OFS GitHub repos by clicking the GitHub repository icon at the top left corner of this site page.
"},{"location":"#what-fim-reference-designs-are-available","title":"What FIM Reference Designs Are Available?","text":"
Below summarizes the five current reference designs (aka FIMs) you can choose from:
OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xR-tile, F-tile)
Key Feature Description Target OPN AGIB027R29A1E2VR3 PCIe R-tile PCIe* Gen5x16, 2xGen5x8, Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory * Four Fabric DDR4 channels, x64 (no ECC), 2666 MHz, 8GB Ethernet 2x4x25GbE, 2x200GbE, 2x400GbE Hard Processor System Not enabled Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xF-tile)
Key Feature Description Target OPN AGFB027R24C2E2VR2 PCIe F-tile PCIe* Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 3 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 2400 MHz, 1GB each* Two Fabric DDR4 banks, x64 (no ECC), 2400 MHz, 8GB Ethernet 2x4x25GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
OFS FIM Targeting Agilex\u00ae7 PCIe Attach (P-tile, E-tile)
Key Feature Description Target OPN AGFB014R24A2E2V PCIe P-tile PCIe* Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 5 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each* Four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB Ethernet 2x4x25GbE, 2x4x10GbE or 2x100GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration)* Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Intel\u00ae FPGA SmartNIC N6001-PL
Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
OFS FIM Features Targeting Agilex\u00ae 7 SoC Attach
Key Feature Description Device OPN AGFC023R25A2E2VR0 PCIe P-tile PCIe* Gen4x16 to the HostP-tile PCIe* Gen4x16 to the SoC (IceLake-D) Virtualization Host: 2 physical functions SoC: 1 physical function and 3 Virtual functions Memory Four Fabric DDR4 banks, x40 (optional ECC, be configured as x32 and ECC x8 ), 1200 MHz, 4GB Ethernet 2x4x25GbE Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) * Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported Software Support * Linux DFL drivers targeting OFS FIMs * OPAE Software Development Kit * OPAE Tools Target Board Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL
Note: Source code for BMC RTL/Firmware that works with this reference FIM can be obtained by contacting your Altera Sales Representative.
Click here for the OFS Collateral for Agilex\u00ae SoC Attach Reference FIM documentation collection.
OFS FIM Targeting Stratix\u00ae 10 FPGA PCIe Attach
Key Feature Description Device OPN 1SX280HN2F43E2VG Ethernet Configuration 1x10GbE example with 2x100GbE capability PCIe Gen3x16 EMIF Up to four DDR channels PF/VF 1 PF/3 VFs Management FPGA Management Engine (FME) with FIM management registers Interface Arm\u00ae AMBA\u00ae4 AXI Interface HLD support oneAPI Software Kernel code upstreamed to Linux.org Target Board Intel\u00ae FPGA Programmable Acceleration Card D5005
Click here for the OFS Collateral for Stratix\u00ae 10 FPGA PCIe Attach Reference FIM) documentation.
To find information on the latest releases, go to the Discussions Tab in the OFS GitHub repository.
"},{"location":"#open-fpga-stack-repositories","title":"Open FPGA Stack Repositories","text":"Accessing OFS ingredients to use within the development framework is easy. The github.com/OFS site provides all the hardware and software repositories in one location.
Development Focus Repository Folder Description Hardware ofs-agx7-pcie-attach Provides RTL, unit tests, and build scripts to create an Agilex\u00ae 7 FIM and is leveraged as a starting point for a custom PCIe Attach design. The provided reference FIM can target the following boards:\u00a0\u00a0\u00a0\u00a0\u2022 Intel\u00ae FPGA SmartNIC N6001-PL Platform \u00a0\u00a0\u00a0\u00a0\u2022 Agilex 7 FPGA F-Series Development Kit (2x F-Tile)\u00a0\u00a0\u00a0\u00a0\u2022 Agilex 7 FPGA I-Series Development Kit (2 x R-Tile, 1 x F-Tile) Hardware ofs-f2000x-pl Provides RTL, unit tests, and build scripts to create Agilex\u00ae FIM and is leveraged as a starting point for a custom SoC Attach design. The reference FIM targets an Intel\u00ae FPGA IPU F2000X-PL Platform. Hardware ofs-d5005 Provides RTL, unit tests, and build scripts to create Stratix 10\u00ae FIM and is leveraged as a starting point for a custom PCIe Attach design. The reference FIM targets an Intel\u00ae FPGA PAC D5005 development board. Hardware oneapi-asp Contains the files to generate the support package that works with the reference shells and allows you to use OneAPI. This is an optional repository for developers interested in OneAPI Hardware ofs-fim-common Provides RTL components that are shared among all new platforms that are introduced in OFS. This folder is a subumodule in each platform repository folder. Hardware examples-afu Provides simple Accelerator Functional Unit (AFU) examples you can use as a template for starting your own workload design. Hardware ofs-platform-afu-bbb Contains the hardware code to build a standard interface between the FIM and your workload. Software linux-dfl This repository is a mirror of the linux.org Git site and contains the most up-to-date drivers that are being developed and upstreamed for OFS platforms. Software meta-ofs This repository provides the Linux\u00ae DFL kernel and the OPAE SDK for the Yocto\u00ae Project. Software opae-sdk Contains the ingredients to build the OFS Open Programmable Acceleration Engine (OPAE) Software Development Kit which provides APIs and userspace tools for OFS FPGA management. Software opae-sim This repository is used to build the AFU Hardware/Software Co-Simulation Environment workload developers can use to ensure their AFU can work with the OFS software stack. Software linux-dfl-backport A place for finding and leveraging out-of-tree backported drivers for older OS versions . Software opae-legacy Supports OFS platforms built on the legacy version of OPAE software. Not used in current OFS designs Documentation ofs.github.io Contains the hardware and software collateral that surfaces on the OFS website: https://ofs.github.io "},{"location":"d5005/adp_board_installation_guidelines/","title":"Board Installation Guidelines: Intel\u00ae FPGA SmartNIC N6000/1-PL, Intel\u00ae FPGA PAC D5005","text":"Last updated: November 19, 2024
"},{"location":"d5005/adp_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"d5005/adp_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup - the Intel\u00ae FPGA SmartNIC N6000/1-PL and the Intel\u00ae FPGA PAC D5005. After reading the document a user shall be able to:
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to all three platforms.
"},{"location":"d5005/adp_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported ADP platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"d5005/adp_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"d5005/adp_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"d5005/adp_board_installation_guidelines/#table-2-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 2: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platforms you have if you are unsure. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers from sections, whose installation is covered in the [Software Installation Guide: OFS for PCIe Attach FPGAs].
fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table provides a picture reference for the hardware components discussed in the rest of the document.
"},{"location":"d5005/adp_board_installation_guidelines/#table-3-hardware-bkc","title":"Table 3: Hardware BKC","text":"Component Image Intel\u00ae FPGA SmartNIC N6001-PL (SKU2) Supermicro Server SYS-220HE Intel FPGA Download Cable II (Only Required for manual flashing) 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing)In addition to the above, all OFS ADP platforms require an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector. This cable will differ between server vendors - review the pinout of the power connector on the Intel\u00ae FPGA Programmable Acceleration Card D5005 Data Sheet or Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based ADP.
"},{"location":"d5005/adp_board_installation_guidelines/#20-initial-server-setup","title":"2.0 Initial Server Setup","text":""},{"location":"d5005/adp_board_installation_guidelines/#21-server-information-for-intel-fpga-smartnic-n60001-pl","title":"2.1 Server Information for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Both the server BIOS and BMC need to match the versions listed below in Table 4: Supermicro Server BMC BKC. These updates only apply for this specific Best Known Configuration (BKC) - other server manufacturers may require different BIOS updates. Please consult your server's user guide and release notes for update information.
"},{"location":"d5005/adp_board_installation_guidelines/#table-4-supermicro-server-bmc-bkc","title":"Table 4: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4)Information about the server\u2019s currently loaded firmware can be found on the BMC web portal dashboard. Accessing this page requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d. During boot the BMC\u2019s login IP will be presented on the screen.
Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case. It is recommended the user change their BMC\u2019s default username as soon as they are able.
After logging in you should be able to review information about the BMC and BIOS by referring to the System box, visible upon initial loading of the page. Double check that the values match those in Table 4. If they do not, you may download the appropriate versions from the SuperMicro product page by selecting the BIOS option and downloading the most recent \u201cBundled Software File Name\u201d. Follow the BMC and BIOS update instructions included in the SuperMicro manuals page in the document X12/H12 BMC Manual in Appendix A.2 Updating Firmware Using BMC Web GUI.
If using a different server model, refer to that server\u2019s user guide for instructions on remote system management. Ensure that any system you end up using meets all the following requirements:
Refer to sections 2.1-2.3 of the Intel Acceleration Stack Quick Start Guide: Intel FPGA Programmable Acceleration Card D5005 for a complete overview of the physical installation process and ESD precautions for the D5005 platform.
Ensure that the system meets all the following requirements before proceeding to install the Intel\u00ae FPGA PAC D5005 into a server.
Detailed mechanical for information can be found on the D5005 Data Sheet and in section 4.0 Mechanical Information of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837).
"},{"location":"d5005/adp_board_installation_guidelines/#30-server-settings","title":"3.0 Server Settings","text":""},{"location":"d5005/adp_board_installation_guidelines/#31-bios-settings","title":"3.1 BIOS Settings","text":"You must enable Intel VT-x/VT-d technologies for the PCIe slot housing your ADP. The following steps are known to work on a SuperMicro SYS-220HE server platform.
To enter the Supermicro server\u2019s BIOS setup page, reboot, and press \\<Delete> when prompted. You can browse the tabs / options with a combination of arrow keys along with \\<Escape> and \\<Enter>.
Navigate right to the Advanced tab, then select the following menu options: Chipset Configuration -> North Bridge -> IIO Configuration -> Intel VT for Directed I/O (VT-d).
If not already, enable the option Intel VT for Directed I/O (VT-d).
The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
Please refer to sections 8.1 and 8.2 of the Intel FPGA Programmable Acceleration Card D5005 Data Sheet or section 6.0 of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) for guidance on cooling specifications that must be met when using these platforms. Failure to adhere to these guidelines may result in thermal runaway and/or performance degradation.
"},{"location":"d5005/adp_board_installation_guidelines/#40-board-installation-procedure","title":"4.0 Board Installation Procedure","text":""},{"location":"d5005/adp_board_installation_guidelines/#41-pcie-slot-mappings-for-intel-fpga-smartnic-n60001-pl","title":"4.1 PCIe Slot Mappings for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"The Intel N6000/1-PL FPGA SmartNIC Platforms are officially verified in the upper middle PCIe x16 slot (Slot 3). If using a different slot, refer to the information in Table 5 PCIe Slot Mapping for which port settings to change in server BIOS.
"},{"location":"d5005/adp_board_installation_guidelines/#table-5-pcie-slot-mapping","title":"Table 5: PCIe Slot Mapping","text":"CPU Number Port Number (in BIOS) PCIe Slot CPU1 Port 2 5 and 6 CPU1 Port 4 7 and 8 CPU2 Port 2 1 and 2 CPU2 Port 4 3 and 4"},{"location":"d5005/adp_board_installation_guidelines/#42-installation-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.2 Installation Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe installation of an ADP platform into a supported server. While an Intel\u00ae FPGA SmartNIC N6001-PL is shown in the images below, this procedure applies to all three platforms.
Do not bend the card while inserting into a slot. Do not apply much pressure in regions 2 or 3 while inserting.
"},{"location":"d5005/adp_board_installation_guidelines/#43-removal-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.3 Removal Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe removal of the platforms from a supported server.
Do not bend the card while removing it from the slot.
"},{"location":"d5005/sw_install_pcie_attach/","title":"Software Installation Guide: Open FPGA Stack for PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"d5005/sw_install_pcie_attach/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the PCIe Attach software stack on the host. This document will not cover the process of board installation or platform bring-up. After reviewing this document, a user shall be able to:
The information in this document is intended for customers evaluating a PCIe Attach shell. The PCIe Attach shell design is supported on a number of board offerings, including the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile), Intel\u00ae FPGA SmartNIC N6000/1-PL, and Intel\u00ae FPGA PAC D5005.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"d5005/sw_install_pcie_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"d5005/sw_install_pcie_attach/#table-2-software-and-component-version-summary-for-ofs-pcie-attach","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach","text":"The OFS PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are where the source code resides for each of the components discussed in this document.
Component Version Download Link Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 link OPAE SDK 2.13.0-3 2.13.0-3 Linux DFL intel-1.11.0-2 intel-1.11.0-2"},{"location":"d5005/sw_install_pcie_attach/#table-3-release-pages-for-each-pcie-attach-platform","title":"Table 3: Release Page(s) for each PCIe Attach Platform","text":"This is a comprehensive list of the platform(s) whose software build and installation steps are covered in this document.
Platform Release Page Link Stratix\u00ae 10 FPGA https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Intel\u00ae FPGA SmartNIC N6001-PL https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1"},{"location":"d5005/sw_install_pcie_attach/#12-server-requirements","title":"1.2 Server Requirements","text":""},{"location":"d5005/sw_install_pcie_attach/#121-host-bios","title":"1.2.1 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"d5005/sw_install_pcie_attach/#122-host-server-kernel-and-grub-configuration","title":"1.2.2 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack, on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"d5005/sw_install_pcie_attach/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions, and currently only supported the PCIe Attach release. Its usage is detailed in the relevant Quick Start Demonstration Guideline for your platform and will not be covered here.
"},{"location":"d5005/sw_install_pcie_attach/#31-ofs-dfl-backport-kernel-driver-installation-environment-setup","title":"3.1 OFS DFL Backport Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL Backport GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.2.2 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 3.3 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
It is recommended you lock your Red Hat release version to 8.10 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\nInstall the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n\nsudo dnf install kernel-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-headers-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-devel-4.18.0-553.5.1.el8_10.x86_64\nNow you have the choice to either follow the steps in section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source or 3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages.
"},{"location":"d5005/sw_install_pcie_attach/#32-building-and-installing-the-ofs-dfl-backport-kernel-drivers-from-source","title":"3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Select the 4.18.0-553.5.1.el8_10 kernel as your next boot target on the build machine, then reboot.
# For multiple boots (until overwritten), use the following\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\n# Or select the kernel you want during boot time\nsudo reboot\n1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nintel-1.11.0-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n3. Build the kernel.
```bash\ncd /home/OFS/linux-dfl-backport\nmake && make rpm\n```\n4. Install the newly compiled RPM package and reboot.
```bash\nsudo rpm -i intel-fpga-dfl-dkms-1.11.0-2.2024.07.25.g276007e.noarch.rpm\n\nsudo reboot\n```\n5. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/4.18.0-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\nIf an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n6. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n7. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n8. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n9. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-4.18.0-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\nA list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"d5005/sw_install_pcie_attach/#33-installing-the-ofs-dfl-backport-kernel-drivers-from-pre-built-packages","title":"3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page under the Artifacts tab. The name will resemble kernel-*.tar.gz. For the backport repository you can also choose to download packages under the GitHub action Build dkms packages, although your version will differ from the release version of the software.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\ntar xf kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\nsudo rpm -i kernel-4.18.0_dfl_2024.06.14_276007e/intel-fpga-dfl-dkms-1.11.0-2.2024.06.14.g276007e.noarch.rpm\n\n# Set this kernel as your new default boot target\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\nsudo reboot\nContinue from step 5 of Section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source.
"},{"location":"d5005/sw_install_pcie_attach/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 4.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"d5005/sw_install_pcie_attach/#41-opae-sdk-installation-environment-setup","title":"4.1 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"d5005/sw_install_pcie_attach/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package.Remove any currently installed OPAE packages.
sudo dnf remove opae*\nInitialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\nVerify that the correct tag/branch have been checkout out.
git describe --tags\n2.13.0-3\nSet up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\nThe following packages will be built in the same directory as create:
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\nCheck that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\nYou can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.13.0-3.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.13.0-3.x86_64-*.tar.gz\nFor a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"d5005/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\nHere is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\nIn main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\nMapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\nThe below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\nThe below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\nIn main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\nThe host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\nWhen the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\nA Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\nTo run the host application, you will need to:
See the associated AFU Developer Guide for details.
"},{"location":"d5005/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"d5005/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"d5005/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"d5005/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":"When using ASE in the application development cycle, consider the following limitations:
AFU development using the ASE includes the following four stages:
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"d5005/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\nThis directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\nBefore configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\nafu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.generate_ase_environment.py: This script instantiates your simulated platform configuration.The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
synopsys_sim.setup and vcs_run.tcl in the configuration directory.vsim_run.tcl in the configuration directory.The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"d5005/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\nThe AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code.cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"d5005/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"d5005/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\nASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\nCreate a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\nOptionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> .. /usr. $ sudo make install ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\nChange directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n... The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\nYou can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c. # Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\nNote: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\nUpon completion, the simulation generates the following files:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"d5005/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
ASE_LOG to 0.Two log levels are supported in ASE, controlled by env(ASE_LOG)
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\nThe following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If usingASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"d5005/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"d5005/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI XA Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\nThe OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"d5005/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support:<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.To interface the PIM with an AFU:
For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\nofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"d5005/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"d5005/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\nThe Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\nUsing the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\nThe following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\nLocal memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"d5005/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\nUsing the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\nThe following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\nThe High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"d5005/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\nIn digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\nInstantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\nThe AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"d5005/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\nThe AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\nThe Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\nThe MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"d5005/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"d5005/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\nThe AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.svThe HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\nThe host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"d5005/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"d5005/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\nThe ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\nThe Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\nThe source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\nBelow is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
The Open FPGA Stack (OFS) Docker image has two main personas:
A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"d5005/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"d5005/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"d5005/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"d5005/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"d5005/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\nAdd the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\nInstall the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\nStart the Docker daemon:
sudo systemctl start docker\nEnable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\nVerify that Docker is installed and running:
sudo systemctl status docker\nYou should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\nYou will need to log out and back in for the changes to take effect.
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\nThe Docker installation steps for Ubuntu are the following:
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\nInstall packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\nAdd Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\nThe following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\nUpdate the package manager index again:
sudo apt-get update\nInstall Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\nStart the Docker daemon:
sudo systemctl start docker\nEnable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\nVerify that Docker is installed and running:
sudo systemctl status docker\nYou should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\nYou will need to log out and back in for the changes to take effect.
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\nThe Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"d5005/ug_docker/#build-the-image","title":"Build the image","text":"You can download the Dockefile from OFS GitHub Docker.
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\nSave the file
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\nNote: Never remove --no-cache this could cause issues with your environmental variables inside of the container
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\nDocker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\nFor more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\nInside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\nThe docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nTip: you can change myOFS with any other value. The value is the given name of the container.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\nNow the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\nYour Container ID is bdc1289fb081.
"},{"location":"d5005/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\nThe container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\nEverything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
The OFS Docker image already includes the evaluation script.
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\nThe Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\nIf you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\nall the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"d5005/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"d5005/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nExample, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\nTip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\nNow, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n\u200b Your Container ID is 25b41eb4d232.
"},{"location":"d5005/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\nThe container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\nEverything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
The OFS Docker image already includes the eval script.
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\nNow you can control and compile the design using the interactive or de-attach mode.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\nThis command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\nIn this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"d5005/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":"lsmod | grep kvm\nIf the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\nVirtual Machine Manager (also known as libvirt) can be installed by following the below steps:
Update your system package index by running the following command:
sudo dnf update\nsudo apt update\nInstall the libvirt package and any required dependencies by running the following command:
sudo dnf install @virtualization\nsudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\nStart the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\nsudo dnf install virt-manager\nsudo usermod -a -G libvirt USERNAME\nvirt-manager as the non-root user.Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\nYou will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\nreboot\nAfter completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"d5005/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
virt-manager GUI by running the following command:sudo virt-manager&\nCreate a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
In the virt-manager window, click the \"New virtual machine\" button.
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
Click on the \"+\" icon (Bottom left) to create the storage pool.
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"d5005/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n\u200b
"},{"location":"d5005/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3 Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\nCheck that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\nThe following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"d5005/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbbEnsure you add the desired VF in your PCIE devices list.
For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nPass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\nEdit the XML file for your machine and include the following
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\nInside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\nEnsure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\nNote: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
Save the changes \"Apply\"
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\nEnsure your devices are enumerated properly.
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n2. Deployment Mode:
B1:00.5\nUnder the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n2. Deployment Mode:
177:00.0\nClick on \"Begin Installation.\" and follow the wizard installation of the OS.
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
Under your virtual machine, configure your VM proxy:
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\nUse the Virtual function (Not supported at management mode)
Ensure the DFL kernel drivers is install in your VM system
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\nVerify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\nTest the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\nAfter the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/sw_install_soc_attach/","title":"Software Installation Guide: Intel Agilex 7 SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"f2000x/sw_install_soc_attach/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the OFS SoC Attach software stack on the host and platform. After reviewing this document, a user shall be able to:
This document does not cover first time setup of the IPU Platform F2000X-PL platform.
"},{"location":"f2000x/sw_install_soc_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating an SoC Attach release. This document will cover key topics related to initial bring up of the IPU Platform F2000X-PL software stack, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"f2000x/sw_install_soc_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"f2000x/sw_install_soc_attach/#table-2-software-component-version-summary-for-soc-attach","title":"Table 2: Software Component Version Summary for SoC Attach","text":"Name Location META-OFS https://github.com/OFS/meta-ofs, tag: ofs-2024.1-2 Host Operating System Ubuntu 22.04 LTS Host OPAE SDK 2.12.0-5 SoC OS meta-intel-ese Reference Distro 1.0-ESE (kirkstone) SoC Kernel Version 6.1.78-dfl SoC OPAE SDK 2.12.0-5 SoC Linux DFL ofs-2024.1-6.1-2Not all components shown in Table 2 will have an update available upon release. The OPAE SDK and Linux DFL software stacks are incorporated into a Yocto image and do not need to be downloaded separately.
"},{"location":"f2000x/sw_install_soc_attach/#20-updating-the-ipu-platform-f2000x-pl","title":"2.0 Updating the IPU Platform F2000X-PL","text":"Every IPU Platform F2000X-PL ships with pre-programmed firmware for the FPGA user1, user2, and factory images, the Cyclone 10 BMC RTL and FW, the SoC NVMe, and the SoC BIOS. In this software installation guide, we will only be focusing on the building and loading of a new SoC NVMe image. Board setup and configuration for the IPU Platform F2000X-PL is discussed in the Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"f2000x/sw_install_soc_attach/#30-compiling-a-custom-yocto-soc-image","title":"3.0 Compiling a Custom Yocto SoC Image","text":"Current Yocto image architecture for SoC Attach is based off of the IOTG Yoct-based ESE BSP, with the addition of the Linux DFL kernel including the latest DFL drivers for FPGA devices alongside the OPAE SDK user space software. The image targets x86_64 SoC FPGA devices but should boot on most UEFI-based machines. The source code and documentation for this image is hosted on the meta-ofs repository.
Build requirements exceed 100 GiB of disk space, depending on which image is built. As a reference point, on a system with two Intel(R) Xeon(R) E5-2699 v4 for a total of 44 CPU cores, the initial, non-incremental build takes less than an hour of wall time.
The repo tool is needed to clone the various Yocto layer repositories used in this example.
Note: If you are behind a firewall that prevents you from accessing references using the git:// protocol, you can use the following to redirect Git to use the corresponding https repositories for Yocto only: git config --global url.https://git.yoctoproject.org/.insteadOf git://git.yoctoproject.org/.
To compile the image as-is, use the following steps (as provided in meta-ofs):
Create and initialize the source directory.
mkdir ofs-yocto && cd ofs-yocto\ngit clone --recurse-submodules --shallow-submodules https://github.com/OFS/meta-ofs\ncd meta-ofs\ngit checkout tags/ofs-2024.1-2\nBuild packages and create an image.
cd examples/iotg-yocto-ese\nTEMPLATECONF=$PWD/conf source openembedded-core/oe-init-build-env build\nbitbake mc:x86-2022-minimal:core-image-full-cmdline\nThe resulting GPT disk image is available in uncompressed (.wic) and compressed form (.wic.gz) in meta-ofs/examples/iotg-yocto-ese/build/tmp-x86-2021-minimal-glibc/deploy/images/intel-corei7-64/. With no changes the uncompressed image size is ~21 GB.
The image type core-image-full-cmdline includes the familiar GNU core utilities, as opposed to core-image-minimal which uses BusyBox instead.
The example build configuration files under build/conf/ are symlinked from examples/iotg-yocto-ese/. To customize the image, start by modifying local.conf and bblayers.conf.
The uncompressed Yocto image can be loaded onto a flash drive as discussed in section 1.5.5 Creating a Bootable USB Flash Drive for the SoC and written to NVMe as the default boot target for the SoC as demonstrated in section 2.1 Updating the F2000X-PL ICX-D SoC NVMe.
"},{"location":"f2000x/sw_install_soc_attach/#40-verifying-the-icx-d-soc-software-stack","title":"4.0 Verifying the ICX-D SoC Software Stack","text":"The reference SoC Attach FIM and unaltered FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Full supported functionality of the HEMs is documented in this platform's associated Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs and will not be covered here.
"},{"location":"f2000x/sw_install_soc_attach/#50-setting-up-the-host","title":"5.0 Setting up the Host","text":"External SoC Attach supports testing Host to FPGA latency, MMIO latency, and MMIO bandwidth. This testing is accomplished using the utility host_exerciser on the host, which is included as a part of OPAE. This section will cover the installation and verification flow for a host interacting with the SoC Attach workload.
Review Section 1.2 Server Requirements of the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL for a list of changes required on the host to support an IPU Platform F2000X-PL and for a list of supported OS distributions. Installation will require an active internet connection to resolve dependencies.
The following software checks may be run on the host to verify the FPGA has been detected and has auto-negotiated the correct PCIe link width/speed. These commands do not require any packages to be installed. We are using PCIe BDF b1:00.0 as an example address.
# Check that the board has enumerated successfully.\n# Your PCIe BDF may differ from what is shown below.\n$ lspci | grep accel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce b1:00.1 Processing accelerators: Intel Corporation Device bcce\n\n# Check PCIe link status and speed. Width should be x16, and speed whould be 16GT/s\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\nThe OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit the opae.io page, and the Software Reference Manual
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access. If you wish to install pre-built artifacts instead of building the release yourself, skip steps 3 and 4.
Before Installing the newest version of OPAE you must remove any prior OPAE framework installations.
$ sudo apt-get remove opae*\nThe following system and Python3 package dependencies must be installed before OPAE may be built.
$ sudo apt-get install bison flex git ssh pandoc devscripts debhelper cmake python3-dev libjson-c-dev uuid-dev libhwloc-dev doxygen libtbb-dev libncurses-dev libspdlog-dev libspdlog1 python3-pip libedit-dev pkg-config libcli11-dev libssl-dev dkms libelf-dev gawk openssl libudev-dev libpci-dev libiberty-dev autoconf llvm\n\n$ python3 -m pip install setuptools pybind11 jsonschema\nClone the OPAE SDK repo. In this example we will use the top level directory OFS for our package installs.
$ mkdir OFS && cd OFS\n$ git init\n$ git clone https://github.com/OFS/opae-sdk\n$ cd opae-sdk\n$ git checkout tags/2.12.0-5\n\n# Verifying we are on the correct release tag\n$ git describe --tags\n2.12.0-5\nNavigate to the automatic DEB package build script location and execute.
$ cd OFS/opae-sdk/packaging/opae/deb $ ./create\n\n# Verify all packages are present\n$ ls | grep opae.*.deb\nopae_2.12.0-5_amd64.deb\nopae-dbgsym_2.12.0-5_amd64.ddeb\nopae-devel_2.12.0-5_amd64.deb\nopae-devel-dbgsym_2.12.0-5_amd64.ddeb\nopae-extra-tools_2.12.0-5_amd64.deb\nopae-extra-tools-dbgsym_2.12.0-5_amd64.ddeb\nInstall your newly built OPAE SDK packages.
$ cd OFS/opae-sdk/packaging/opae/deb\n$ sudo dpkg -i opae*.deb\nThe OPAE SDK version installed the host are identical to those installed on the SoC. A set of pre-compiled OPAE SDK artifacts are included in this release. These can be downloaded from ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell and installed without building/configuring.
$ tar xf opae-*.tar.gz\n$ sudo dpkg -i opae*.deb\nEnable iommu=on, pcie=realloc, and set hugepages as host kernel parameters.
# Check if parameters are already enabled\n$ cat /proc/cmdline\nIf you do not see intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200, then add them manually.
$ sudo vim /etc/default/grub\n\n# Edit the value for GRUB_CMDLINE_LINUX, add the values at the end of the variable inside of the double quotes. Example: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/rhel00-swap rd.lvm.lv=rhel00/root rd.lvm.lv=rhel00/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\"\n# Save your changes, then apply them with the following\n$ sudo grub2-mkconfig\n$ sudo reboot now\nAfter rebooting, check that proc/cmdline reflects your changes.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"f2000x/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\nHere is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\nIn main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\nMapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\nThe below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\nThe below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\nIn main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\nThe host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\nWhen the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\nA Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\nTo run the host application, you will need to:
See the associated AFU Developer Guide for details.
"},{"location":"f2000x/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"f2000x/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"f2000x/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"f2000x/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":"When using ASE in the application development cycle, consider the following limitations:
AFU development using the ASE includes the following four stages:
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"f2000x/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\nThis directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\nBefore configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\nafu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.generate_ase_environment.py: This script instantiates your simulated platform configuration.The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
synopsys_sim.setup and vcs_run.tcl in the configuration directory.vsim_run.tcl in the configuration directory.The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"f2000x/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\nThe AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code.cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"f2000x/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"f2000x/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\nASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\nCreate a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\nOptionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> .. /usr. $ sudo make install ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\nChange directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n... The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\nYou can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c. # Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\nNote: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\nUpon completion, the simulation generates the following files:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"f2000x/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
ASE_LOG to 0.Two log levels are supported in ASE, controlled by env(ASE_LOG)
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\nThe following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If usingASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"f2000x/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"f2000x/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI XA Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\nThe OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"f2000x/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support:<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.To interface the PIM with an AFU:
For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\nofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"f2000x/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"f2000x/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\nThe Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\nUsing the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\nThe following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\nLocal memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"f2000x/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\nUsing the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\nThe following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\nThe High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"f2000x/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\nIn digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\nInstantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\nThe AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"f2000x/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\nThe AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\nThe Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\nThe MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"f2000x/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"f2000x/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\nThe AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.svThe HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\nThe host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"f2000x/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"f2000x/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\nThe ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\nThe Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\nThe source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\nBelow is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
The Open FPGA Stack (OFS) Docker image has two main personas:
A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"f2000x/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"f2000x/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"f2000x/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"f2000x/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"f2000x/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\nAdd the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\nInstall the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\nStart the Docker daemon:
sudo systemctl start docker\nEnable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\nVerify that Docker is installed and running:
sudo systemctl status docker\nYou should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\nYou will need to log out and back in for the changes to take effect.
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\nThe Docker installation steps for Ubuntu are the following:
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\nInstall packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\nAdd Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\nThe following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\nUpdate the package manager index again:
sudo apt-get update\nInstall Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\nStart the Docker daemon:
sudo systemctl start docker\nEnable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\nVerify that Docker is installed and running:
sudo systemctl status docker\nYou should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\nYou will need to log out and back in for the changes to take effect.
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\nThe Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"f2000x/ug_docker/#build-the-image","title":"Build the image","text":"You can download the Dockefile from OFS GitHub Docker.
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\nSave the file
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\nNote: Never remove --no-cache this could cause issues with your environmental variables inside of the container
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\nDocker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\nFor more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\nInside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\nThe docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nTip: you can change myOFS with any other value. The value is the given name of the container.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\nNow the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\nYour Container ID is bdc1289fb081.
"},{"location":"f2000x/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\nThe container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\nEverything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
The OFS Docker image already includes the evaluation script.
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\nThe Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\nIf you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\nall the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"f2000x/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"f2000x/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nExample, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\nTip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\nNow, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n\u200b Your Container ID is 25b41eb4d232.
"},{"location":"f2000x/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\nThe container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\nEverything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
The OFS Docker image already includes the eval script.
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\nNow you can control and compile the design using the interactive or de-attach mode.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\nThis command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\nIn this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"f2000x/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":"lsmod | grep kvm\nIf the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\nVirtual Machine Manager (also known as libvirt) can be installed by following the below steps:
Update your system package index by running the following command:
sudo dnf update\nsudo apt update\nInstall the libvirt package and any required dependencies by running the following command:
sudo dnf install @virtualization\nsudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\nStart the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\nsudo dnf install virt-manager\nsudo usermod -a -G libvirt USERNAME\nvirt-manager as the non-root user.Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\nYou will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\nreboot\nAfter completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"f2000x/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
virt-manager GUI by running the following command:sudo virt-manager&\nCreate a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
In the virt-manager window, click the \"New virtual machine\" button.
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
Click on the \"+\" icon (Bottom left) to create the storage pool.
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"f2000x/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n\u200b
"},{"location":"f2000x/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3 Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\nCheck that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\nThe following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"f2000x/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbbEnsure you add the desired VF in your PCIE devices list.
For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nPass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\nEdit the XML file for your machine and include the following
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\nInside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\nEnsure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\nNote: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
Save the changes \"Apply\"
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\nEnsure your devices are enumerated properly.
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n2. Deployment Mode:
B1:00.5\nUnder the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n2. Deployment Mode:
177:00.0\nClick on \"Begin Installation.\" and follow the wizard installation of the OS.
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
Under your virtual machine, configure your VM proxy:
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\nUse the Virtual function (Not supported at management mode)
Ensure the DFL kernel drivers is install in your VM system
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\nVerify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\nTest the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\nAfter the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/","title":"Board Installation Guidelines: Intel\u00ae FPGA SmartNIC N6000/1-PL, Intel\u00ae FPGA PAC D5005","text":"Last updated: November 19, 2024
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup - the Intel\u00ae FPGA SmartNIC N6000/1-PL and the Intel\u00ae FPGA PAC D5005. After reading the document a user shall be able to:
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to all three platforms.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported ADP platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-2-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 2: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platforms you have if you are unsure. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers from sections, whose installation is covered in the [Software Installation Guide: OFS for PCIe Attach FPGAs].
fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table provides a picture reference for the hardware components discussed in the rest of the document.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-3-hardware-bkc","title":"Table 3: Hardware BKC","text":"Component Image Intel\u00ae FPGA SmartNIC N6001-PL (SKU2) Supermicro Server SYS-220HE Intel FPGA Download Cable II (Only Required for manual flashing) 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing)In addition to the above, all OFS ADP platforms require an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector. This cable will differ between server vendors - review the pinout of the power connector on the Intel\u00ae FPGA Programmable Acceleration Card D5005 Data Sheet or Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based ADP.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#20-initial-server-setup","title":"2.0 Initial Server Setup","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#21-server-information-for-intel-fpga-smartnic-n60001-pl","title":"2.1 Server Information for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Both the server BIOS and BMC need to match the versions listed below in Table 4: Supermicro Server BMC BKC. These updates only apply for this specific Best Known Configuration (BKC) - other server manufacturers may require different BIOS updates. Please consult your server's user guide and release notes for update information.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-4-supermicro-server-bmc-bkc","title":"Table 4: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4)Information about the server\u2019s currently loaded firmware can be found on the BMC web portal dashboard. Accessing this page requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d. During boot the BMC\u2019s login IP will be presented on the screen.
Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case. It is recommended the user change their BMC\u2019s default username as soon as they are able.
After logging in you should be able to review information about the BMC and BIOS by referring to the System box, visible upon initial loading of the page. Double check that the values match those in Table 4. If they do not, you may download the appropriate versions from the SuperMicro product page by selecting the BIOS option and downloading the most recent \u201cBundled Software File Name\u201d. Follow the BMC and BIOS update instructions included in the SuperMicro manuals page in the document X12/H12 BMC Manual in Appendix A.2 Updating Firmware Using BMC Web GUI.
If using a different server model, refer to that server\u2019s user guide for instructions on remote system management. Ensure that any system you end up using meets all the following requirements:
Refer to sections 2.1-2.3 of the Intel Acceleration Stack Quick Start Guide: Intel FPGA Programmable Acceleration Card D5005 for a complete overview of the physical installation process and ESD precautions for the D5005 platform.
Ensure that the system meets all the following requirements before proceeding to install the Intel\u00ae FPGA PAC D5005 into a server.
Detailed mechanical for information can be found on the D5005 Data Sheet and in section 4.0 Mechanical Information of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837).
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#30-server-settings","title":"3.0 Server Settings","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#31-bios-settings","title":"3.1 BIOS Settings","text":"You must enable Intel VT-x/VT-d technologies for the PCIe slot housing your ADP. The following steps are known to work on a SuperMicro SYS-220HE server platform.
To enter the Supermicro server\u2019s BIOS setup page, reboot, and press \\<Delete> when prompted. You can browse the tabs / options with a combination of arrow keys along with \\<Escape> and \\<Enter>.
Navigate right to the Advanced tab, then select the following menu options: Chipset Configuration -> North Bridge -> IIO Configuration -> Intel VT for Directed I/O (VT-d).
If not already, enable the option Intel VT for Directed I/O (VT-d).
The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
Please refer to sections 8.1 and 8.2 of the Intel FPGA Programmable Acceleration Card D5005 Data Sheet or section 6.0 of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) for guidance on cooling specifications that must be met when using these platforms. Failure to adhere to these guidelines may result in thermal runaway and/or performance degradation.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#40-board-installation-procedure","title":"4.0 Board Installation Procedure","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#41-pcie-slot-mappings-for-intel-fpga-smartnic-n60001-pl","title":"4.1 PCIe Slot Mappings for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"The Intel N6000/1-PL FPGA SmartNIC Platforms are officially verified in the upper middle PCIe x16 slot (Slot 3). If using a different slot, refer to the information in Table 5 PCIe Slot Mapping for which port settings to change in server BIOS.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-5-pcie-slot-mapping","title":"Table 5: PCIe Slot Mapping","text":"CPU Number Port Number (in BIOS) PCIe Slot CPU1 Port 2 5 and 6 CPU1 Port 4 7 and 8 CPU2 Port 2 1 and 2 CPU2 Port 4 3 and 4"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#42-installation-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.2 Installation Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe installation of an ADP platform into a supported server. While an Intel\u00ae FPGA SmartNIC N6001-PL is shown in the images below, this procedure applies to all three platforms.
Do not bend the card while inserting into a slot. Do not apply much pressure in regions 2 or 3 while inserting.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#43-removal-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.3 Removal Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe removal of the platforms from a supported server.
Do not bend the card while removing it from the slot.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/","title":"Board Installation Guidelines: Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)","text":"Last updated: November 19, 2024
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup for the F-Series (2x F-Tile) and I-Series (2x R-Tile and 1xF-Tile) Development Kits. After reading the document a user shall be able to:
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to both platforms.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported development kit platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#table-2-hardware-bkc-for-ofs-pcie-attach-targeting-the-f-series-development-kit","title":"Table 2: Hardware BKC for OFS PCIe Attach targeting the F-Series Development Kit","text":"The following table highlights the hardware which composes the Best Known Configuation (BKC) for the OFS 2024.2-1 PCIe Attach release targeting F-Series Development Kit.
Note: The Dell R750 server product line is known not to work with this release.
Component Link Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/agilex/agf027-and-agf023.html Intel FPGA Download Cable II (optional) https://www.intel.com/content/www/us/en/products/sku/215664/intel-fpga-download-cable-ii/specifications.html SuperMicro SYS-220HE-FTNR https://www.supermicro.com/en/products/system/hyper/2u/sys-220he-ftnrIn addition to the above, both OFS enabled development kit platforms require either an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector, or the standalone AC Power Supply with associated 2x4 power connector. You must choose one of these modes to operate your device in.
A server ATX power cable will differ between vendors - review the pinout of the power connector on the Intel Agilex\u00ae 7 FPGA I-Series Development Kit User Guide or Intel Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit User Guide as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based development kit.
In addition, the server used to house either dev kit must meet the following guidelines:
Both platforms ship with an embedded Intel\u00ae FPGA Download Cable II on the rear of the device which will be used for device programming. An on-board micro-USB connector on the rear of the development kit provides the data to the Intel\u00ae MAX\u00ae 10. This allows configuration of the FPGA using a USB cable directly connected to a PC running the Intel\u00ae Quartus\u00ae Prime software without requiring the external download cable dongle. An external download cable dongle can also be used on J10 to configure the FPGA, although both cannot be used at the same time.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#20-server-settings","title":"2.0 Server Settings","text":""},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#21-bios-settings","title":"2.1 BIOS Settings","text":"These are the host BIOS settings known to work with the either dev kit. Information about the server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings.
The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
Light pipes located on the top of the QSFP cages for the F-Series Dev Kit may or may not cause physical fit issues for some server platforms. If you run into any issues during installation you may remove the light pipes:
The DK-DEV-AGF027F1ES (or it is called the F - tile Dev Kit, or FM86 Dev Kit) has LED light pipes on top of the QSFP cages.
These light pipes interfere with the server PCIe slot faceplate.
The light pipes can be easily removed by prying them off using a small screwdriver for leverage, then pushing the light pipes back to remove the retaining clips from the QSFP cage.
Double check that your development kit switch settings match those listed as the default positions in the user guide prior to installation. An F Tile Dev Kit is used as an example in this section.
Board switch definitions can be found in the Intel Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit User Guide or Intel Agilex\u00ae 7 FPGA I-Series Development Kit User Guide.
See the image below for SW1, SW4 and SW3.
Before inserting into a server, set SW5 to 'ON'.
Below shows an F-Series Dev Kit installed into a PCIe riser with the light pipes removed.
The following instructions will help ensure safe installation of the Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) into a supported server platform. Safety and Regulatory information can be found under the product page for either development kit. It is assumed you have previously removed the light pipes mounted above the F-Series Dev Kit's QSFP cages before attempting to slot into a server mounted riser if they are an issue.
Both Development Kits have an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG. Perform the following steps to establish a JTAG connection:
Pre-requisites:
jtagconfig tool.Steps:
Refer to the following figure for Steps 2 and 3.
Locate Single DIP Switch SW2 and 4-position DIP switch SW3 on the fseries-dk. These switches control the JTAG setup for the board. Ensure that both SW2 and SW3.3 are set to ON.
Locate the J10 Micro-USB port on the fseries-dk. Connect a Micro-USB to USB-A cable between the J10 port and the workstation that has Quartus Prime Pro tools installed.
Use the jtagconfig tool to check that the JTAG chain contains the AGFB027R24C2E2VR2 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\nExample expected output:
1) Agilex F-Series FPGA Dev Kit [1-6]\n0343B0DD AGFB027R24C(.|R2|R0)/..\n020D10DD VTAP10\nThis concludes the walkthrough for establishing a JTAG connection on the fseries-dk.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/","title":"Board Installation Guidelines: IPU Platform F2000X-PL","text":"Last updated: November 19, 2024
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users prepare their server and install the IPU Platform F2000X-PL. After reviewing this document, a user shall be able to:
The information in this document is intended for customers evaluating the IPU Platform F2000X-PL. The card is an acceleration development platform (ADP) intended to be used as a starting point for evaluation and development. This document will cover key topics related to server bring-up and physical platform installation, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-2-related-documentation","title":"Table 2: Related Documentation","text":"Name [Automated Evaluation User Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Software Reference Manual: Open FPGA Stack] [Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Workload Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Security User Guide: Open FPGA Stack] [KVM User Guide: Open FPGA Stack] [Docker User Guide: Open FPGA Stack] [OFS 2024.1 F2000X-PL Release Notes] - under \"Important Notes\" Cyclone\u00ae 10 LP Board Management Controller (BMC) User Guide for Intel\u00ae IPU F2000X-PL v1.2.4 (Document ID: 789706) 11 Work with your Altera sales representative for access.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#12-server-requirements","title":"1.2 Server Requirements","text":"The following requirements must be met when purchasing a server to support the IPU Platform F2000X-PL.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#121-host-server-specifications","title":"1.2.1 Host Server Specifications","text":"The host server must meet the following minimal specifications:
Te Host BIOS settings known to work with the IPU Platform F2000X-PL:
Specific BIOS paths are not listed here, as they can differ between BIOS vendors and versions.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#123-host-server-kernel-and-grub-configuration","title":"1.2.3 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested:
The IPU Platform F2000X-PL is a high-performance processing card with a passive heat sink to dissipate device heat and must be installed in a server with sufficient forced airflow cooling to keep all devices operating below maximum temperature. The table below lists the thermal terms and descriptions used in thermal analysis.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-5-thermal-terms-and-descriptions","title":"Table 5: Thermal Terms and Descriptions","text":"Term Description Cubic Feet per Minute (CFM) Volumetric airflow rate, in cubic feet per minute, of air passing through faceplate. Tj FPGA Junction Temperature TLA Local Ambient temperature. Temperature of forced air as it enters the IPU Platform F2000X-PL. \u00a0 Note: In many systems, this is higher than the room ambient due to heating effects of chassis components.Note: The FPGA junction temperature must not exceed 100\u00b0C. The case temperature of the QSFP modules must meet the module vendor's specification.
Note: The table below provides the thermal targets for which the IPU Platform F2000X-PL was designed. As a card manufacturer, you must qualify your own production cards.
The maximum card inlet air temperatures must support continuous operation under the worst-case power scenario of 150W TDP.
The airflow requirements for optimal heat sink performance at minimum is characteristic of CAT 3 servers or PCIe SIG Level 7 thermal profiles, in both, forward & reverse flow, see figure below:
As the IPU Platform F2000X-PL is a development platform, it is not integrated into the server baseband management controller closed loop cooling control. It is strongly recommended that you set your server's fan settings to run constantly at 100% with the server chassis lid closed to prevent unwanted IPU Platform F2000X-PL thermal shutdown.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#14-external-connections","title":"1.4 External Connections","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-1-external-connections","title":"Figure 1: External Connections","text":"The items listed Table 6 in are known to work for external connectivity. Specific links are given for convenience, other products may be used but have not been tested.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-6-external-connection-cables","title":"Table 6: External Connection Cables","text":"Item Part Number Link to source RS-232 to USB Adapter DTECH FTDI USB to TTL Serial Adapter, 3 m USB to TTL Serial USB to Ethernet Adapter, Aluminum 3 Port USB 3.0 Teknet USB Hub with Ethernet adapter Flash Drive, 64 GB or larger SanDisk QSFP DAC Cable \u00a0FS.com Generic 2m 100G QSP28 Passive Direct Attach Copper QSFP28 DAC (optional) Intel FPGA Download Cable II PL-USB2-BLASTER USB-Blaster II"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#15-preparing-the-ipu-platform-f2000x-pl-for-installation","title":"1.5 Preparing the IPU Platform F2000X-PL for Installation","text":"Turn the board over to back side and remove the Kapton tape covering switches SW2 and SW3 and make sure the switches are set as shown in Figure 1.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-7-switch-settings","title":"Table 7: Switch Settings","text":"Name Value SW3.1 off SW3.2 off SW3.2 on SW3.2 off SW2 off"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-2-board-switch-settings","title":"Figure 2: Board Switch Settings","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#151-usb-to-serial-adapter","title":"1.5.1 USB to Serial Adapter","text":"The IPU Platform F2000X-PL has a serial UART for access located on back edge of the board. This connection is useful for making BIOS and boot settings and for monitoring the SoC. In most servers, you will need to remove a riser card and route the USB to serial cable and (optional) Intel FPGA USB Blaster through an unused PCIe slot above or below where the IPU is installed. See Figure 3 for an example of cable routing.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-3-cable-routing","title":"Figure 3: Cable Routing","text":"The USB to serial connection is shown in Figure 4 where the White wire is TXD, Black wire is ground and Green wire is RXD.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-4-usb-to-serial-adapter-connection","title":"Figure 4: USB to Serial Adapter connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#152-ipu-jtag","title":"1.5.2 IPU JTAG","text":"The IPU Platform F2000X-PL provides a 10 pin JTAG header for FPGA and Cyclone 10 Board Management Controller development work using a Intel FPGA Download Cable II. This JTAG connection is optional for initial bring-up but is useful for manual image reprogramming and debug. See Figure 5 noting the orientation of the connection. The orientation of the USB Blaster II requires careful installation in a PCIe bay that has additional room in the adjacent bay. This may require you to either install the board over the PSU of the server, or to temporarily remove an adjacent riser while programming.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-5-usb-blaster-ii-connection","title":"Figure 5: USB Blaster II Connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-6-usb-blaster-ii-installation","title":"Figure 6: USB Blaster II Installation","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#153-power","title":"1.5.3 Power","text":"The IPU Platform F2000X-PL must receive power from both the 12 V and 3.3V PCIe slot and the 12 V Auxiliary 2\u00d74 power connector. The board does not power up if any of the 12 V and 3.3 V PCIe slot, or 12 V Auxiliary power sources are disconnected.
PCIe specifications define 12 V Auxiliary power connector pin assignment. The IPU Platform F2000X-PL implements an 8-position right angle (R/A) through-hole PCB header assembly on the top right side of the board as depicted in the picture below.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-7-12v-pcie-aux-connector-location","title":"Figure 7: 12V PCIe AUX Connector Location","text":"Refer the table below for pinout details.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-8-12v-2x3-aux-connector-pin-out","title":"Table 8: 12V (2x3) AUX Connector Pin Out","text":"Pin Description 1 +12V 2 +12V 3 +12V 4 Sense 1 5 Ground 6 Sense 0 7 Ground 8 GroundSee Auxiliary power connection in Figure 8.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-8-auxiliary-power-connection","title":"Figure 8: Auxiliary Power Connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#154-usb-hub-connection","title":"1.5.4 USB Hub Connection","text":"The USB Hub is connected to the USB type A connector on the front panel. Additionally, attach a network connected Ethernet connection to the USB hub. See Figure 9.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-9-usb-hub-connection","title":"Figure 9: USB Hub Connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#155-creating-a-bootable-usb-flash-drive-for-the-soc","title":"1.5.5 Creating a Bootable USB Flash Drive for the SoC","text":"Connect your flash drive to an available Linux host. In this section the USB will set up to be used as a secondary boot source for the SoC and will also be used to update the NVMe from which the ICX-D SoC boots in section 2.1 Updating the F2000X-PL ICX-D SoC NVMe.
You will load the latest pre-compiled Yocto core-image-minimal WIC image into USB flash. This image can be downloaded from ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell, under assets, or compiled from meta-ofs. Compilation is discussed in section 4.0 Compiling a Custom Yocto SoC Image.
Insert a 64 GB or larger USB Flash Drive into the USB slot of a computer/server you can use to format the drive. The following instructions assume you are using some flavor of GNU+Linux. You need sudo access privileges on this machine.
In a terminal window, find the device name of the USB flash drive and unmount the device:
$ lsblk\n\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\nsda 8:0 0 1.8T 0 disk\n\u251c\u2500sda1 8:1 0 600M 0 part /boot/efi\n\u251c\u2500sda2 8:2 0 1G 0 part /boot\n\u2514\u2500sda3 8:3 0 1.8T 0 part\n\u251c\u2500rhel-root 253:0 0 50G 0 lvm /\n\u251c\u2500rhel-swap 253:1 0 4G 0 lvm \\[SWAP\\]\n\u2514\u2500rhel-home 253:6 0 1.7T 0 lvm /home\nsdb 8:16 0 447.1G 0 disk\n\u251c\u2500sdb1 8:17 0 600M 0 part\n\u251c\u2500sdb2 8:18 0 1G 0 part\n\u2514\u2500sdb3 8:19 0 445.5G 0 part\n\u251c\u2500fedora_localhost\\--live-swap 253:2 0 4G 0 lvm\n\u251c\u2500fedora_localhost\\--live-home 253:3 0 301G 0 lvm\n\u251c\u2500fedora_localhost\\--live-root 253:4 0 70G 0 lvm\n\u2514\u2500fedora_localhost\\--live-centos_root 253:5 0 70.5G 0 lvm\nsdd 8:48 1 57.3G 0 disk\n\u2514\u2500sdd1 8:49 1 57.3G 0 part\nIn the above example, the 64 GB USB Flash device is designated sdd. Note, your device file name may be different. You are looking for an entry that matches the size of your USB Flash. You can also check the output of dmesg after manually plugging in your USB Flash device to view the name the kernel has given it in an auto-generated event.
Unmount the USB flash (if not already unmounted).
$ sudo umount /dev/sdd1\numount: /dev/sdd1: not mounted.\nDownload the Yocto WIC image. To prevent boot errors that may arise when using the same boot image loaded in both USB flash and on-board NVMe, you must choose an older version of the Yocto WIC to load onto the USB. Browse the tagged Yocto release images on GitHub and choose the second newest release image as the temporary USB boot target. In this example we will use the OFS 2023.1 RC3 release. You will also need to download the newest Yocto release image (core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.gz).
# Download an older version of the Yocto release image to use as the USB boot target, version 2023.1 RC3 shown here\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2023.1-3/core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2023.1-3/core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz.sha256\n# Verify the checksum of the downloaded image\n$ sha256sum -c https://github.com/OFS/meta-ofs/releases/download/ofs-2023.1-3/core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz.sha256\n# Uncompress the package\n$ gzip -d core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz\n\n# Download the most recent Yocto release image, which will overwrite on-board NVMe\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2024.1-2/core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2024.1-2/core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.sha256\n# Verify the checksum of the downloaded image\nsha256sum -c core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.sha256\n# Uncompress the package\n$ gzip -d core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic\nCopy core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic to the USB flash. This process may take several minutes.
$ sudo dd if=core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic of=/dev/sdd1 bs=512k status=progress conv=sync $ sgdisk -e /dev/sdd\nCreate a partition to store the Yocto image, which will be used to overwrite on-board NVMe as the default boot target.
$ sudo fdisk /dev/sdd\n Command (m for help): p\n Command (m for help): n\n Partition number (4-128, default 4): <<press enter>>\n First sector (14617908-125045390, default 14618624): <<press enter>>\n Last sector, +/-sectors or +/-size{K,M,G,T,P} (14618624-125045390, default\n 125045390): <<press enter>>\n Created a new partition 4 of type 'Linux filesystem' and of size 92 GiB.\n Command (m for help): p\nCommand (m for help): w\nVerify USB flash is partitioned.
$ lsblk\n NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\n sdd 8:0 1 114.6G 0 disk\n |-sdd1 8:1 1 300M 0 part\n |-sdd2 8:2 1 22.1G 0 part\n |-sdd3 8:3 1 44M 0 part\n `-sdd4 8:4 1 92.2G 0 part\nFormat the new partition.
$ mkfs -t ext4 /dev/sdd4\n$ mount /dev/sda4 /mnt\nCopy compressed core-image-minimal WIC into /mnt.
$ cp core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic /mnt\nRemove the USB flash from the Linux computer and install the flash drive in the USB hub attached to the IPU Platform F2000X-PL.
"},{"location":"hw/common/doc_modules/links/","title":"AFU Dev","text":""},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/","title":"Driver Development Quick Start Guidelines for Linux DFL","text":""},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#11-audience","title":"1.1 Audience","text":"The intended audience for this document is for developers looking to get starting in creating their own custom drivers with Linux DFL/DFH integration. It is assumed you have some prior knowledge of FPGA-based terminology and Linux Kernel Module development. It will be extremely helpful to have knowledge of the FPGA OPAE framework including plugins, SR-IOV, and MMIO. This information is intended as a starting point, with links where users can dive deeper.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#12-terminology","title":"1.2 Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#13-linux-dfl","title":"1.3 Linux DFL","text":"The Device Feature List is a kernel-based framework that provides support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in the PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.
A detailed explanation of the DFL framework and its parts and their functions has been upstreamed in the Linux Kernel documentation at dfl.html. Here you can find an overview of the DFL linked list structure, a breakdown of DFHv0 and DFHv1 header layouts, and introduction to the FPGA Management Engine (FME), FIU-Ports, Partial Reconfiguration, virtualization using SR-IOV, device enumeration, performance counters, interrupt support, and more. This is considered required reading before moving forward as terms and concepts will be used through the rest of the document.
There are different flavors of the DFL framework currently on offer on GitHub - an explanation of their purpose and branch structure can be found on the linux-dfl GitHub wiki pages.
If you wish to build existing DFL offerings from source, these steps are provided in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
The examples shown in this document are referencing values from the DFL implementation of a Virtual UART 8250_dfl.c, which demonstrates a minimal level implementation of a UART driver which plugs into the DFL framework. You may also find it helpful to review spi-altera-dfl.c, which similarly glues existing kernel SPI logic together with the DFL framework.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#20-information-handoff-with-workload-developer","title":"2.0 Information Handoff with Workload Developer","text":"When writing DFL enabled kernel-space drivers there are several key pieces of information you will need to gather from the RTL developer as a part of their design process. For a specific feature that is exposed to kernel-space that you are writing a driver for, you need the following:
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#table-1-hw-to-sw-id-mappings","title":"Table 1 - HW to SW ID Mappings","text":"Variable Description FME_FEATURE_ID The current value set can be seen in Device Feature List (DFL) Feature IDs. Create a pull request to merge your feature IDs with the current ID table. The RTL developer may pick any unused value for testing purposes while building their design as described in the Shell Technical Reference Manual. PCIe VID/DID/sVID/sDID The PCIe Vendor ID, Device ID, Subsystem Vendor ID, and Subsystem Device ID. A list of all values used by Altera is found in PCI IDs Containing Device Feature Lists. Information for modifying these values in your design are found in the Shell Developer Guide. param_id ID of the DFL paramater for the feature for features exposed to software in the RTL design, as set in FeatureID. Ensure the ID is not already reserved.In addition to the above, your driver logic will depend on bitmasks, register offsets, register configurations, data widths, and other general values which will be defined at the top of your driver file in the form of C preprocessor macros. These are entirely dependent on the workload being developed and will need to be communicated from the workload to the driver developer.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#30-module-definitions","title":"3.0 Module Definitions","text":"The following Macros are required when writing your own kernel driver code.
Creation of driver software involves interfacing with the same DFL-specific data structures and functions. The following is not a fully inclusive list:
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_device","title":"dfl_device*","text":"/**\n * struct dfl_device - represent an dfl device on dfl bus\n *\n * @dev: generic device interface.\n * @id: id of the dfl device.\n * @type: type of DFL FIU of the device. See enum dfl_id_type.\n * @feature_id: feature identifier local to its DFL FIU type.\n * @revision: revision of this dfl device feature.\n * @mmio_res: mmio resource of this dfl device.\n * @irqs: list of Linux IRQ numbers of this dfl device.\n * @num_irqs: number of IRQs supported by this dfl device.\n * @cdev: pointer to DFL FPGA container device this dfl device belongs to.\n * @id_entry: matched id entry in dfl driver's id table.\n * @dfh_version: version of DFH for the device\n * @param_size: size of the block parameters in bytes\n * @params: pointer to block of parameters copied memory\n */\nstruct dfl_device {\nstruct device dev;\nint id;\nu16 type;\nu16 feature_id;\nu8 revision;\nstruct resource mmio_res;\nint *irqs;\nunsigned int num_irqs;\nstruct dfl_fpga_cdev *cdev;\nconst struct dfl_device_id *id_entry;\nu8 dfh_version;\nunsigned int param_size;\nvoid *params;\n};\nRepresents a DFL device on the DFL bus. This value is automatically passed to drivers that register their devices in the Macro module_dfl_driver(__dfl_driver). The struct device dev local parameter is a pointer to the standard Linux container for devices. Other values are auto-populated in DFH by the RTL Developer as a part of workload development and exposed to you by the DFL framework.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_driver","title":"dfl_driver*","text":"/**\n * struct dfl_driver - represent an dfl device driver\n *\n * @drv: driver model structure.\n * @id_table: pointer to table of device IDs the driver is interested in.\n * { } member terminated.\n * @probe: mandatory callback for device binding.\n * @remove: callback for device unbinding.\n */\nstruct dfl_driver {\nstruct device_driver drv;\nconst struct dfl_device_id *id_table;\nint (*probe)(struct dfl_device *dfl_dev);\nvoid (*remove)(struct dfl_device *dfl_dev);\n};\nRepresents a DFL device driver as required by the kernel. The struct device_driver drv local parameter is a pointer to the standard Linux device_driver. Importantly it is up to the user to populate the dfl_device_id table, and provide a pointer to both the probe and remove functions for their DFL driver. dfl_device_id is an upstreamed data structure which identifies DFL devices to the kernel.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_device_id","title":"dfl_device_id*","text":"/**\n * struct dfl_device_id - dfl device identifier\n * @type: DFL FIU type of the device. See enum dfl_id_type.\n * @feature_id: feature identifier local to its DFL FIU type.\n * @driver_data: driver specific data.\n */\nstruct dfl_device_id {\n__u16 type;\n__u16 feature_id;\nkernel_ulong_t driver_data;\n};\nSimilarly, dfl_device_id is an upstreamed data structure used to identify devices as part of the DFL bus to the kernel.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_find_param","title":"dfl_find_param()","text":"/**\n * dfh_find_param() - find parameter block for the given parameter id\n * @dfl_dev: dfl device\n * @param_id: id of dfl parameter\n * @psize: destination to store size of parameter data in bytes\n *\n * Return: pointer to start of parameter data, PTR_ERR otherwise.\n */\nvoid *dfh_find_param(struct dfl_device *dfl_dev, int param_id, size_t *psize)\nThis function will find and return the value of a parameter as identified by the dfl_dev and param_id. This functions similarly to the PCIe find capability in that it scans PCIe bar space for the correct DFL information offsets to read off feature capabilities. Ultimately this function relies on the Linux Kernel Macro FIELD_GET(mask, reg) to extract a value from a hardware register give a bitmask and register value. This function iterates through all DFH offsets and returns NULL if it is unable to locate the param_id requested.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#50-probe-and-remove-functions-using-dfh","title":"5.0 Probe and Remove Functions Using DFH","text":"Every DFL driver requires a probe and removal function to be defined for matching devices. The examples shown here are referencing values from the DFL implementation of a Virtual UART 8250_dfl.c, which demonstrates a minimal level implementation of a UART driver which plugs into the DFL framework.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#51-driver-probing","title":"5.1 Driver Probing","text":"dfl_uart_probe() = The DFL enabled vUART probe function. A probe function is passed a struct * pcie_dev for any devices matching its ID table in most cases. Because we have registered our device as a dfl_uart_driver using module_dfl_driver_() Macro, we are instead passed a dfl_device * dev. In this example we use the pre-existing Serial 8250 driver implementation in the kernel while also plugging the device into the DFL framework.
static int dfl_uart_probe(struct dfl_device *dfl_dev)\nTo probe our device we need to accomplish the following:
Initializing the serial port is the only vUART specific function in this flow and can be replaced with the initialization functionality of the driver you are implementing.
Because the probe function is passed a (dfl_dev ) struct we have access to DFL specific functions for accessing device parameters via the Device Feature Header (DFH) structure. The DFL function (void *dfh_find_param(struct dfl_device *dfl_dev, int param_id, size_t *psize) can be used to read parameters as defined in the DFH given a **param_id*. This information allows us to read configuration information for our driver directly from DFH registers and pass it along to our initialization function.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#52-driver-removal","title":"5.2 Driver Removal","text":"dfl_uart_remove() is the DFL enabled vUART driver remove function. The logic in this function is related to the closure of any non-DFL structures from the kernel used by our driver. In this case we only need to worry about freeing the UART serial line that was previously claimed by our driver. If you have no logic tied to the removal of our driver this function is not necessary; the kernel will free memory automatically because of the work we put in to integrating our software with the driver framework.
static void dfl_uart_remove(struct dfl_device *dfl_dev)\n{\nstruct dfl_uart *dfluart = dev_get_drvdata(&dfl_dev->dev);\nserial8250_unregister_port(dfluart->line);\n}\nThis example provided below instantiates a DFL compatible driver without any feature-specific logic. This code will build an out-of-tree module and insert into a kernel, although without modification it doesn't accomplish anything. It will also not register itself with any known devices. Comments are provided in-line on where you can add in extra logic for your feature's use case, alongside some printk statements which will display in dmesg after the driver has been build and inserted with insmod <file>.ko.
You will need to install the Linux DFL driver framework before attempting to build as shown in Software Installation Guide: Open FPGA Stack for PCIe Attach alongside linux-headers-, linux-devel-, and any other minimal requirements. If you do not have any DFL compatible devices installed on your build machine, you will need to load the DFL kernel module code with sudo modprobe dfl before building.
#include <linux/bitfield.h>\n#include <linux/device.h>\n#include <linux/dfl.h>\n#include <linux/errno.h>\n#include <linux/ioport.h>\n#include <linux/module.h>\n#include <linux/mod_devicetable.h>\n#include <linux/types.h>\n/*\nInsert all build dependent #includes.\n*/\n/*\nInsert all feature-specific Macros that will be referenced as a part of your kernel probe, removal, and logic implementation. These values can be pulled from the RTL developer's build of your design and vary depending on requirements.\n*/\nstatic int dfh_get_u64_param_val(struct dfl_device *dfl_dev, int param_id, u64 *pval)\n{\nsize_t psize;\nu64 *p;\np = dfh_find_param(dfl_dev, param_id, &psize);\nif (IS_ERR(p))\nreturn PTR_ERR(p);\nif (psize != sizeof(*pval))\nreturn -EINVAL;\n*pval = *p;\nreturn 0;\n}\n/*\nUsing dfh_get_u64_param_val above, create a function that reads all relevant information from DFH for your design.\n*/\n/*\nYou will need to insert driver probe logic for your specific feature in this function. We use a stand-in struct 'driver_data' with bogus data.\n*/\nstruct driver_data {\nint field1;\nint field2;\nunsigned char field3;\nu64 size;\nu32 capabilities;\n};\nstatic int dfl_skeleton_probe(struct dfl_device *dfl_dev)\n{\nstruct device *dev = &dfl_dev->dev;\nstruct driver_data *data;\ndata = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);\nif (!data) {\nreturn -ENOMEM;\n}\nint ret;\ndev_set_drvdata(dev, data);\nprintk(KERN_INFO \"DFL Skeleton Driver probe function has completed!\\n\");\nreturn 0;\n}\nstatic void dfl_skeleton_remove(struct dfl_device *dfl_dev)\n{\nprintk(KERN_INFO \"DFL Skeleton Driver probe remove has completed!\\n\");\n}\n#define FME_FEATURE_ID_SKELETON 0x27\n//#define FME_GUIDS \\\n// GUID_INIT() \n// This section commented out to prevent build errors. Fill this in with your device's PCIe addressing information.\n// Add FME_GUIDS as the third item after FME_FEATURE_ID_SKELETON if you have used the Kernel Macro\nstatic const struct dfl_device_id dfl_driver_ids[] = {\n{ FME_ID, FME_FEATURE_ID_SKELETON },\n{ }\n};\nMODULE_DEVICE_TABLE(dfl, dfl_driver_ids);\nstatic struct dfl_driver dfl_skeleton_driver = {\n.drv = {\n.name = \"dfl-skeleton\",\n},\n.id_table = dfl_driver_ids,\n.probe = dfl_skeleton_probe,\n.remove = dfl_skeleton_remove,\n};\nmodule_dfl_driver(dfl_skeleton_driver);\nMODULE_DESCRIPTION(\"DFL Skeleton driver\");\nMODULE_AUTHOR(\"Your Name\");\nMODULE_LICENSE(\"GPL\");\nmake:obj-m += skeleton.o\n\nall:\nmake -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules\n\nclean:\nmake -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean\nThe information presented in this document is intended to be used by software developers looking to increase their knowledge of the OPAE SDK user-space software stack and the kernel-space linux-dfl drivers. This information is intended as a starting point, with links to where users can deep dive on specific topics.
Former OPAE and DFL software documents were combined with the Software Reference Manual to reduce clutter and more cleanly document OFS software capabilities. The following documents are no longer available as standalone as of ofs-2023.3:
The OPAE C library is a lightweight user-space library that provides abstraction for FPGA resources in a compute environment. Built on top of the OPAE Intel\u00ae FPGA driver stack that supports Intel\u00ae FPGA platforms, the library abstracts away hardware specific and OS specific details and exposes the underlying FPGA resources as a set of features accessible from within software programs running on the host. The OPAE source code is available on the OPAE SDK repository.
By providing a unified C API, the library supports different FPGA integration and deployment models, ranging from single-node systems with one or a few FPGA devices to large-scale FPGA deployments in a data center. At one end of the spectrum, the API supports a simple application using a PCIe link to reconfigure the FPGA with different accelerator functions. At the other end of the spectrum, resource management and orchestration services in a data center can use this API to discover and select FPGA resources and then allocate them for use by acceleration workloads.
The OPAE provides consistent interfaces to crucial components of the platform. OPAE does not constrain frameworks and applications by making optimizations with limited applicability. When the OPAE does provide convenience functions or optimizations, they are optional. For example, the OPAE provides an interface to allocate physically contiguous buffers in system memory that user-space software and an accelerator can share. This interface enables the most basic feature set of allocating and sharing a large page of memory in one API call. However, it does not provide a malloc()-like interface backed by a memory pool or slab allocator. Higher layers of the software stack can make such domain-specific optimizations.
Most of the information related to OPAE can be found on the official Open FPGA Stack (OFS) Collateral Site and in the OPAE SDK repository. The following is a summary of the information present on this web page:
The remaining sections on OPAE in this document are unique and build on basic principles explained in opae.github.io.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#table-1-additional-websites-and-links","title":"Table 1: Additional Websites and Links","text":"Document Link OPAE SDK on github OPAE SDK repository OPAE Documents Open FPGA Stack (OFS) Collateral Site pybind11 https://pybind11.readthedocs.io/en/stable/ CLI11 https://github.com/CLIUtils/CLI11 spdlog https://github.com/gabime/spdlog"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#table-2-ofs-hardware-terminology","title":"Table 2: OFS Hardware Terminology","text":"Term Description FPGA: Field Programmable Gate Array a discrete or integrated device connecting to a host CPU via PCIe or other type of interconnects. Accelerator Function Unit (AFU) The AFU is the supplied implementation of an accelerator, typically in HDL. AFUs implement a function such as compression, encryption, or mathematical operations.The Quartus Prime Pro software synthesizes the RTL logic into a bitstream. Accelerator Function (AF) The AF is the compiled binary for an AFU. An AF is a raw binary file (.rbf) bitstream. A tool (fpgaconf) reconfigures the FPGA using an AF bitstream. Reconfiguration The process of reprogramming the FPGA with a different AF."},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#20-opae-c-api","title":"2.0 OPAE C API","text":"The following OPAE data structures and functions integrate AFUs into the OPAE environment. The OPAE C API models these data structures and functions. For more information on the object models refer to the Object model section.
Linking with this library is straightforward. Code using the OPAE library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, here is the simplest compile and link command:
gcc myprog.c -I</path/to/fpga.h> -L</path/to/libopae-c.so> -lopae-c -luuid -ljson-c -lpthread
.. note::
The OPAE library uses the third-party `libuuid` and `libjson-c` libraries that are not distributed with \nthe OPAE library. Make sure to install these libraries.\nThe library source includes two code samples. Use these samples to learn how to call functions in the library. Build and run these samples to determine if your installation and environment are set up properly.
Refer to the Running the Hello FPGA Example chapter in the Intel\u00ae Acceleration Stack Quick Start Guide for for Intel Programmable Acceleration Card with Intel Arria\u00ae 10 GX FPGA for more information about using the sample code.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#high-level-directory-structure","title":"High-Level Directory Structure","text":"Building and installing the OPAE library results in the following directory structure on the Linux OS. Windows and MacOS have similar directories and files.
Directory & Files Contents include/opae Directory containing all header files include/opae/fpga.h Top-level header for user code to include include/opae/access.h Header file for accelerator acquire/release, MMIO, memory management, event handling, and so on include/opae/bitstream.h Header file for bitstream manipulation functions include/opae/common.h Header file for error reporting functions include/opae/enum.h Header file for AFU enumeration functions include/opae/manage.h Header file for FPGA management functions include/opae/types.h Various type definitions lib Directory containing shared library files lib/libopae-c.so The shared dynamic library for linking with the user application doc Directory containing API documentation doc/html Directory for documentation of HTML format doc/latex Directory for documentation of LaTex format doc/man Directory for documentation of Unix man page format"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#object-models","title":"Object Models","text":"fpga_objtype: An enum type that represents the type of an FPGA resource, either FPGA_DEVICE or FPGA_ACCELERATOR. An FPGA_DEVICE object corresponds to a physical FPGA device. Only FPGA_DEVICE objects can invoke management functions. The FPGA_ACCELERATOR represents an instance of an AFU. fpga_token: An opaque type that represents a resource known to, but not necessarily owned by, the calling process. The calling process must own a resource before it can invoke functions of the resource.fpga_handle: An opaque type that represents a resource owned by the calling process. The API functions fpgaOpen() and fpgaClose() acquire and release ownership of a resource that an fpga_handle represents. (Refer to the Functions section for more information.)fpga_properties: An opaque type for a properties object. Your applications use these properties to query and search for appropriate resources. The FPGA Resource Properties section documents properties visible to your applications.fpga_event_handle: An opaque handle the FPGA driver uses to notify your application about an event. fpga_event_type: An enum type that represents the types of events. The following are valid values: FPGA_EVENT_INTERRUPT, FPGA_EVENT_ERROR, and FPGA_EVENT_POWER_THERMAL. (The Intel Programmable Acceleration Card (PAC) with Intel Arria 10 GX FPGA does not handle thermal and power events.)fpga_result: An enum type to represent the result of an API function. If the function returns successfully the result is FPGA_OK. Otherwise, the result is the appropriate error codes. Function fpgaErrStr() translates an error code into human-readable strings.The table below groups important API calls by their functionality. For more information about each of the functions, refer to the OPAE C API reference manual.
Functionality API Call FPGA Accelerator Description EnumerationfpgaEnumerate() Yes Yes Query FPGA resources that match certain properties Enumeration: Properties fpga[Get, Update, Clear, Clone, Destroy Properties]() Yes Yes Manage fpga_properties life cycle fpgaPropertiesGet[Prop]() Yes Yes Get the specified property Prop, from the FPGA Resource Properties table fpgaPropertiesSet[Prop]() Yes Yes Set the specified property Prop, from the FPGA Resource Properties table Access: Ownership fpga[Open, Close]() Yes Yes Acquire/release ownership Access: Reset fpgaReset() Yes Yes Reset an accelerator Access: Event handling fpga[Register, Unregister]Event() Yes Yes Register/unregister an event to be notified about fpga[Create, Destroy]EventHandle() Yes Yes Manage fpga_event_handle life cycle Access: MMIO fpgaMapMMIO(), fpgaUnMapMMIO() Yes Yes Map/unmap MMIO space fpgaGetMMIOInfo() Yes Yes Get information about the specified MMIO space fpgaReadMMIO[32, 64]() Yes Yes Read a 32-bit or 64-bit value from MMIO space fpgaWriteMMIO[32, 64]() Yes Yes Write a 32-bit or 64-bit value to MMIO space Memory management: Shared memory fpga[Prepare, Release]Buffer() Yes Yes Manage memory buffer shared between the calling process and an accelerator fpgaGetIOAddress() Yes Yes Return the device I/O address of a shared memory buffer fpgaBindSVA() Yes Yes Bind IOMMU shared virtual addressing Management: Reconfiguration fpgaReconfigureSlot() Yes No Replace an existing AFU with a new one Error report fpgaErrStr() Yes Yes Map an error code to a human readable string .. note::
The UMsg APIs are not supported for the Intel(R) PAC cards. They will be deprecated in a future release.\nApplications query resource properties by specifying the property name for Prop in the fpgaPropertiesGet[Prop]() and fpgaPropertiesSet[Prop]() functions. The FPGA and Accelerator columns state whether or not the Property is available for the FPGA or Accelerator objects.
fpga_token of the parent object ObjectType Yes Yes The type of the resource: either FPGA_DEVICE or FPGA_ACCELERATOR Bus Yes Yes The bus number Device Yes Yes The PCI device number Function Yes Yes The PCI function number SocketId Yes Yes The socket ID DeviceId Yes Yes The device ID NumSlots Yes No Number of AFU slots available on an FPGA_DEVICE resource BBSID Yes No The FPGA Interface Manager (FIM) ID of an FPGA_DEVICE resource BBSVersion Yes No The FIM version of an FPGA_DEVICE resource VendorId Yes No The vendor ID of an FPGA_DEVICE resource GUID Yes Yes The GUID of an FPGA_DEVICE or FPGA_ACCELERATOR resource NumMMIO No Yes The number of MMIO space of an FPGA_ACCELERATOR resource NumInterrupts No Yes The number of interrupts of an FPGA_ACCELERATOR resource AcceleratorState No Yes The state of an FPGA_ACCELERATOR resource: either FPGA_ACCELERATOR_ASSIGNED or FPGA_ACCELERATOR_UNASSIGNED"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#opae-c-api-return-codes","title":"OPAE C API Return Codes","text":"The OPAE C library returns a code for every exported public API function. FPGA_OK indicates successful completion of the requested operation. Any return code other than FPGA_OK indicates an error or unexpected behavior. When using the OPAE C API, always check the API return codes.
FPGA_OK Operation completed successfully FPGA_INVALID_PARAM Invalid parameter supplied FPGA_BUSY Resource is busy FPGA_EXCEPTION An exception occurred FPGA_NOT_FOUND A required resource was not found FPGA_NO_MEMORY Not enough memory to complete operation FPGA_NOT_SUPPORTED Requested operation is not supported FPGA_NO_DRIVER Driver is not loaded FPGA_NO_DAEMON FPGA Daemon (fpgad) is not running FPGA_NO_ACCESS Insufficient privileges or permissions FPGA_RECONF_ERROR Error while reconfiguring FPGA"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#211-device-abstraction","title":"2.1.1 Device Abstraction","text":"The OPAE C API relies on two base abstractions concerning how the FIM and accelerator are presented to and manipulated by the user. The FIM is concerned with management functionality. Access to the FIM and its interfaces is typically restricted to privileged (root) users. The accelerator contains the user-defined logic in its reconfigurable region. Most OPAE end-user applications are concerned with querying and opening the accelerator device, then interacting with the AFU via MMIO and shared memory.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2111-device-types","title":"2.1.1.1 Device types","text":"The C enum fpga_objtype defines two variants. The FPGA_DEVICE variant corresponds to the FIM portion of the device, and the FPGA_ACCELERATOR refers to the accelerator, also known as the AFU.
An FPGA_DEVICE refers loosely to the sysfs tree rooted at the dfl-fme.X directory, for example /sys/class/fpga_region/region0/dfl-fme.0, and its associated device file /dev/dfl-fme.0.
An FPGA_ACCELERATOR refers loosely to the sysfs tree rooted at the dfl-port.X directory, for example /sys/class/fpga_region/region0/dfl-port.0, and its associated device file /dev/dfl-port.0.
The number X in dfl-fme.X and dfl-port.X refers to a numeric ID that is assigned by the DFL device driver to uniquely identify an instance of the FIM/Accelerator. Systems with multiple FPGA acceleration devices will have multiple dfl-fme.X\u2019s and matching dfl-port.X\u2019s.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2112-tokens-and-handles","title":"2.1.1.2 Tokens and Handles","text":"An fpga_token is an opaque data structure that uniquely represents an FPGA_DEVICE or an FPGA_ACCELERATOR. Tokens convey existence, but not ownership. Tokens are retrieved via the OPAE enumeration process described below using the fpgaEnumerate() call.
An fpga_handle is an opaque data structure that corresponds to an opened device instance, whether FPGA_DEVICE or FPGA_ACCELERATOR. A Handle is obtained from a token via the fpgaOpen() call. A handle conveys that the /dev/dfl-fme.X or /dev/dfl-port.X device file has been opened and is ready for interaction via its IOCTL interface.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#212-enumeration","title":"2.1.2 Enumeration","text":"Enumeration is the process by which an OPAE application becomes aware of the existence of FPGA_DEVICE\u2019s and FPGA_ACCELERATOR\u2019s. Refer to the signature of the fpgaEnumerate() call:
fpga_result fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters,\nfpaa_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\nFigure 1 fpgaEnumerate()
The typical enumeration flow involves an initial call to fpgaEnumerate() to discover the number of available tokens.
uint32_t num_matches = 0;\nfpgaEnumerate(NULL, 0, NULL, 0, &num_matches);\nFigure 2 Discovering Number of Tokens
Once the number of available tokens is known, the application can allocate the correct amount of space to hold the tokens:
fpga_token *tokens;\nuint32_t num_tokens = num_matches;\ntokens = (fpga_token *)calloc(num_tokens, sizeof(fpga_token));\nfpgaEnumerate(NULL, 0, tokens, num_tokens, &num_matches);\nFigure 3 Enumerating All Tokens
Note that parameters filters and num_filters were not used in the preceding example, as they were NULL and 0. When no filtering criteria are provided, fpgaEnumerate() returns all tokens that can be enumerated.
The function will assert a reset on the currently loaded AFU for a selected device through an ioctl call on its VFIO_GROUP.
An fpga_properties is an opaque data structure used to retrieve all of the properties concerning an FPGA_DEVICE or FPGA_ACCELERATOR. These properties can be included in the filters parameter to fpgaEnumerate() to select tokens by specific criteria.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#21211-common-properties","title":"2.1.2.1.1 Common Properties","text":"Table 3 lists the set of properties that are common to FPGA_DEVICE and FPGA_ACCELERATOR:
Property Description fpga_guid guid; FPGA_DEVICE: PR Interface ID FPGA_ACCELERATOR: AFU ID fpga_token parent; FPGA_DEVICE: always NULL FPGA_ACCELERATOR: the token of the corresponding FPGA_DEVICE, if any. Otherwise, NULL. fpga_objtype objtype; FPGA_DEVICE or FPGA_ACCELERATOR uint16_t segment; The segment portion of the PCIe address: ssss:bb:dd.f uint8_t bus;The bus portion of the PCIe address:
ssss:bb:dd.f
uint8_t device;The device portion of the PCIe address:
ssss:bb:dd.f
uint8_t function; The function portion of the PCIe address: ssss:bb:dd.f uint64_t object_id; A unique 64-bit value that identifies this token on the system. uint16_t vendor_id; The PCIe Vendor ID uint16_t device_id; The PCIe Device ID uint32_t num_errors; The number of error sysfs nodes available for this token. fpga_interface interface; An identifier for the underlying plugin-based access method.Table 3 Common Properties
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#21212-fpga_device-properties","title":"2.1.2.1.2 FPGA_DEVICE Properties","text":"Table 4 lists the set of properties that are specific to FPGA_DEVICE token types.
Property Description uint64_t bbs_id; FIM-specific Blue Bitstream ID fpga_version bbs_version; BBS versionTable 4 FPGA_DEVICE Properties
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#21213-fpga_accelerator-properties","title":"2.1.2.1.3 FPGA_ACCELERATOR Properties","text":"Table 5 lists the set of properties that are specific to FPGA_ACCELERATOR token types.
Property Description fpga_accelerator_state state; Whether the Accelerator is currently open uint32_t num_mmio; The number of MMIO regions available uint32_t num_interrupts; The number of interrupts availableTable 5 FPGA_ACCELERATOR Properties
Following is an example of using fpga_properties to enumerate a specific AFU:
#define NLB0_AFU \"D8424DC4-A4A3-C413-F89E-433683F9040B\"\nfpga_properties filter = NULL;\nfpga_guid afu_id;\nfpgaGetProperties(NULL, &filter); // NULL: a new empty properties\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nuuid_parse(NLB0_AFU, afu_id);\nfpgaPropertiesSetGuid(filter, afu_id);\nfpgaEnumerate(&filter, 1, tokens, num_tokens, &num_matches);\nFigure 4 Filtering During Enumeration
Note that fpga_properties and fpga_token\u2019s are allocated resources that must be freed by their respective API calls, ie fpgaDestroyProperties() and fpgaDestroyToken().
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#213-access","title":"2.1.3 Access","text":"Once a token is discovered and returned to the caller by fpgaEnumerate(), the token can be converted into a handle by fpgaOpen(). Upon a successful call to fpgaOpen(), the associated /dev/dfl-fme.X (FPGA_DEVICE) or /dev/dfl-port.X (FPGA_ACCELERATOR) is opened and ready for use. Having acquired an fpga_handle, the application can then use the handle with any of the OPAE APIs that require an fpga_handle as an input parameter.
Like tokens and properties, handles are allocated resources. When a handle is no longer needed, it should be closed and released by calling fpgaClose().
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#214-events","title":"2.1.4 Events","text":"Event registration in OPAE is a two-step process. First, the type of event must be identified. The following fpga_event_type variants are defined:
Event Description FPGA_EVENT_INTERRUPT AFU interrupt FPGA_EVENT_ERROR Infrastructure error event (FME/Port Error)Table 6 FPGA Event Types
Once the desired event type is known, an fpga_event_handle is created via fpgaCreateEventHandle(). Once the event handle is available, the event notification is registered using fpgaRegisterEvent(). In the example below, note the use of the flags field for passing the desired IRQ vector when the event type is FPGA_EVENT_INTERRUPT. With the event registered, the application can then use fpgaGetOSObjectFromEventHandle() to obtain a file descriptor for use with the poll() system call. When the interrupt occurs, the file descriptor will be set to the signaled state by the DFL driver.
fpga_event_handle event_handle = NULL;\nint fd = -1;\nfpgaCreateEventHandle(&event_handle);\nfpgaRegisterEvent(fpga_handle, FPGA_EVENT_INTERRUPT,\nevent_handle, irq_vector);\nfpgaGetOSObjectFromEventHandle(event_handle, &fd);\nFigure 5 Creating and Registering Events
When an event notification is no longer needed, it should be released by calling fpgaUnregisterEvent(). Like device handles, event handles are allocated resources that must be freed when no longer used. To free an event handle, use the fpgaDestroyEventHandle() call.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#215-mmio-and-shared-memory","title":"2.1.5 MMIO and Shared Memory","text":"Communication with the AFU is achieved via reading and writing CSRs and by reading and writing to AFU/host shared memory buffers. An AFU\u2019s CSRs are memory-mapped into the application process address space by way of the fpgaMapMMIO() call.
uint32_t mmio_num = 0;\nfpgaMapMMIO(fpga_handle, mmio_num, NULL);\nfpgaWriteMMIO64(fpga_handle, mmio_num, MY_CSR, 0xa);\nFigure 6 Mapping and Accessing CSRs
The second parameter, mmio_num, is the zero-based index identifying the desired MMIO region. The maximum number of MMIO regions for a particular handle is found by accessing the num_mmio property. Refer to the fpgaPropertiesGetNumMMIO() call.
Once the AFU CSRs are mapped into the process address space, the application can use the fpgaReadMMIO**XX**() and fpgaWriteMMIO**XX**() family of functions, eg fpgaReadMMIO64() and fpgaWriteMMIO64(). When an MMIO region is no longer needed, it should be unmapped from the process address space using the fpgaUnmapMMIO() call.
Shared memory buffers are allocated by way of the fpgaPrepareBuffer() call.
fpga_result fpgaPrepareBuffer(fpga_handle handle,\nuint64_t len,\nvoid **buf_addr,\nuint64_t *wsid,\nint flags);\nFigure 7 fpgaPrepareBuffer()
Three buffer lengths are supported by this allocation method:
Length Description 4096 (4KiB) No memory configuration needed. 2097152 (2MiB) Requires 2MiB huge pages to be allocated. 1073741824 (1GiB) Requires 1GiB huge pages to be allocated.Table 7 fpgaPrepareBuffer() Lengths
echo 8 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages\necho 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages\nFigure 8 Configuring Huge Pages
The buf_addr parameter to fpgaPrepareBuffer() is a pointer to a void * that accepts the user virtual base address of the newly-created buffer. The wsid parameter is a pointer to a uint64_t that receives a unique workspace ID for the buffer allocation. This workspace ID is used in subsequent calls to fpgaReleaseBuffer(), which should be called when the buffer is no longer needed and in calls to fpgaGetIOAddress() which is used to query the IO base address of the buffer. The IO base address can be programmed into the AFU by means of the AFU CSR space. For example, here is a code snippet from the hello_fpga sample that demonstrates programming a shared buffer\u2019s IO base address into an AFU CSR in MMIO region 0:
#define LOG2_CL 6\n#define CACHELINE_ALIGNED_ADDR(p) ((p) >> LOG2_CL)\nfpgaGetIOAddress(accelerator_handle, input_wsid, &iova);\nfpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_SRC_ADDR,\nCACHELINE_ALIGNED_ADDR(iova));\nFigure 9 Programming Shared Memory
If applications need to map a shared buffer that has been allocated by some other means than fpgaPrepareBuffer(), then the flags parameter can be set to FPGA_BUF_PREALLOCATED. This causes fpgaPrepareBuffer() to skip the allocation portion of the call and to only memory map the given buf_addr into the application process address space.
Buffers can also be allocated and mapped as read-only by specifying FPGA_BUF_READ_ONLY.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#216-management","title":"2.1.6 Management","text":"The management feature in OPAE concerns re-programming the programmable region of the Port. To program the Port bitstream, pass a handle to the FPGA_DEVICE associated with the desired Port. The slot parameter identifies which Port to program in the case of multi-port implementations. Most designs will only pass zero as the slot parameter. The bitstream parameter is a buffer that contains the entire bitstream contents, including the JSON bitstream header information. The bitstream_len field gives the length of bitstream in bytes.
fpgaReconfigureSlot() first checks whether the FPGA_ACCELERATOR corresponding to the FPGA_DEVICE in fme_handle is open. If it is open, then the programming request is aborted with an error code. The application may pass FPGA_RECONF_FORCE in the flags parameter in order to avoid this open check and forcefully program the bitstream.
fpga_result fpgaReconfigureSlot(fpga_handle fme_handle,\nuint32_t slot,\nconst uint8_t *bitstream,\nsize_t bitstream_len,\nint flags);\nFigure 10 fpgaReconfigureSlot()
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#217-errors","title":"2.1.7 Errors","text":"The OPAE errors API provides a means to query and clear both FPGA_DEVICE and FPGA_ACCELERATOR errors. Each FPGA device exports a collection of error registers via the DFL drivers\u2019 sysfs tree, for both the FME and the Port. Each register is typically an unsigned 64-bit mask of the current errors, where each bit or some collection of bits specifies an error type. An error is signaled if its bit or collection of bits is non-zero. Note that the 32-bit error index may vary from one process execution to the next. Applications should use fpgaGetErrorInfo() and examine the error name returned in the struct fpga_error_info to identify the desired 64-bit error mask.
struct fpga_error_info {\nchar name[FPGA_ERROR_NAME_MAX];\nbool can_clear;\n};\nFigure 11 struct fpga_error_info
Each 64-bit mask of errors is assigned a unique 32-bit integer index and a unique name. Given an fpga_token and an error index, fpgaGetErrorInfo() retrieves the struct fpga_error_info corresponding to the error.
fpga_result fpgaGetErrorInfo(fpga_token token,\nuint32_t error_num,\nstruct fpga_error_info *error_info);\nFigure 12 fpgaGetErrorInfo()
fpgaReadError() provides access to the raw 64-bit error mask, given the unique error index. fpgaClearError() clears the errors for a particular index. fpgaClearAllErrors() clears all the errors for the given fpga_token.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#218-metrics","title":"2.1.8 Metrics","text":"The OPAE metrics API refers to a group of functions and data structures that allow querying the various device metrics from the Board Management Controller component of the FPGA device. A metric is described by an instance of struct fpga_metric_info.
typedef struct fpga_metric_info {\nuint64_t metric_num;\nfpga_guid metric_guid;\nchar qualifier_name[FPGA_METRIC_STR_SIZE];\nchar group_name[FPGA_METRIC_STR_SIZE];\nchar metric_name[FPGA_METRIC_STR_SIZE];\nchar metric_units[FPGA_METRIC_STR_SIZE];\nenum fpga_metric_datatype metric_datatype;\nenum fpga_metric_type metric_type;\n} fpga_metric_info;\nFigure 13 fpga_metric_info
The group_name field holds a string describing the broad categorization of the metric. Some sample values for group_name are \u201cthermal_mgmt\u201d and \u201cpower_mgmt\u201d. The metric_name field contains the metric\u2019s name. The number and names of metrics may vary from one FPGA platform to the next. The qualifier_name field is a concatenation of group_name and metric_name, with a colon character in between. The metric_units field contains the string name of the unit of measurement for the specific metric. Some examples for metric_units are \u201cVolts\u201d, \u201cAmps\u201d, and \u201cCelsius\u201d.
The metric_datatype field uniquely identifies the underlying C data type for the metric\u2019s value:
enum fpga_metric_datatype {\nFPGA_METRIC_DATATYPE_INT,\nFPGA_METRIC_DATATYPE_FLOAT,\nFPGA_METRIC_DATATYPE_DOUBLE,\nFPGA_METRIC_DATATYPE_BOOL,\nFPGA_METRIC_DATATYPE_UNKNOWN\n};\nFigure 14 enum fpga_metric_datatype
The metric_type field classifies the metric into a broad category. This information is redundant with the group_name field.
enum fpga_metric_type {\nFPGA_METRIC_TYPE_POWER,\nFPGA_METRIC_TYPE_THERMAL,\nFPGA_METRIC_TYPE_PERFORMANCE_CTR,\nFPGA_METRIC_TYPE_AFU,\nFPGA_METRIC_TYPE_UNKNOWN\n};\nFigure 15 enum fpga_metric_type
In order to enumerate the information for each of the metrics available from the FPGA device, determine the number of metrics using fpgaGetNumMetrics().
uint64_t num_metrics = 0;\nfpgaGetNumMetrics(handle, &num_metrics);\nFigure 16 Determining Number of Metrics
This call retrieves the number of available metrics for the FPGA_DEVICE that is opened behind the handle parameter to the call. Refer to 2.1.3 Access for information about the fpgaOpen() call. When the number of available metrics is known, allocate a buffer large enough to hold that many fpga_metric_info data structures, and call fpgaGetMetricsInfo() to populate the entries:
fpga_metric_info *metric_info;\nuint64_t metric_infos = num_metrics;\nmetric_info = calloc(num_metrics, sizeof(fpga_metric_info));\nfpgaGetMetricsInfo(handle, metric_info, &metric_infos);\nFigure 17 Querying Metrics Info
The fpga_metric structure is the representation of a metric\u2019s value:
typedef struct fpga_metric {\nuint64_t metric_num;\nmetric_value value;\nbool isvalid;\n} fpga_metric;\nFigure 18 struct fpga_metric
The metric_num field matches the metric_num field of the fpga_metric_info structure. value contains the metric value, which is encoded in the C data type identified by the metric_datatype field of fpga_metric_info. Finally, the isvalid field denotes whether the metric value is valid.
There are two methods of obtaining a metric\u2019s value, given the information in the fpga_metric_info structure:
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2181-querying-metric-values-by-index","title":"2.1.8.1 Querying Metric Values by Index","text":"fpgaGetMetricsByIndex() retrieves a metric value using the metric_num field of the metric info:
uint64_t metric_num = metric_info[0]->metric_num;\nfpga_metric metric0;\nfpgaGetMetricsByIndex(handle, &metric_num, 1, &metric0);\nFigure 19 Retrieve Metric by Index
This call allows retrieving one or more metric values, each identified by their unique metric_num. The second and fourth parameters allow passing arrays so that multiple values can be fetched in a single call.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2182-querying-metric-values-by-name","title":"2.1.8.2 Querying Metric Values by Name","text":"fpgaGetMetricsByName() retrieves a metric value using the metric_name field of the metric info:
char *metric_name = metric_info[1]->metric_name;\nfpga_metric metric1;\nfpgaGetMetricsByName(handle, &metric_name, 1, &metric1);\nThis call also allows retrieving one or more metric values, each identified by their unique metric_name. The second and fourth parameters allow passing arrays so that multiple values can be fetched in a single call.
The fpgaGetMetricsThresholdInfo() call is provided for legacy implementations only. It should be considered deprecated for current and future FPGA designs.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#219-sysobject","title":"2.1.9 SysObject","text":"When the hardware access method in use is the DFL drivers (see 2.3.2 libxfpga Plugin), the sysfs tree rooted at the struct _fpga_token\u2019s sysfspath member is accessible via the OPAE SDK SysObject API. The SysObject API provides an abstraction to search, traverse, read, and write sysfs entities. These sysfs entities may take the form of directories, which are referred to as containers, or files, which are referred to as attributes. Figure 20 enum fpga_sysobject_type shows the API\u2019s means of distinguishing between the two types.
enum fpga_sysobject_type {\nFPGA_OBJECT_CONTAINER = (1u << 0),\nFPGA_OBJECT_ATTRIBUTE = (1u << 1)\n};\nFigure 20 enum fpga_sysobject_type
The SysObject API introduces another opaque structure type, fpga_object. An fpga_object can be queried from an fpga_token or an fpga_handle by way of the fpgaTokenGetObject() and fpgaHandleGetObject() API\u2019s.
fpga_result fpgaTokenGetObject(fpga_token token, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaHandleGetObject(fpga_handle handle, const char *name,\nfpga_object *object, int flags);\nFigure 21 fpgaTokenGetObject() / fpgaHandleGetObject()
The remainder of the SysObject API is broken into two categories of calls, depending on the fpga_object\u2019s type. The type of an fpga_object is learned via fpgaObjectGetType().
fpga_result fpgaObjectGetType(fpga_object obj,\nenum fpga_sysobject_type *type);\nFigure 22 fpgaObjectGetType()
When an fpga_object is no longer needed, it should be freed via fpgaDestroyObject().
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2191-fpga_object_container-apis","title":"2.1.9.1 FPGA_OBJECT_CONTAINER API\u2019s","text":"For directory sysfs entities, passing a value of FPGA_OBJECT_RECURSE_ONE or FPGA_OBJECT_RECURSE_ALL in the flags parameter to fpgaTokenGetObject() or fpgaHandleGetObject() causes these two API\u2019s to treat the target object as either a single-layer or multi-layer directory structure, making its child entities available for query via fpgaObjectGetObject() and fpgaObjectGetObjectAt().
fpga_result fpgaObjectGetObject(fpga_object parent, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaObjectGetObjectAt(fpga_object parent, size_t idx,\nfpga_object *object);\nFigure 23 fpgaObjectGetObject() / fpgaObjectGetObjectAt()
Any child object resulting from fpgaObjectGetObject() or fpgaObjectGetObjectAt() must be freed via fpgaDestroyObject() when it is no longer needed.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2192-fpga_object_attribute-apis","title":"2.1.9.2 FPGA_OBJECT_ATTRIBUTE API\u2019s","text":"Attribute sysfs entities may be queried for their size and read from or written to. In order to determine the size of an attribute\u2019s data, use fpgaObjectGetSize().
fpga_result fpgaObjectGetSize(fpga_object obj,\nuint32_t *value, int flags);\nFigure 24 fpgaObjectGetSize()
Attributes containing arbitrary string data can be read with fpgaObjectRead().
fpga_result fpgaObjectRead(fpga_object obj, uint8_t *buffer,\nsize_t offset, size_t len, int flags);\nFigure 25 fpgaObjectRead()
If an attribute contains an unsigned integer value, its value can be read with fpgaObjectRead64() and written with fpgaObjectWrite64().
fpga_result fpgaObjectRead64(fpga_object obj,\nuint64_t *value, int flags);\nfpga_result fpgaObjectWrite64(fpga_object obj,\nuint64_t value, int flags);\nFigure 26 fpgaObjectRead64() / fpgaObjectWrite64()
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2110-utilities","title":"2.1.10 Utilities","text":"The fpga_result enumeration defines a set of error codes used throughout OPAE. In order to convert an fpga_result error code into a printable string, the application can use the fpgaErrStr() call.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#22-dfl-driver-ioctl-interfaces","title":"2.2 DFL Driver IOCTL Interfaces","text":"The DFL drivers export an IOCTL interface which the libxfpga.so plugin consumes in order to query and configure aspects of the FME and Port. These interfaces are used only internally by the SDK; they are not customer-facing. The description here is provided for completeness only.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#221-port-reset","title":"2.2.1 Port Reset","text":"The DFL_FPGA_PORT_RESET ioctl is used by the fpgaReset() call in order to perform a Port reset. The fpga_handle passed to fpgaReset() must be a valid open handle to an FPGA_ACCELERATOR. The ioctl requires no input/output parameters.
Note: fpgaReset() will always reset the entire FIM and AFU region of a device, and will not accept VFIO devices as an input argument. If you wish to issue a reset on the currently loaded AFU only, use the fpgaEnmuerate() function instead.
The DFL_FPGA_PORT_GET_INFO ioctl is used to query properties of the Port, notably the number of associated MMIO regions. The ioctl requires a pointer to a struct dfl_fpga_port_info.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#223-mmio-region-information","title":"2.2.3 MMIO Region Information","text":"The DFL_FPGA_PORT_GET_REGION_INFO ioctl is used to query the details of an MMIO region. The ioctl requires a pointer to a struct dfl_fpga_port_region_info. The index field of the struct is populated by the caller, and the padding, size, and offset values are populated by the DFL driver.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#224-shared-memory-mapping-and-unmapping","title":"2.2.4 Shared Memory Mapping and Unmapping","text":"The DFL_FPGA_PORT_DMA_MAP ioctl is used to map a memory buffer into the application\u2019s process address space. The ioctl requires a pointer to a struct dfl_fpga_port_dma_map.
The DFL_FPGA_PORT_DMA_UNMAP ioctl is used to unmap a memory buffer from the application\u2019s process address space. The ioctl requires a pointer to a struct dfl_fpga_port_dma_unmap.
These ioctls provide the underpinnings of the fpgaPrepareBuffer() and fpgaReleaseBuffer() calls.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#225-number-of-port-error-irqs","title":"2.2.5 Number of Port Error IRQs","text":"The DFL_FPGA_PORT_ERR_GET_IRQ_NUM ioctl is used to query the number of Port error interrupt vectors available. The ioctl requires a pointer to a uint32_t that receives the Port error interrupt count.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#226-port-error-interrupt-configuration","title":"2.2.6 Port Error Interrupt Configuration","text":"The DFL_FPGA_PORT_ERR_SET_IRQ ioctl is used to configure one or more file descriptors for the Port Error interrupt. The ioctl requires a pointer to a struct dfl_fpga_irq_set. The values stored in the evtfds field of this struct should be populated with the event file descriptors for the interrupt, as returned by the eventfd() C standard library API.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#227-number-of-afu-interrupts","title":"2.2.7 Number of AFU Interrupts","text":"The DFL_FPGA_PORT_UINT_GET_IRQ_NUM ioctl is used to query the number of AFU interrupt vectors available. The ioctl requires a pointer to a uint32_t that receives the AFU interrupt count.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#228-user-afu-interrupt-configuration","title":"2.2.8 User AFU Interrupt Configuration","text":"The DFL_FPGA_PORT_UINT_SET_IRQ ioctl is used to configure one or more file descriptors for the AFU interrupt. The ioctl requires a pointer to a struct dfl_fpga_irq_set. The values stored in the evtfds field of this struct should be populated with the event file descriptors for the interrupt, as returned by the eventfd() C standard library API.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#229-partial-reconfiguration","title":"2.2.9 Partial Reconfiguration","text":"The DFL_FPGA_FME_PORT_PR ioctl is used to update the logic stored in the Port\u2019s programmable region. This ioctl must be issued on the device file descriptor corresponding to the FPGA_DEVICE (/dev/dfl-fme.X). The ioctl requires a pointer to a struct dfl_fpga_fme_port_pr with each of the fields populated.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2210-number-of-fme-error-irqs","title":"2.2.10 Number of FME Error IRQs","text":"The DFL_FPGA_FME_ERR_GET_IRQ_NUM ioctl is used to query the number of FME error interrupt vectors available. The ioctl requires a pointer to a uint32_t that receives the FME error interrupt count.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2211-fme-error-interrupt-configuration","title":"2.2.11 FME Error Interrupt Configuration","text":"The DFL_FPGA_FME_ERR_SET_IRQ ioctl is used to configure one or more file descriptors for the FME Error interrupt. The ioctl requires a pointer to a struct dfl_fpga_irq_set. The values stored in the evtfds field of this struct should be populated with the event file descriptors for the interrupt, as returned by the eventfd() C standard library API. as returned by the eventfd() C standard library API.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#23-plugin-manager","title":"2.3 Plugin Manager","text":"The OPAE Plugin Manager refers to initialization code in libopae-c that examines an FPGA device\u2019s PCIe Vendor and Device ID and makes an association between a particular FPGA device and its access method. OPAE currently supports three device access methods:
Access Method
Plugin Module
Device Feature List drivers libxfpga.so Virtual Function I/O libopae-v.so AFU Simulation Environment libase.soTable 9 Plugin Device Access Methods
The Plugin Manager allows code that is written to a specific API signature to access FPGA hardware via different mechanisms. In other words, the end user codes to the OPAE API; and the OPAE API, based on configuration data, routes the hardware access to the device via different means.
As an example, consider an API configuration that accesses FPGA device_A via the Device Feature List drivers and that accesses FPGA device_B via VFIO. The application is coded against the OPAE API.
As part of its initialization process, the application enumerates and discovers an fpga_token corresponding to device_A. That fpga_token is opened and its MMIO region 0 is mapped via a call to fpgaMapMMIO().
The API configuration for device_A is such that the fpga_handle corresponding to device_A routes its hardware access calls through libxfpga.so. The call to fpgaMapMMIO() is redirected to libxfpga.so\u2019s implementation of the MMIO mapping function, xfpga_fpgaMapMMIO(). As a result, the call to xfpga_fpgaMapMMIO() uses its AFU file descriptor to communicate with the DFL driver to map the MMIO region.
Subsequently, the application enumerates and discovers an fpga_token corresponding to device_B. That fpga_token is opened and its MMIO region 0 is mapped via a call to fpgaMapMMIO().
The API configuration for device_B is such that the fpga_handle corresponding to device_B routes its hardware access calls through libopae-v.so. The call to fpgaMapMMIO() is redirected to libopae-v.so\u2019s implementation of the MMIO mapping function, vfio_fpgaMapMMIO(). As a result, the call to vfio_fpgaMapMMIO() uses the MMIO mapping performed by libopaevfio.so during initialization of the VFIO session.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#231-plugin-model","title":"2.3.1 Plugin Model","text":"The OPAE SDK plugin model is facilitated by its use of opaque C structures for fpga_token and fpga_handle. These types are both declared as void *; and this allows the parameters to the OPAE SDK functions to take different forms, depending on the layer of the SDK being used.
At the topmost layer, for example when calling fpgaEnumerate(), the output fpga_token parameter array is actually an array of pointers to opae_wrapped_token struct\u2019s.
typedef struct _opae_wrapped_token {\nuint32_t magic;\nfpga_token opae_token;\nuint32_t ref_count;\nstruct _opae_wrapped_token *prev;\nstruct _opae_wrapped_token *next;\nopae_api_adapter_table *adapter_table;\n} opae_wrapped_token;\nFigure 27 opae_wrapped_token
An opae_wrapped_token, as the name suggests, is a thin wrapper around the lower-layer token which is stored in struct member opae_token. The adapter_table struct member is a pointer to a plugin-specific adapter table. The adapter table provides a mapping between the top-layer opae_wrapped_token and its underlying plugin-specific API entry points, which are called using the opae_token struct member (the lower-level token).
typedef struct _opae_api_adapter_table {\nstruct _opae_api_adapter_table *next;\nopae_plugin plugin;\n...\nfpga_result (*fpgaEnumerate)(const fpga_properties *filters,\nuint32_t num_filters,\nfpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\nint (*initialize)(void);\nint (*finalize)(void);\n} opae_api_adapter_table;\nFigure 28 opae_api_adapter_table
When libopae-c loads, the plugin manager uses the plugin configuration data to open and configure a session to each of the required plugin libraries. During this configuration process, each plugin is passed an empty adapter table struct. The purpose of the plugin configuration is to populate this adapter table struct with each of the plugin-specific API entry points.
When the top-level fpgaEnumerate() is called, each adapter table\u2019s plugin-specific fpgaEnumerate() struct member is called; and the output fpga_token\u2019s are collected. At this point, these fpga_token\u2019s are the lower-level token structure types. Before the top-level fpgaEnumerate() returns, these plugin-specific tokens are wrapped inside opae_wrapped_token structures, along with a pointer to the respective adapter table.
After enumeration is complete, the application goes on to call other top-level OPAE SDK functions with the wrapped tokens. Each top-level entry point which accepts an fpga_token knows that it is actually being passed an opae_wrapped_token. With this knowledge, the entry point peeks inside the wrapped token and calls through to the plugin-specific API entry point using the adapter table, passing the lower-level opae_token struct member.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#232-libxfpga-plugin","title":"2.3.2 libxfpga Plugin","text":"2.3.1 Plugin Model introduced the concept of an opae_wrapped_token and a corresponding plugin-specific token structure. libxfpga.so is the plugin library that implements the DFL driver hardware access method. Its plugin-specific token data structure is struct _fpga_token.
struct _fpga_token {\nfpga_token_header hdr;\nuint32_t device_instance;\nuint32_t subdev_instance;\nchar sysfspath[SYSFS_PATH_MAX];\nchar devpath[DEV_PATH_MAX];\nstruct error_list *errors;\n};\nFigure 29 struct _fpga_token
A struct _fpga_token corresponding to the Port will have sysfspath and devpath members that contain strings like the following example paths:
sysfspath: \u201c/sys/class/fpga_region/region0/dfl-port.0\u201d\ndevpath: \u201c/dev/dfl-port.0\u201d\nFigure 30 libxfpga Port Token
Likewise, a struct _fpga_token corresponding to the FME will have sysfspath and devpath members that contain strings like the following example paths:
sysfspath: \u201c/sys/class/fpga_region/region0/dfl-fme.0\u201d\ndevpath: \u201c/dev/dfl-fme.0\u201d\nFigure 31 libxfpga FME Token
When a call to the top-level fpgaOpen() is made, the lower-level token is unwrapped and passed to xfpga_fpgaOpen(). In return, xfpga_fpgaOpen() opens the character device file identified by the devpath member of the struct _fpga_token. It then allocates and initializes an instance of libxfpga.so\u2019s plugin-specific handle data structure, struct _fpga_handle.
struct _fpga_handle {\npthread_mutex_t lock;\nuint64_t magic;\nfpga_token token;\nint fddev;\nint fdfpgad;\nuint32_t num_irqs;\nuint32_t irq_set;\nstruct wsid_tracker *wsid_root;\nstruct wsid_tracker *mmio_root;\nvoid *umsg_virt;\nuint64_t umsg_size;\nuint64_t *umsg_iova;\nbool metric_enum_status;\nfpga_metric_vector fpga_enum_metric_vector;\nvoid *bmc_handle;\nstruct _fpga_bmc_metric *_bmc_metric_cache_value;\nuint64_t num_bmc_metric;\nuint32_t flags;\n};\nFigure 32 struct _fpga_handle
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#233-libopae-v-plugin","title":"2.3.3 libopae-v Plugin","text":"libopae-v.so is the plugin library that implements the VFIO hardware access method. Its plugin-specific token data structure is vfio_token.
#define USER_MMIO_MAX 8\ntypedef struct _vfio_token {\nfpga_token_header hdr;\nfpga_guid compat_id;\npci_device_t *device;\nuint32_t region;\nuint32_t offset;\nuint32_t mmio_size;\nuint32_t pr_control;\nuint32_t user_mmio_count;\nuint32_t user_mmio[USER_MMIO_MAX];\nuint64_t bitstream_id;\nuint64_t bitstream_mdata;\nuint8_t num_ports;\nstruct _vfio_token *parent;\nstruct _vfio_token *next;\nvfio_ops ops;\n} vfio_token;\nFigure 33 vfio_token
When a call to the top-level fpgaOpen() is made, the lower-level token is unwrapped and passed to vfio_fpgaOpen(). In return, vfio_fpgaOpen() opens the VFIO device matching the device address found in the input vfio_token. It then allocates and initializes an instance of libopae-v.so\u2019s plugin-specific handle data structure, vfio_handle.
typedef struct _vfio_handle {\nuint32_t magic;\nstruct _vfio_token *token;\nvfio_pair_t *vfio_pair;\nvolatile uint8_t *mmio_base;\nsize_t mmio_size;\npthread_mutex_t lock;\n#define OPAE_FLAG_HAS_AVX512 (1u << 0)\nuint32_t flags;\n} vfio_handle;\nFigure 34 vfio_handle
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2331-supporting-libraries","title":"2.3.3.1 Supporting Libraries","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#23311-libopaevfio","title":"2.3.3.1.1 libopaevfio","text":"libopaevfio.so is OPAE\u2019s implementation of the Linux kernel\u2019s Virtual Function I/O interface. This VFIO interface presents a generic means of configuring and accessing PCIe endpoints from a user-space process via a supporting Linux kernel device driver, vfio-pci.
libopaevfio.so provides APIs for opening/closing a VFIO device instance, for mapping/unmapping MMIO spaces, for allocating/freeing DMA buffers, and for configuring interrupts for the device.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#23312-libopaemem","title":"2.3.3.1.2 libopaemem","text":"Each DMA buffer allocation made by libopaevfio.so\u2019s opae_vfio_buffer_allocate() and opae_vfio_buffer_allocate_ex() APIs requires a backing I/O Virtual Address range. These address ranges are discovered at VFIO device open time by way of the VFIO_IOMMU_GET_INFO ioctl.
Each range specifies a large contiguous block of I/O Virtual Address space. The typical DMA buffer allocation size is significantly less than one of these IOVA blocks, so the division of each block into allocatable segments must be managed so that multiple DMA buffer allocations can be made from a single block. In other words, the IOVA blocks must be memory-managed in order to make efficient use of them.
libopaemem.so provides a generic means of managing a large memory space, consisting of individual large memory blocks of contiguous address space. When a DMA buffer allocation is requested, libopaevfio.so uses this generic memory manager to carve out a small chunk of contiguous IOVA address space in order for the DMA mapping to be made. The IOVA space corresponding to the allocation is marked as allocated, and the rest of the large block remains as allocatable space within the memory manager. Subsequent de-allocation returns a chunk of IOVA space to the free state, coalescing contiguous chunks as they are freed. The allocations and de-allocations of the IOVA space can occur in any order with respect to each other. libopaemem.so tracks both the allocated and free block space, carving out small chunks from the large IOVA blocks on allocations, and coalescing small chunks back into larger ones on frees.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2332-configuring-pcie-virtual-functions","title":"2.3.3.2 Configuring PCIe Virtual Functions","text":"Before an AFU can be accessed with VFIO, the FPGA Physical Function must be configured to enable its Virtual Functions. Then, each VF must be bound to the vfio-pci Linux kernel driver.
As of the Arrow Creek program, the FPGA hardware allows multiple AFU\u2019s to co-exist by placing each AFU in its own PCIe Virtual Function. Upon system startup, no PCIe VF\u2019s exist. The pci_device command can be used to enable the VF\u2019s and their AFU\u2019s. First, use the lspci command to examine the current device topology:
# lspci | grep cel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.3 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\nFigure 35 lspci Device Topology
In this example, VF\u2019s are controlled by PF 0, as highlighted in Figure 35 lspci Device Topology. In the figure, each PF is shown as having the Arrow Creek PF PCIe device ID of bcce.
Now, use the pci_device command to enable three VF\u2019s for PF0:
# pci_device 0000:b1:00.0 vf 3\n# lspci | grep cel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\nb1:00.5 Processing accelerators: Intel Corporation Device bccf\nb1:00.6 Processing accelerators: Intel Corporation Device bccf\nb1:00.7 Processing accelerators: Intel Corporation Device bccf\nFigure 36 Enable Virtual Functions
Figure 20 Enable Virtual Functions shows that three VF\u2019s were created. Each VF is shown as having the Arrow Creek VF PCIe device ID of bccf.
Now, each Virtual Function must be bound to the vfio-pci Linux kernel driver so that it can be accessed via VFIO:
# opaevfio -i -u myuser -g mygroup 0000:b1:00.5\nBinding (0x8086,0xbccf) at 0000:b1:00.5 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.5 is 318\nFigure 37 Bind VF's to vfio-pci
Here, myuser and mygroup identify the unprivileged user/group that requires access to the device. The opaevfio command will change the ownership of the device per the values given.
Once the VF\u2019s are bound to vfio-pci, the OPAE SDK will find and enumerate them with libopae-v.so:
# fpgainfo port\n//****** PORT ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:B1:00.0\nDevice Id : 0xBCCE\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nDevice Id : 0xBCCF\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nDevice Id : 0xBCCF\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nDevice Id : 0xBCCF\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\nFigure 38 List VF's with fpgainfo
When the VF\u2019s are no longer needed, they can be unbound from the vfio-pci driver:
# opaevfio -r 0000:b1:00.5\nReleasing (0x8086,0xbccf) at 0000:b1:00.5 from vfio-pci\n# opaevfio -r 0000:b1:00.6\nReleasing (0x8086,0xbccf) at 0000:b1:00.6 from vfio-pci\n# opaevfio -r 0000:b1:00.7\nReleasing (0x8086,0xbccf) at 0000:b1:00.7 from vfio-pci\nFigure 39 Unbind VF's from vfio-pci
Finally, the VF\u2019s can be disabled:
# pci_device 0000:b1:00.0 vf 0\n# lspci | grep cel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\nFigure 40 Disable Virtual Functions
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#24-application-flow","title":"2.4 Application Flow","text":"A typical OPAE application that interacts with an AFU via MMIO and shared memory will have a flow similar to the one described in this section.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#241-create-filter-criteria","title":"2.4.1 Create Filter Criteria","text":"Refer to 2.1.2 Enumeration. When enumerating AFU\u2019s, if no filtering criteria is specified, then fpgaEnumerate() returns fpga_token\u2019s for each AFU that is present in the system. In order to limit the enumeration search to a specific AFU, create an fpga_properties object and set its guid to that of the desired AFU:
#define MY_AFU_GUID \u201c57fa0b03-ab4f-4b02-b4eb-d3fe1ec18518\u201d\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpgaGetProperties(NULL, &filter);\nuuid_parse(MY_AFU_GUID, guid);\nfpgaPropertiesSetGUID(filter, guid);\nFigure 41 Flow: Create Filter Criteria
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#242-enumerate-the-afu","title":"2.4.2 Enumerate the AFU","text":"With the filtering criteria in place, enumerate to obtain an fpga_token for the AFU:
fpga_token afu_token = NULL;\nuint32_t num_matches = 0;\nfpgaEnumerate(&filter, 1, &afu_token, 1, &num_matches);\nFigure 42 Flow: Enumerate the AFU
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#243-open-the-afu","title":"2.4.3 Open the AFU","text":"After finding an fpga_token for the AFU using fpgaEnumerate(), the token must be opened with fpgaOpen() to establish a session with the AFU. The process of opening an fpga_token creates an fpga_handle:
fpga_handle afu_handle = NULL;\nfpgaOpen(afu_token, &afu_handle, 0);\nFigure 43 Flow: Open the AFU
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#244-map-mmio-region","title":"2.4.4 Map MMIO Region","text":"In order to access the MMIO region of the AFU to program its CSR\u2019s, the region must first be mapped into the application\u2019s process address space. This is accomplished using fpgaMapMMIO():
uint32_t region = 0;\nfpgaMapMMIO(afu_handle, region, NULL);\nFigure 44 Flow: Map MMIO Region
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#245-allocate-dma-buffers","title":"2.4.5 Allocate DMA Buffers","text":"If the AFU is DMA-capable, shared memory buffers can be allocated and mapped into the process address space and the IOMMU with fpgaPrepareBuffer(). Refer to Figure 8 Configuring Huge Pages for instructions on configuring 2MiB and 1GiB huge pages.
#define BUF_SIZE (2 * 1024 * 1024)\nvolatile uint64_t *src_ptr = NULL;\nuint64_t src_wsid = 0;\nvolatile uint64_t *dest_ptr = NULL;\nuint64_t dest_wsid = 0;\nfpgaPrepareBuffer(afu_handle, BUF_SIZE, (void **)&src_ptr,\n&src_wsid, 0);\nfpgaPrepareBuffer(afu_handle, BUF_SIZE, (void **)&dest_ptr,\n&dest_wsid, 0);\nmemset(src_ptr, 0xaf, BUF_SIZE);\nmemset(dest_ptr, 0xbe, BUF_SIZE);\nFigure 45 Flow: Allocate DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#246-make-afu-aware-of-dma-buffers","title":"2.4.6 Make AFU Aware of DMA Buffers","text":"The process by which locations of shared memory buffers and their sizes are made known to the AFU is entirely AFU-specific. This example shows the method used by the Native Loopback AFU. Each buffer I/O virtual address is cacheline-aligned and programmed into a unique AFU CSR; then the buffer size in lines is programmed into a length CSR:
#define CSR_SRC_ADDR 0x000A // AFU-specific\n#define CSR_DEST_ADDR 0x000B // AFU-specific\n#define CSR_NUM_LINES 0x000C // AFU-specific\nuint64_t src_iova = 0;\nuint64_t dest_iova = 0;\nfpgaGetIOAddress(afu_handle, src_wsid, &src_iova);\nfpgaGetIOAddress(afu_handle, dest_wsid, &dest_iova);\nfpgaWriteMMIO64(afu_handle, 0, CSR_SRC_ADDR, src_iova >> 6);\nfpgaWriteMMIO64(afu_handle, 0, CSR_DEST_ADDR, dest_iova >> 6);\nfpgaWriteMMIO32(afu_handle, 0, CSR_NUM_LINES, BUF_SIZE / 64);\nFigure 46 Flow: Make AFU Aware of DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#247-initiate-an-acceleration-task","title":"2.4.7 Initiate an Acceleration Task","text":"With the shared buffer configuration complete, the AFU can be told to initiate the acceleration task. This process is AFU-specific. The Native Loopback AFU starts the acceleration task by writing a value to its control CSR:
#define CSR_CTRL 0x000D // AFU-specific\nfpgaWriteMMIO32(afu_handle, 0, CSR_CTRL, 3);\nFigure 47 Initiate an Acceleration Task
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#248-wait-for-task-completion","title":"2.4.8 Wait for Task Completion","text":"Once the acceleration task is initiated, the application may poll the AFU for a completion status. This process is AFU-specific. The AFU may provide a status CSR for the application to poll; or the AFU may communicate status to the application by means of a result code written to a shared buffer.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#249-free-dma-buffers","title":"2.4.9 Free DMA Buffers","text":"When the acceleration task completes and the AFU is quiesced such that there are no outstanding memory transactions targeted for the shared memory, the DMA buffers can be unmapped and freed using fpgaReleaseBuffer():
fpgaReleaseBuffer(afu_handle, src_wsid);\nfpgaReleaseBuffer(afu_handle, dest_wsid);\nFigure 48 Flow: Free DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2410-unmap-mmio-region","title":"2.4.10 Unmap MMIO Region","text":"The MMIO regions should also be unmapped using fpgaUnmapMMIO():
fpgaUnmapMMIO(afu_handle, region);\nFigure 49 Flow: Unmap MMIO Region
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2411-close-the-afu","title":"2.4.11 Close the AFU","text":"The AFU handle should be closed via fpgaClose() to release its resources:
fpgaClose(afu_handle);\nThe fpga_token\u2019s returned by fpgaEnumerate() should be destroyed using the fpgaDestroyToken() API. The fpga_properties objects should be destroyed using the fpgaDestroyProperties() API:
fpgaDestroyToken(&afu_token);\nfpgaDestroyProperties(&filter);\nFigure 51 Flow: Release the Tokens and Properties
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#30-opae-c-api","title":"3.0 OPAE C++ API","text":"The OPAE C++ API refers to a C++ layer that sits on top of the OPAE C API, providing object-oriented implementations of the main OPAE C API abstractions: properties, tokens, handles, dma buffers, etc. Like the OPAE C API, the C++ API headers contain Doxygen markup for each of the provided classes.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#31-libopae-cxx-core","title":"3.1 libopae-cxx-core","text":"The implementation files for the C++ API are compiled into libopae-cxx-core.so. A convenience header, core.h, provides a quick means of including each of the C++ API headers. Each of the types comprising the C++ API is located within the opae::fpga::types C++ namespace.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#311-properties","title":"3.1.1 Properties","text":"Class properties provides the C++ implementation of the fpga_properties type and its associated APIs.
properties::ptr_t filter = properties::get();\nFigure 52 C++ Create New Empty Properties
Class properties provides member variables for each fpga_properties item that can be manipulated with fpgaPropertiesGet\u2026() and fpgaPropertiesSet\u2026(). For example, to set the AFU ID in a properties instance and to set that instance\u2019s type to FPGA_ACCELERATOR:
#define MY_AFU_ID \u201c8ad74241-d13b-48eb-b428-7986dcbcab14\u201d\nfilter->guid.parse(MY_AFU_ID);\nfilter->type = FPGA_ACCELERATOR;\nFigure 53 C++ Properties Set GUID and Type
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#312-tokens","title":"3.1.2 Tokens","text":"Class token provides the C++ implementation of the fpga_token type and its associated APIs. Class token also provides the enumerate() static member function:
std::vector<token::ptr_t> tokens = token::enumerate({filter});\nif (tokens.size() < 1) {\n// flag error and return\n}\ntoken::ptr_t tok = tokens[0];\nFigure 54 C++ Enumeration
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#313-handles","title":"3.1.3 Handles","text":"Class handle provides the C++ implementation of the fpga_handle type and its associated APIs. The handle class provides member functions for opening and closing a token, for reading and writing to MMIO space, and for reconfiguring the FPGA\u2019s Programmable Region.
handle::ptr_t accel = handle::open(tok, 0);\nFigure 55 C++ Opening a Handle
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#314-shared-memory","title":"3.1.4 Shared Memory","text":"The shared_buffer class provides member functions for allocating and releasing DMA buffers, for querying buffer attributes, and for reading and writing buffers.
#define BUF_SIZE (2 * 1024 * 1024)\nshared_buffer::ptr_t input = shared_buffer::allocate(accel, BUF_SIZE);\nshared_buffer::ptr_t output = shared_buffer::allocate(accel, BUF_SIZE);\nstd::fill_n(input->c_type(), BUF_SIZE, 0xaf);\nstd::fill_n(output->c_type(), BUF_SIZE, 0xbe);\nFigure 56 C++ Allocate and Init Buffers
Once DMA buffers have been allocated, their IO addresses are programmed into AFU-specific CSRs to enable the DMA. Here, the IO address of each buffer is aligned to the nearest cache line before programming it into the AFU CSR space. The number of cache lines is then programmed into the appropriate AFU CSR.
#define CSR_SRC_ADDR 0x000A // AFU-specific\n#define CSR_DEST_ADDR 0x000B // AFU-specific\n#define CSR_NUM_LINES 0x000C // AFU-specific\n#define LOG2_CL 6\naccel->write_csr64(CSR_SRC_ADDR, input->io_address() >> LOG2_CL);\naccel->write_csr64(CSR_DEST_ADDR, output->io_address() >> LOG2_CL);\naccel->write_csr32(CSR_NUM_LINES, BUF_SIZE / 64);\nFigure 57 C++ Make the AFU Aware of DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#315-events","title":"3.1.5 Events","text":"The event class provides member functions for event registration. In order to register an event, provide the handle::ptr_t for the desired device, along with the event type and optional flags.
int vect = 2;\nevent::ptr_t evt = event::register_event(accel,\nFPGA_EVENT_INTERRUPT,\nvect);\nint evt_fd = evt.os_object();\nFigure 58 C++ Event Registration
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#316-errors","title":"3.1.6 Errors","text":"Class error provides a means of querying the device errors given a token::ptr_t. The token and integer ID provided to the error::get() static member function uniquely identify one of the 64-bit error masks associated with the token.
error::ptr_t err = error::get(tok, 0); Figure 59 C++ Query Device Errors
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#317-sysobject","title":"3.1.7 SysObject","text":"Class sysobject is the C++ implementation of the OPAE SysObject API. sysobject provides a means of creating class instances via its two sysobject::get() static member functions. A third non-static sysobject::get() enables creating a sysobject instance given a parent sysobject instance. The read64() and write64() member functions allow reading and writing the sysobject\u2019s value as a 64-bit unsigned integer. The bytes() member functions allow reading a sysobject\u2019s value as a raw byte stream.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#40-opae-python-api","title":"4.0 OPAE Python API","text":"The OPAE Python API is built on top of the OPAE C++ Core API and its object model. Because of this, developing OPAE applications in Python is very similar to developing OPAE applications in C++ which significantly reduces the learning curve required to adapt to the Python API. While the object model remains the same, some static factory functions in the OPAE C++ Core API have been moved to module level methods in the OPAE Python API with the exception of the properties class. The goal of the OPAE Python API is to enable fast prototyping, test automation, infrastructure managment, and an easy to use framework for FPGA resource interactions that don\u2019t rely on software algorithms with a high runtime complexity.
The major benefits of using pybind11 for developing the OPAE Python API include, but are not limited to, the following features of pybind11:
Currently, the only Python package that is part of OPAE is opae.fpga. Because opae.fpga is built on top of the opae-cxx-core API, it does require that the runtime libraries for both opae-cxx-core and opae-c be installed on the system (as well as any other libraries they depend on). Those libraries can be installed using the opae-libs package (from either RPM or DEB format - depending on your Linux distribution). Installation for the Python OPAE bindings are included in the Getting Started Guide for your platform.
The Python API is coded as a pybind11 project, which allows C++ code to directly interface with Python internals. Each C++ API concept is encoded into a Python equivalent. The functionality exists as a Python extension module, compiled into _opae.so.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#411-enumeration","title":"4.1.1 Enumeration","text":"Enumeration is somewhat simplified as compared to the OPAE C/C++ APIs. The fpga.enumerate() function accepts keyword arguments for each of the property names that are defined in the C++ API. As an example, to enumerate for an FPGA_ACCELERATOR by its GUID:
from opae import fpga\nMY_ACCEL = \u201cd573b29e-176f-4cb7-b810-efbf7be34cc9\u201d\ntokens = fpga.enumerate(type=fpga.ACCELERATOR, guid=MY_ACCEL)\nassert tokens, \u201cNo accelerator matches {}\u201d.format(MY_ACCEL)\nFigure 60 Python Enumeration
The return value from the fpga.enumerate() function is a list of all the token objects matching the search criteria.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#412-properties","title":"4.1.2 Properties","text":"Querying properties for a token or handle is also a bit different in the Python API. In order to query properties for one of these objects, pass the object to the fpga.properties() constructor. The return value is a properties object with each of the property names defined as instance attributes.
prop = fpga.properties(tokens[0])\nprint(f'0x{prop.vendor_id:04x} 0x{prop.device_id:04x}')\nFigure 61 Python Get Token Properties
Properties objects may also be created by invoking the fpga.properties() constructor, passing the same keyword arguments as those to fpga.enumerate(). Properties objects created in this way are also useful for enumeration purposes:
props = fpga.properties(type=fpga.ACCELERATOR)\ntokens = fpga.enumerate([props])\nFigure 62 Python Properties Constructor
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#413-tokens","title":"4.1.3 Tokens","text":"Tokens overload both the __getitem__ and __getattr__ methods to enable the SysObject API. Both of the following are valid forms of accessing the \u2018errors/first_error\u2019 sysfs attribute, given a token object:
tok = tokens[0]\nferr = tok['errors/first_error']\nprint(f'first error: 0x{ferr.read64():0x}')\nprint('0x{:0x}'.format(tok.errors.first_error.read64()))\nFigure 63 Python Tokens and SysObject API
Tokens also implement a find() method, which accepts a glob expression in order to search sysfs. The following example finds the \u201cid\u201d sysfs entry in the given token\u2019s sysfs tree.
my_id = tok.find(\u2018i?\u2019)\nprint(f'{my_id.read64()}')\nFigure 64 Python Token Find
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#414-handles","title":"4.1.4 Handles","text":"Tokens are converted to handles by way of the fpga.open() function. The flags (second) parameter to fpga.open() may be zero or fpga.OPEN_SHARED.
with fpga.open(tok, fpga.OPEN_SHARED) as handle:\nuse_the_handle(handle)\nFigure 65 Python Open Handle
Like token objects, handle objects overload __getitem__ and __getattr__ methods to enable the SysObject API. handle also provides a find() method similar to token\u2019s find().
err = handle['errors/errors']\nprint(f'errors: 0x{err.read64():0x}')\nprint('first error: 0x{:0x}'.format(\nhandle.errors.first_error.read64()))\nmy_id = handle.find('i?')\nprint(f'{my_id.read64()}')\nFigure 66 Python Handles and SysObject API
Partial reconfiguration is provided by class handle\u2019s reconfigure() method. The first parameter, slot, will be zero in most designs. The second parameter is an opened file descriptor to the file containing the GBS image. The third parameter, flags, defaults to zero.
with open(\u2018my.gbs\u2019, \u2018rb\u2019) as fd:\nhandle.reconfigure(0, fd)\nFigure 67 Python Partial Reconfiguration
Device reset is accomplished by means of handle\u2019s reset() method, which takes no parameters.
Finally for handles, CSR space reads are accomplished via read_csr32() and read_csr64(). Both methods accept the register offset as the first parameter and an optional csr_space index, which defaults to zero, as the second parameter. CSR space writes are accomplished via write_csr32() and write_csr64(). Both methods accept the register offset as the first parameter, the value to write as the second, and an optional csr_space index, which defaults to zero, as the third.
print(\u20190x{:0x}\u2019.format(handle.read_csr32(0x000a)))\nprint(\u20180x{:0x}\u2019.format(handle.read_csr64(0x000c)))\nhandle.write_csr32(0x000b, 0xdecafbad, 2)\nhandle.write_csr64(0x000e, 0xc0cac01adecafbad, 2)\nFigure 68 Python Read/Write CSR
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#415-shared-memory","title":"4.1.5 Shared Memory","text":"The fpga.allocate_shared_buffer() function provides access to the OPAE memory allocator. The allocation sizes and required huge page configurations are the same as those noted in 2.1.5 MMIO and Shared Memory.
The fpga.allocate_shared_buffer() function returns an object instance of type shared_buffer. The shared_buffer class implements methods size(), wsid(), and io_address(), which return the buffer size in bytes, the unique workspace ID, and the IO address respectively:
buf = fpga.allocate_shared_buffer(handle, 4096)\nprint(f\u2019size: {buf.size()}\u2019)\nprint(f\u2019wsid: 0x{buf.wsid():0x}\u2019)\nprint(f\u2019io_address: 0x{buf.io_address():0x}\u2019)\nFigure 69 Python Allocate Shared Memory
The shared_buffer class implements a fill() method which takes an integer parameter which is applied to each byte of the buffer (similar to C standard library\u2019s memset()). The compare() method compares the contents of the first size bytes of one buffer to another. The value returned from compare() is the same as the C standard library\u2019s memcmp(). The copy() method copies the first size bytes of the calling buffer into the argument buffer.
b0 = fpga.allocate_shared_buffer(handle, 4096)\nb1 = fpga.allocate_shared_buffer(handle, 4096)\nb0.fill(0xa5)\nb1.fill(0xa5)\nprint(f'compare: {b0.compare(b1, 4096)}')\nb1.fill(0xa0)\nb0.copy(b1, 4096)\nprint(f'compare: {b0.compare(b1, 4096)}')\nFigure 70 Python Buffer Fill, Copy, Compare
shared_buffer\u2019s read32() and read64() methods read a 32- or 64-bit value from the given offset. The write32() and write64() methods write a 32- or 64-bit value to the given offset.
print(f'value at 0: 0x{b0.read32(0):0x}')\nprint(f'value at 4: 0x{b0.read64(4):0x}')\nb0.write32(0xabadbeef, 0)\nb0.write64(0xdecafbadabadbeef, 4)\nprint(f'value at 0: 0x{b0.read32(0):0x}')\nprint(f'value at 4: 0x{b0.read64(4):0x}')\nFigure 71 Python Buffer Read and Write
The shared_buffer class provides three polling methods: poll(), poll32(), and poll64(). Each method takes an offset as its first parameter. The second parameter is a value and the third is a mask. The value and mask parameters are 8 bits wide for poll(), 32 bits wide for poll32(), and 64 bits wide for poll64(). The fourth and last parameter is a timeout value which defaults to 1000 microseconds.
Each polling method reads the n-bit wide item at offset and applies (logical AND) the mask to that value. The masked value created in the previous step is then compared to the second parameter, value. If the two values are equal, then the method returns true immediately. Otherwise, the method continues to loop, attempting the same comparison over and over without sleeping. Finally, if the elapsed time from the beginning of the call to the current time is greater than or equal to the timeout value, then the method times out and returns false.
if b0.poll32(0, 0xbebebebe, 0xffffffff, 250):\nprint(\u2018Got it!\u2019)\nFigure 72 Python Buffer Poll
The shared_buffer split() method allows creating two or more buffer objects from one larger buffer object. The return value is a list of shared_buffer instances whose sizes match the arguments given to split().
b1, b2 = b1.split(2048, 2048)\nprint(f'b1 io_address: 0x{b1.io_address():0x}')\nprint(f'b2 io_address: 0x{b2.io_address():0x}')\nFigure 73 Python Splitting Buffer
Finally, the shared_buffer class implements the Python buffer protocol to support memoryview objects. The Python buffer protocol allows access to an object\u2019s underlying memory without copying that memory. As a brief example:
mv = memoryview(b1)\nassert mv\nassert mv[0] == 0xbe\nb1[15] = int(65536)\nassert struct.unpack('<L', bytearray(b1[15:19]))[0] == 65536\nFigure 74 Python memoryview
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#416-events","title":"4.1.6 Events","text":"Given a handle and an event type, the fpga.register_event() function returns an object of type event. The event class implements one method, os_object(), which returns the underlying file descriptor that can be used to poll for the event:
import select\nevt = fpga.register_event(handle, fpga.EVENT_ERROR)\nos_object = evt.os_object()\nreceived_event = False\nepoll = select.epoll()\nepoll.register(os_object, select.EPOLLIN)\nfor fileno, ev in epoll.poll(1):\nif fileno == os_object:\nreceived_event = True\nprint(f'received: {received_event}')\nFigure 75 Python Events
In addition to fpga.EVENT_ERROR, fpga.EVENT_INTERRUPT, and fpga.EVENT_POWER_THERMAL are also supported.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#417-errors","title":"4.1.7 Errors","text":"Given a token, the fpga.errors() function returns a list of objects of type error. Each error instance represents a 64-bit mask of error values. The error bit masks are platform-dependent. Each error instance has two attributes: name and can_clear and one method: read_value() which returns the 64-bit error mask.
for e in fpga.errors(tok):\nprint(f\u2019name: \"{e.name}\"\u2019)\nprint(f\u2019can_clear: {e.can_clear}\u2019)\nprint(f\u2019value: {e.read_value()}\u2019)\nFigure 76 Python Get Errors
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#418-sysobject","title":"4.1.8 SysObject","text":"The Python API\u2019s SysObject implementation is introduced in 4.1.3 Tokens and 4.1.4 Handles. When the index operator (__getitem__) or attribute reference (__getattr__) is used and the referenced string or attribute name corresponds to a sysfs entry in the sysfs path of either a token or a handle, then an object of type sysobject is returned.
The size() method returns the length of the sysfs entry in bytes. Note that a typical sysfs entry is terminated with a \u2018\\n\u2019 followed by the \u2018\\0\u2019 NULL terminator. The bytes() method returns the sysfs entry\u2019s value as a string.
afu_id = tok['afu_id']\nassert afu_id\nprint(f'size: {afu_id.size()} bytes: {afu_id.bytes().rstrip()}')\nFigure 77 Python sysobject as Bytes
The sysobject read64() and write64() methods provide a means to read and write a sysfs entry\u2019s value as an unsigned 64-bit integer. The sysobject class itself also implements the __getitem__ and __getattr__ methods so that a sysobject of type FPGA_OBJECT_CONTAINER can retrieve sysobject instances for child sysfs entries.
errs = tok['errors']\nfirst = errs['first_error']\nassert first\nprint(f'first 0x{first.read64():0x}')\nThe following example is an implementation of the sample, hello_fpga.c, which is designed to configure the NLB0 diagnostic accelerator for a simple loopback.
import time\nfrom opae import fpga\nNLB0 = \"d8424dc4-a4a3-c413-f89e-433683f9040b\"\nCTL = 0x138\nCFG = 0x140\nNUM_LINES = 0x130\nSRC_ADDR = 0x0120\nDST_ADDR = 0x0128\nDSM_ADDR = 0x0110\nDSM_STATUS = 0x40\ndef cl_align(addr):\nreturn addr >> 6\ntokens = fpga.enumerate(type=fpga.ACCELERATOR, guid=NLB0)\nassert tokens, \"Could not enumerate accelerator: {}\".format(NlB0)\nwith fpga.open(tokens[0], fpga.OPEN_SHARED) as handle:\nsrc = fpga.allocate_shared_buffer(handle, 4096)\ndst = fpga.allocate_shared_buffer(handle, 4096)\ndsm = fpga.allocate_shared_buffer(handle, 4096)\nhandle.write_csr32(CTL, 0)\nhandle.write_csr32(CTL, 1)\nhandle.write_csr64(DSM_ADDR, dsm.io_address())\nhandle.write_csr64(SRC_ADDR, cl_align(src.io_address())) # cacheline-aligned\nhandle.write_csr64(DST_ADDR, cl_align(dst.io_address())) # cacheline-aligned\nhandle.write_csr32(CFG, 0x42000)\nhandle.write_csr32(NUM_LINES, 4096/64)\nhandle.write_csr32(CTL, 3)\nwhile dsm[DSM_STATUS] & 0x1 == 0:\ntime.sleep(0.001)\nhandle.write_csr32(CTL, 7)\nThis example shows how one might reprogram (Partial Reconfiguration) an accelerator on a given bus, 0x5e, using a bitstream file, m0.gbs.
from opae import fpga\nBUS = 0x5e\nGBS = 'm0.gbs'\ntokens = fpga.enumerate(type=fpga.DEVICE, bus=BUS)\nassert tokens, \"Could not enumerate device on bus: {}\".format(BUS)\nwith open(GBS, 'rb') as fd, fpga.open(tokens[0]) as device:\ndevice.reconfigure(0, fd)\nFigure 78 Python sysobject Container
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#50-management-interfaces-opaeadmin","title":"5.0 Management Interfaces - opae.admin","text":"While the OPAE SDK C, C++, and Python APIs focus on presenting the AFU and all its related functionality to the end user, there is also a need for a maintenance functionality to aid in configuring the platform and performing secure firmware updates for the FPGA device and its components. opae.admin is a Python framework which provides abstractions for performing these types of maintenance tasks on FPGA devices. opae.admin provides Python classes which model the FPGA and the sysfs interfaces provided by the DFL drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#51-sysfs","title":"5.1 sysfs","text":"opae.admin\u2019s sysfs module provides abstractions for interacting with sysfs nodes, which comprise the base entity abstraction of opae.admin.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#511-sysfs_node","title":"5.1.1 sysfs_node","text":"A sysfs_node is an object that tracks a unique path within a sysfs directory tree. sysfs_node provides methods for finding and constructing other sysfs_node objects, based on the root path of the parent sysfs_node object. sysfs_node also provides a mechanism to read and write sysfs file contents. sysfs_node serves as the base class for many of the sysfs module\u2019s other classes.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#512-pci_node","title":"5.1.2 pci_node","text":"A pci_node is a sysfs_node that is rooted at /sys/bus/pci/devices. Each pci_node has a unique PCIe address corresponding to the PCIe device it represents. Methods for finding the pci_node\u2019s children, for determining the PCIe device tree rooted at the node, for manipulating the node\u2019s PCIe address, for determining the vendor and device ID\u2019s, and for removing, unbinding, and rescanning the device are provided.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#513-sysfs_driver","title":"5.1.3 sysfs_driver","text":"A sysfs_driver is a sysfs_node that provides a method for unbinding a sysfs_device object.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#514-sysfs_device","title":"5.1.4 sysfs_device","text":"A sysfs_device is a sysfs_node that is located under /sys/class or /sys/bus. sysfs_device provides the basis for opae.admin\u2019s FPGA enumeration capability.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#515-pcie_device","title":"5.1.5 pcie_device","text":"A pcie_device is a sysfs_device that is rooted at /sys/bus/pci/devices.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#52-fpga","title":"5.2 fpga","text":"opae.admin\u2019s fpga module provides classes which abstract an FPGA and its components.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#521-region","title":"5.2.1 region","text":"A region is a sysfs_node that has an associated Linux character device, rooted at /dev. Methods for opening the region\u2019s character device file and for interacting with the character device via its IOCTL interface are provided.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#522-fme","title":"5.2.2 fme","text":"An fme is a region that represents an FPGA device\u2019s FME component. An fme provides accessors for the PR interface ID, the various bus paths that may exist under an FME, and the BMC firmware revision information.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#523-port","title":"5.2.3 port","text":"A port is a region that represents an FPGA device\u2019s Port component. A port provides an accessor for the Port AFU ID.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#524-fpga_base","title":"5.2.4 fpga_base","text":"An fpga_base is a sysfs_device that provides accessors for the FPGA device\u2019s FME, for the FPGA device\u2019s Port, and for the secure update sysfs controls. fpga_base provides routines for enabling and disabling AER and for performing device RSU.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#525-fpga","title":"5.2.5 fpga","text":"An fpga (derived from fpga_base) is the basis for representing the FPGA device in opae.admin. Utilities such as fpgasupdate rely on fpga\u2019s enum classmethod to enumerate all of the FPGA devices in the system. In order for a device to enumerate via this mechanism, it must be bound to the dfl-pci driver at the time of enumeration.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#53-opaeadmin-utilities","title":"5.3 opae.admin Utilities","text":"Several utilities are written on top of opae.admin\u2019s class abstractions. The following sections highlight some of the most commonly-used utilities.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#531-fpgasupdate","title":"5.3.1 fpgasupdate","text":"fpgasupdate, or FPGA Secure Update, is used to apply firmware updates to the components of the FPGA. As the name implies, these updates target a secure FPGA device, one that has the ability to implement a secure root of trust. The command-line interface to fpgasupdate was designed to be as simple as possible for the end user. The command simply takes a path to the firmware update file to be applied and the PCIe address of the targeted FPGA device.
# fpgasupdate update-file.bin 0000:b2:00.0\nFigure 79 fpgasupdate Interface
fpgasupdate can apply a variety of firmware image updates. | Image| Description| | -----| -----| |Programmable Region Image| .gbs or Green BitStream| |SR Root Key Hash| Static Region RKH| |PR Root Key Hash| Programmable Region RKH| |FPGA Firmware Image| Static Region Device Firmware| |PR Authentication Certificate| Programmable Region Auth Cert| |BMC Firmware Image| Board Management Controller Firmware| |SR Thermal Image| Static Region Thermal Sensor Thresholds| |PR Thermal Image| Programmable Region Thermal Sensor Thresholds| |CSK Cancelation| Code Signing Key Cancelation Request| |SDM Image| Secure Device Manager Firmware|
Table 10 fpgasupdate Image Types
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#532-pci_device","title":"5.3.2 pci_device","text":"pci_device is a utility that provides a convenient interface to some of the Linux Kernel\u2019s standard PCIe device capabilities.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5321-pci_device-aer-subcommand","title":"5.3.2.1 pci_device aer subcommand","text":"The aer dump subcommand displays the Correctable, Fatal, and NonFatal device errors.
# pci_device 0000:b2:00.0 aer dump\nFigure 80 pci_device aer dump
The aer mask subcommand displays, masks, or unmasks errors using the syntax of the setpci command.
# pci_device 0000:b2:00.0 aer mask show\n0x00010000 0x000031c1\n# pci_device 0000:b2:00.0 aer mask all\n# pci_device 0000:b2:00.0 aer mask off\n# pci_device 0000:b2:00.0 aer mask 0x01010101 0x10101010\nFigure 81 pci_device aer mask
The aer clear subcommand clears the current errors.
# pci_device 0000:b2:00.0 aer clear\naer clear errors: 00000000\nFigure 82 pci_device aer clear
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5322-pci_device-unbind-subcommand","title":"5.3.2.2 pci_device unbind subcommand","text":"The unbind subcommand unbinds the target device from the currently-bound device driver.
# pci_device 0000:b2:00.0 unbind\nFigure 83 pci_device unbind
In order to re-bind the device to a driver, eg dfl-pci, use the following commands:
# cd /sys/bus/pci/drivers/dfl-pci\n# echo 0000:b2:00.0 > bind\nFigure 84 Re-binding a Driver
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5323-pci_device-rescan-subcommand","title":"5.3.2.3 pci_device rescan subcommand","text":"The rescan subcommand triggers a PCIe bus rescan of all PCIe devices.
# pci_device 0000:b2:00.0 rescan\nFigure 85 pci_device rescan
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5324-pci_device-remove-subcommand","title":"5.3.2.4 pci_device remove subcommand","text":"The remove subcommand removes the target device from Linux kernel management.
# pci_device 0000:b2:00.0 remove\nFigure 86 pci_device remove
Note: a reboot may be required in order to re-establish the Linux kernel management for the device.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5325-pci_device-topology-subcommand","title":"5.3.2.5 pci_device topology subcommand","text":"The topology subcommand shows a tab-delimited depiction of the target device as it exists in the PCIe device tree in the Linux kernel.
# pci_device 0000:b2:00.0 topology\n[pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)] (pcieport)\n[pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:3c:09.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:b2:00.0), pci_id(0x8086, 0x0b30)] (dfl-pci)\n[pci_address(0000:3c:11.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:43:00.0), pci_id(0x8086, 0x0b32)] (no driver)\n[pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:3d:00.1), pci_id(0x8086, 0x0d58)] (i40e)\n[pci_address(0000:3d:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n[pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:41:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n[pci_address(0000:41:00.1), pci_id(0x8086, 0x0d58)] (i40e)\nFigure 87 pci_device topology
The green output indicates the target device. The other endpoint devices are shown in blue.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5326-pci_device-vf-subcommand","title":"5.3.2.6 pci_device vf subcommand","text":"The vf subcommand allows setting the value of the sriov_numvfs sysfs node of the target device. This is useful in scenarios where device functionality is presented in the form of one or more PCIe Virtual Functions.
# pci_device 0000:b2:00.0 vf 3\n# pci_device 0000:b2:00.0 vf 0\nFigure 88 pci_device vf
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#533-rsu","title":"5.3.3 rsu","text":"rsu is a utility that performs Remote System Update. rsu is used subsequent to programming a firmware update or other supported file type with fpgasupdate, in order to reset the targeted FPGA entity so that a newly-loaded firmware image becomes active.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5331-rsu-bmc-subcommand","title":"5.3.3.1 rsu bmc subcommand","text":"The bmc subcommand causes a Board Management Controller reset. This command is used to apply a previous fpgasupdate of a BMC firmware image. The --page argument selects the desired boot image. Valid values for --page are \u2018user\u2019 and \u2018factory\u2019.
# rsu bmc --page user 0000:b2:00.0\n# rsu bmc --page factory 0000:b2:00.0\nFigure 89 rsu bmc
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5332-rsu-retimer-subcommand","title":"5.3.3.2 rsu retimer subcommand","text":"The retimer subcommand causes a Parkvale reset (specific to Vista Creek). This command is used to apply a previous fpgasupdate of a BMC firmware image (the Parkvale firmware is contained within the BMC firmware image). The retimer subcommand causes only the Parkvale to reset.
# rsu retimer 0000:b2:00.0\nFigure 90 rsu retimer
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5333-rsu-fpga-subcommand","title":"5.3.3.3 rsu fpga subcommand","text":"The fpga subcommand causes a reconfiguration of the FPGA Static Region. This command is used to apply a previous fpgasupdate of the Static Region image. The --page argument selects the desired boot image. Valid values for --page are \u2018user1\u2019, \u2018user2\u2019, and \u2018factory\u2019.
# rsu fpga --page user1 0000:b2:00.0\n# rsu fpga --page user2 0000:b2:00.0\n# rsu fpga --page factory 0000:b2:00.0\nFigure 91 rsu fpga
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5334-rsu-sdm-subcommand","title":"5.3.3.4 rsu sdm subcommand","text":"The sdm subcommand causes a reset of the Secure Device Manager. This command is used to apply a previous fpgasupdate of the SDM image.
# rsu sdm 0000:b2:00.0\nFigure 92 rsu sdm
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5335-rsu-fpgadefault-subcommand","title":"5.3.3.5 rsu fpgadefault subcommand","text":"The fpgadefault subcommand can be used to display the default FPGA boot sequence; and it can be used to select the image to boot on the next reset of the FPGA. When given without additional parameters, the fpgadefault subcommand displays the default FPGA boot sequence:
# rsu fpgadefault 0000:b2:00.0\nFigure 93 rsu Displaying FPGA Boot Sequence
The parameters to the fpgadefault subcommand are --page and --fallback. The --page parameter accepts \u2018user1\u2019, \u2018user2\u2019, or \u2018factory\u2019, specifying the desired page to boot the FPGA from on the next reset. Note that this subcommand does not actually cause the reset to occur. Please refer to rsu fpga subcommand for an example of resetting the FPGA using the rsu command.
# rsu fpgadefault --page user1 0000:b2:00.0\n# rsu fpgadefault --page user2 0000:b2:00.0\n# rsu fpgadefault --page factory 0000:b2:00.0\nFigure 94 rsu Select FPGA Boot Image
The --fallback parameter accepts a comma-separated list of the keywords \u2018user1\u2019, \u2018user2\u2019, and \u2018factory\u2019. These keywords, in conjunction with the --page value are used to determine a fallback boot sequence for the FPGA. The fallback boot sequence is used to determine which FPGA image to load in the case of a boot failure. For example, given the following command, the FPGA would attempt to boot in the order \u2018factory\u2019, \u2018user1\u2019, \u2018user2\u2019. That is to say, if the \u2018factory\u2019 image failed to boot, then the \u2018user1\u2019 image would be tried. Failing to boot \u2018user1\u2019, the \u2018user2\u2019 image would be tried.
# rsu fpgadefault --page factory --fallback user1,user2 0000:b2:00.0\nFigure 95 rsu Select FPGA Boot Image and Fallback
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#60-sample-applications","title":"6.0 Sample Applications","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#61-afu-test-framework","title":"6.1 afu-test Framework","text":"afu-test refers to a test-writing framework that exists as a set of C++ classes written on top of the OPAE C++ bindings. The first class, afu, serves as the base class for the test application abstraction. Class afu provides integration with CLI11, a C++ \u201911 command-line parsing framework, and with spdlog, a C++ logging library. The second class, command represents a unique test sequence that is called by the afu object. Instances of the command class implement the test-specific workload.
class afu {\npublic:\nafu(const char *name,\nconst char *afu_id = nullptr,\nconst char *log_level = nullptr);\nint open_handle(const char *afu_id);\nint main(int argc, char *argv[]);\nvirtual int run(CLI::App *app, command::ptr_t test);\ntemplate<class T>\nCLI::App *register_command();\n};\nFigure 96 C++ class afu
The afu class constructor initializes the CLI11 command parser with some general, application-wide parameters.
Subcommand Description -g,--guid Accelerator AFU ID. -p,--pci-address Address of the accelerator device. -l,--log-level Requested spdlog output level. -s,--shared Open the AFU in shared mode? -t,--timeout Application timeout in milliseconds.Figure 97 class afu Application Parameters
The register_command() member function adds a test command instance to the afu object. Each test command that an afu object is capable of executing is registered during the test\u2019s startup code. For instance, here is the hssi application\u2019s use of register_command():
hssi_afu app;\nint main(int argc, char *argv[])\n{\napp.register_command<hssi_10g_cmd>();\napp.register_command<hssi_100g_cmd>();\napp.register_command<hssi_pkt_filt_10g_cmd>();\napp.register_command<hssi_pkt_filt_100g_cmd>();\n\u2026\napp.main(argc, argv);\n}\nFigure 98 hssi's app.register_command()
Next, the afu instance\u2019s main() member function is called. main() initializes the spdlog instance, searches its database of registered commands to find the command matching the test requested from the command prompt, uses the open_handle() member function to enumerate for the requested AFU ID, and calls its run() member function, passing the CLI::App and the test command variables. The run() member function initializes a test timeout mechanism, then calls the command parameter\u2019s run() to invoke the test-specific logic.
With all the boiler-plate of application startup, configuration, and running handled by the afu class, the test-specific command class is left to implement only a minimum number of member functions:
class command {\npublic:\nvirtual const char *name() const = 0;\nvirtual const char *description() const = 0;\nvirtual int run(afu *afu, CLI::App *app) = 0;\nvirtual void add_options(CLI::App *app) { }\nvirtual const char *afu_id() const { return nullptr; }\n};\nFigure 99 class command
The name() member function gives the unique command name. Some examples of names from the hssi app are hssi_10g, hssi_100g, pkt_filt_10g, and pkt_filt_100g. The description() member function gives a brief description that is included in the command-specific help output. add_options() adds command-specific command-line options. afu_id() gives the AFU ID for the command, in string form. Finally, run() implements the command-specific test functionality.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#62-afu-test-based-samples","title":"6.2 afu-test Based Samples","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#621-dummy_afu","title":"6.2.1 dummy_afu","text":"The dummy_afu application is a afu-test based application that implements three commands: mmio, ddr, and lpbk.
Target Description # dummy_afu mmio Targets special scratchpad area implemented by the AFU. # dummy_afu ddr Execute dummy_afu-specific DDR test. # dummy_afu lpbk Execute a simple loopback test."},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#622-host_exerciser","title":"6.2.2 host_exerciser","text":"host_exerciser markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#623-hssi","title":"6.2.3 hssi","text":"hssi markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#70-other-utilities","title":"7.0 Other Utilities","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#71-opaeio","title":"7.1 opae.io","text":"opae.io markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#72-bitstreaminfo","title":"7.2 bitstreaminfo","text":"The bitstreaminfo command prints diagnostic information about firmware image files that have been passed through the PACSign utility. PACSign prepends secure block 0 and secure block 1 data headers to the images that it processes. These headers contain signature hashes and other metadata that are consumed by the BMC firmware during a secure update.
To run bitstreaminfo, pass the path to the desired firmware image file:
# bitstreaminfo my_file.bin \nFigure 100 Running bitstreaminfo
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#73-fpgareg","title":"7.3 fpgareg","text":"The fpgareg command prints the register spaces for the following fpga device components:
Command Description # fpgareg 0000:b1:00.0 pcie Walks and prints the DFL for the device. # fpgareg 0000:b1:00.0 bmc Prints the BMC registers for the device. # fpgareg 0000:b1:00.0 hssi Prints the HSSI registers for the device. # fpgareg 0000:b1:00.0 acc Prints the AFU register spaces.Figure 101 fpgareg Commands
Note that fpgareg is only available as of Arrow Creek ADP and forward. It will not work with prior platforms, eg N3000.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#74-opaevfio","title":"7.4 opaevfio","text":"opaevfio markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#80-opae-build-configurations","title":"8.0 OPAE Build Configurations","text":"Refer to the software installation guides on this website for detailed instructions and building and installing both the OPAE SDK and Linux DFL driver set for any given release. The following section has been included to provide more information on the build targets for the DFL drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#81-cmake-options","title":"8.1 CMake Options","text":"Option Description Values Default -DCMAKE_BUILD_TYPE Configure debugging infoDebug
Release
Coverage
RelWithDebInfo
RelWithDebInfo -DCMAKE_INSTALL_PREFIX Root install path /usr/local -DOPAE_BUILD_SPHINX_DOC Enable/Disable docs ON/OFF OFF -DOPAE_BUILD_TESTS Enable/Disable unit tests ON/OFF OFF -DOPAE_ENABLE_MOCK Enable/Disable mock driver for unit tests ON/OFF OFF -DOPAE_INSTALL_RPATH Enable/Disable rpath for install ON/OFF OFF -DOPAE_VERSION_LOCAL Local version string -DOPAE_PRESERVE_REPOS Preserve local changes to external repos? ON/OFF OFF -D OPAE_BUILD_LIBOPAE_CXX Enable C++ bindings ON/OFF ON -DOPAE_WITH_PYBIND11 Enable pybind11 ON/OFF ON -D OPAE_BUILD_PYTHON_DIST Enable Python bindings ON/OFF OFF -DOPAE_BUILD_LIBOPAEVFIO Build libopaevfio.so ON/OFF ON -D OPAE_BUILD_PLUGIN_VFIO Build libopae-v.so ON/OFF ON -DOPAE_BUILD_LIBOPAEUIO Build libopaeuio.so ON/OFF ON -DOPAE_BUILD_LIBOFS Build OFS Copy Engine ON/OFF ON -DOPAE_BUILD_SAMPLES Build Samples ON/OFF ON -DOPAE_BUILD_LEGACY Build legacy repo ON/OFF OFF -DOPAE_LEGACY_TAG Specify legacy build tag master -DOPAE_WITH_CLI11 Enable apps which use CLI11 ON/OFF ON -DOPAE_WITH_SPDLOG Enable apps which use spdlog ON/OFF ON -DOPAE_WITH_LIBEDIT Enable apps which use libedit ON/OFF ON -DOPAE_WITH_HWLOC Enable apps which use hwloc ON/OFF ON -DOPAE_WITH_TBB Enable apps which use Thread Building Blocks ON/OFF ON -DOPAE_MINIMAL_BUILD Enable/Disable minimal build. When set to ON, disable CLI11, spdlog, libedit, hwloc, tbb ON/OFF OFFTable 12 CMake Options
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#82-building-opae-for-debug","title":"8.2 Building OPAE for Debug:","text":"$ cmake .. -DCMAKE_BUILD_TYPE=Debug\nTo ease the RPM creation process, the OPAE SDK provides a simple RPM creation script. The parameter to the RPM create script for either Fedora or RHEL is unrestricted For rhel, the build flag -DOPAE_MINIMAL_BUILD is set to ON, omitting the binaries which have dependencies on external components that RHEL does not include in its base repositories.
In order to create RPMs for Fedora, run the create script on a system loaded with all the Fedora build prerequisites. If prerequisites are missing, the create script will complain until they are resolved.
In order to create RPMs for RHEL, run the create script on a system loaded with all the RHEL build prerequisites. If prerequisites are missing, the create script will complain until they are resolved.
$ cd opae-sdk\n$ ./packaging/opae/rpm/create unrestricted\nThe RPMs will be versioned according to the information found in the file packaging/opae/version. If you edit this file to update the version information, then re-run the create script to create the RPMs.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#90-debugging-opae","title":"9.0 Debugging OPAE","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#91-enabling-debug-logging","title":"9.1 Enabling Debug Logging","text":"The OPAE SDK has a built-in debug logging facility. To enable it, set the cmake flag -DCMAKE_BUILD_TYPE=Debug and then use the following environment variables: | Variable| Description| | ----- | ----- | |LIBOPAE_LOG=1| Enable debug logging output. When not set, only critical error messages are displayed.| |LIBOPAE_LOGFILE=file.log| Capture debug log output to file.log. When not set, the logging appears on stdout and stderr. The file must appear in a relative path or it can be rooted at /tmp.|
Table 13 Logging Environment Variables
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#92-gdb","title":"9.2 GDB","text":"To enable gdb-based debugging, the cmake configuration step must specify a value for -DCMAKE_BUILD_TYPE of either Debug or RelWithDebInfo so that debug symbols are included in the output binaries. The OPAE SDK makes use of dynamically-loaded library modules. When debugging with gdb, the best practice is to remove all OPAE SDK libraries from the system installation paths to ensure that library modules are only loaded from the local build tree:
$ cd opae-sdk/build\n$ LD_LIBRARY_PATH=$PWD/lib gdb --args <some_opae_executable> <args>\nFigure 103 Debugging with GDB
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#100-adding-new-device-support","title":"10.0 Adding New Device Support","text":"As of OPAE 2.2.0 the SDK has transitioned to a single configuration file model. The libraries, plugins, and applications obtain their runtime configuration during startup by examining a single JSON configuration file. In doing so, the original configuration file formats for libopae-c and fpgad have been deprecated in favor of the respective sections in the new configuration file.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#101-configuration-file-search-order","title":"10.1 Configuration File Search Order","text":"By default the OPAE SDK will install its configuration file to /etc/opae/opae.cfg.
/etc/opae/opae.cfg Figure 104 Default Configuration File
The SDK searches for the configuration file during startup by employing the following search algorithm:
First, the environment variable LIBOPAE_CFGFILE is examined. If it is set to a path that represents a valid path to a configuration file, then that configuration file path is used, and the search is complete.
Next, the HOME environment variable is examined. If its value is valid, then it is prepended to the following set of relative paths. If HOME is not set, then the search continues with the value of the current user\u2019s home path as determined by getpwuid(). The home path, if any, determined by getpwuid() is prepended to the following set of relative paths. Searching completes successfully if any of these home-relative search paths is valid.
/.local/opae.cfg /.local/opae/opae.cfg /.config/opae/opae.cfg Figure 105 HOME Relative Search Paths
Finally, the configuration file search continues with the following system-wide paths. If any of these paths is found to contain a configuration file, then searching completes successfully.
usr/local/etc/opae/opae.cfg /etc/opae/opae.cfg Figure 106 System Search Paths
If the search exhausts all of the possible configuration file locations without finding a configuration file, then an internal default configuration is used. This internal default configuration matches that of the opae.cfg file shipped with the OPAE SDK.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#102-configuration-file-format","title":"10.2 Configuration File Format","text":"The OPAE SDK configuration file is stored in JSON formatted text. The file has two main sections: \u201cconfigs\u201d and \u201cconfigurations\u201d. The \u201cconfigs\u201d section is an array of strings. Each value in the \u201cconfigs\u201d array is a key into the data stored in the \u201cconfigurations\u201d section. If a key is present in \u201cconfigs\u201d, then that key is searched for and processed in \u201cconfigurations\u201d. If the key is not found in \u201cconfigs\u201d, then that section of \u201cconfigurations\u201d will not be processed, irrespective of whether it exists in \u201cconfigurations\u201d.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... } }, \u201cconfigs\u201d: [ \u201cc6100\u201d ] } Figure 107 Keyed Configurations
Each keyed section in \u201cconfigurations\u201d has four top-level entries: \u201cenabled\u201d, \u201cplatform\u201d, \u201cdevices\u201d, \u201copae\u201d.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { \u201cenabled\u201d: true, \u201cplatform\u201d: \u201cIntel Acceleration Development Platform C6100\u201d, \u201cdevices\u201d: [ { \u201cname\u201d: \u201cc6100_pf\u201d, \u201cid\u201d: [ ... ] }, { \u201cname\u201d: \u201cc6100_vf\u201d, \u201cid\u201d: [ ... ] } ], \u201copae\u201d: { ... } } }, } Figure 108 Configurations Format
The \u201cenabled\u201d key holds a Boolean value. If the value is false or if the \u201cenabled\u201d key is omitted, then that configuration is skipped when parsing the file. The \u201cplatform\u201d key holds a string that identifies the current configuration item as a product family. The \u201cdevices\u201d key contains the device descriptions.
\u201cdevices\u201d is an array of objects that contain a \u201cname\u201d and an \u201cid\u201d key. The \u201cname\u201d is a shorthand descriptor for a device PF or VF. The value of \u201cname\u201d appears elsewhere in the current \u201cconfigurations\u201d section in order to uniquely identify the device. \u201cid\u201d is an array of four strings, corresponding to the PCIe Vendor ID, Device ID, Subsystem Vendor ID, and Subsystem Device ID of the device. The entries corresponding to Vendor ID and Device ID must contain valid 16-bit hex integers. The entries corresponding to Subsystem Vendor ID and Subsystem Device ID may be 16-bit hex integers or the special wildcard string \u201c*\u201d, which indicates a don\u2019t care condition.
The remaining sections in this chapter outline the format of the \u201copae\u201d configurations key.
\u201cplugin\u201d: libopae-c and libopae-v
The \u201cplugin\u201d key in the \u201copae\u201d section of a configuration is an array of OPAE SDK plugin configuration data. Each item in the array matches one or more PF or VF devices to a plugin library module.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201cplugin\u201d: [ { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibxfpga.so\u201d, \u201cdevices\u201d: [ \u201cc6100_pf\u201d ], \u201cconfiguration\u201d: {} }, { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibopae-v.so\u201d, \u201cdevices\u201d: [ \u201cc6100_pf\u201d, \u201cc6100_vf\u201d ], \u201cconfiguration\u201d: {} } ], } } }, } Figure 109 \"opae\" / \"plugin\" key/
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cplugin\u201d array entry is skipped, and parsing continues. The \u201cmodule\u201d key is a string that identifies the desired plugin module library for the entry. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section. The \u201cconfiguration\u201d key of the \u201cplugin\u201d section specifies a unique plugin-specific configuration. Currently, libopae-c and libopae-v use no plugin-specific config, so these keys are left empty.
\u201cfpgainfo\u201d: fpgainfo application
The \u201cfpgainfo\u201d key in the \u201copae\u201d section of a configuration is an array of fpgainfo plugin configuration data. Each item in the array matches one or more PF or VF devices to an fpgainfo plugin library module.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201cfpgainfo\u201d: [ { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibboard_c6100.so\u201d, \u201cdevices\u201d: [ { \u201cdevice\u201d: \u201cc6100_pf\u201d, \u201cfeature_id\u201d: \u201c0x12\u201d }, { \u201cdevice\u201d: \u201cc6100_vf\u201d, \u201cfeature_id\u201d: \u201c0x12\u201d } ] } ], } } }, } Figure 110 \"opae\" / \"fpgainfo\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cfpgainfo\u201d array entry is skipped, and parsing continues. The \u201cmodule\u201d key is a string that identifies the desired fpgainfo module library for the entry. Each \u201cdevices\u201d array entry gives a PF/VF identifier in its \u201cdevice\u201d key and a DFL feature ID in its \u201cfeature_id\u201d key.
\u201cfpgad\u201d: fpgad daemon process
The \u201cfpgad\u201d key in the \u201copae\u201d section of a configuration is an array of fpgad plugin configuration data. Each item in the array matches one or more PF or VF devices to an fpgad plugin library module.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201cfpgad\u201d: [ { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibfpgad-vc.so\u201d, \u201cdevices\u201d: [ \u201cc6100_pf\u201d ], \u201cconfiguration\u201d: { ... } } ], } } }, } Figure 111 \"opae\" / \"fpgad\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cfpgad\u201d array entry is skipped, and parsing continues. The \u201cmodule\u201d key is a string that identifies the desired fpgad plugin module library for the entry. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section. The \u201cconfiguration\u201d key of the \u201cfpgad\u201d section specifies a unique plugin-specific configuration.
\u201crsu\u201d: rsu script
The \u201crsu\u201d key in the \u201copae\u201d section of a configuration is an array of rsu script configuration data. Each item in the array matches one or more PF devices to an rsu configuration.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201crsu\u201d: [ { \u201cenabled\u201d: true, \u201cdevices\u201d: [ \u201cc6100_pf\u201d ], \u201cfpga_default_sequences\u201d: \u201ccommon_rsu_sequences\u201d } ], } } }, \u201ccommon_rsu_sequences\u201d: [ ... ] } Figure 112 \"opae\" / \"rsu\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201crsu\u201d array entry is skipped, and parsing continues. When disabled, the device(s) mentioned in that array entry will not be available for the rsu command. The \u201cdevices\u201d array lists one or more PF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section. The \u201cfpga_default_sequences\u201d key of the \u201crsu\u201d section specifies a JSON key. The configuration searches for that JSON key at the global level of the configuration file, and when found applies its value as the valid set of fpga boot sequences that can be used with the rsu fpgadefault subcommand.
C \u201cfpgareg\u201d: fpgareg script
The \u201cfpgareg\u201d key in the \u201copae\u201d section of a configuration is an array of fpgareg script configuration data. Each item in the array matches one or more PF/VF devices to an fpgareg configuration.
```C {
\u201cconfigurations\u201d: {
\u201cc6100\u201d: {\n\n ...\n\n \u201copae\u201d: {\n\n \u201cfpgareg\u201d: [\n\n {\n\n \u201cenabled\u201d: true,\n\n \u201cdevices\u201d: [ \u201cc6100_pf\u201d, \u201cc6100_vf\u201d ]\n\n }\n\n ],\n\n }\n\n}\n},
} ```
Figure 113 \"opae\" / \"fpgareg\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cfpgareg\u201d array entry is skipped, and parsing continues. When disabled, the device(s) mentioned in that array entry will not be available for the fpgareg command. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section.
\u201copae.io\u201d: opae.io application The \u201copae.io\u201d key in the \u201copae\u201d section of a configuration is an array of opae.io configuration data. Each item in the array matches one or more PF/VF devices to an opae.io platform string.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201copae.io\u201d: [ { \u201cenabled\u201d: true, \u201cdevices\u201d: [ \u201cc6100_pf\u201d, \u201cc6100_vf\u201d ] } ], } } }, } Figure 114 \"opae\" / \"opae.io\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201copae.io\u201d array entry is skipped, and parsing continues. When disabled, the device(s) mentioned in that array entry will continue to be available for the opae.io command. The device(s) platform string will not be shown in the opae.io ls command. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section.
Libxfpga \u2013 Updating the Metrics API
Edit libraries/plugins/xfpga/sysfs.c. Find the definition of the opae_id_to_hw_type() function. Update the function to add the new vendor/device ID to hw_type mapping.
This mapping is used by the SDK\u2019s metrics API to determine the method of accessing the board sensor information and is very specific to the underlying BMC implementation. It may be necessary to add a new hw_type value and to update the logic in libraries/plugins/xfpga/metrics.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#110-dfl-linux-kernel-drivers","title":"11.0 DFL Linux Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks such as DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality.
The OFS driver software can be found in the OFS repositories linux-dfl and linux-dfl-backport. These repositoryies have an associated OFS repository - linux-dfl that includes the following information:
The Linux Operating System treats the FPGA hardware as a PCIe* device. A predefined data structure, Device Feature List (DFL), allows for dynamic feature discovery in an Intel FPGA solution.
The Linux Device Driver implements PCIe Single Root I/O Virtualization (SR-IOV) for the creation of Virtual Functions (VFs). The device driver can release individual accelerators for assignment to virtual machines (VMs).
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#112-fpga-management-engine-fme","title":"11.2 FPGA Management Engine (FME)","text":"The FPGA Management Engine provides error reporting, reconfiguration, performance reporting, and other infrastructure functions. Each FPGA has one FME which is always accessed through the Physical Function (PF). The Intel Xeon\u00ae Processor with Integrated FPGA also performs power and thermal management. These functions are not available on the Intel Programmable Acceleration Card (PAC).
User-space applications can acquire exclusive access to the FME using open(), and release it using close(). Device access may be managed by standard Linux interfaces and tools.
If an application terminates without freeing the FME or Port resources, Linux closes all file descriptors owned by the terminating process, freeing those resources.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#113-port","title":"11.3 Port","text":"A Port represents the interface between two components: * The FPGA Interface Manager (FIM) which is part of the static FPGA fabric * The Accelerator Function Unit (AFU) which is the partially reconfigurable region
The Port controls the communication from software to the AFU and makes features such as reset and debug available.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#114-accelerator-function-unit-afu","title":"11.4 Accelerator Function Unit (AFU)","text":"An AFU attaches to a Port. The AFU provides a 256 KB memory mapped I/O (MMIO) region for accelerator-specific control registers.
open() on the Port device to acquire access to an AFU associated with the Port device.close()on the Port device to release the AFU associated with the Port device.mmap() on the Port device to map accelerator MMIO regions.Use PR to reconfigure an AFU from a bitstream file. Successful reconfiguration has the following requirement:
In all other cases PR fails and may cause system instability.
Platforms that support 512-bit Partial Reconfiguration require binutils >= version 2.25.
Close any software programs accessing the FPGA, including those running in a virtualized host before initiating PR. For virtualized environments, the recommended sequence is as follows:
Releasing the VF from the guest while an application on the guest is still accessing its resources may lead to VM instabilities. We recommend closing all applications accessing the VF in the guest before releasing the VF.
To enable accelerator access from applications running on a VM, create a VF for the port using the following process:
Release the Port from the PF using the associated ioctl on the FME device.
Use the following command to enable SR-IOV and VFs. Each VF can own a single Port with an AFU. In the following command, N is the number of Port released from the PF.
echo N > $PCI_DEVICE_PATH/sriov_numvfs\nThe number, 'N', cannot be greater than the number of supported VFs. This can be read from $PCI_DEVICE_PATH/sriov_totalvfs.
Pass the VFs through to VMs using hypervisor interfaces.
Access the AFU on a VF from applications running on the VM using the same driver inside the VM.
Creating VFs is only supported for port devices. Consequently, PR and other management functions are only available through the PF.
If assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices).
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#117-driver-organization","title":"11.7 Driver Organization","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1171-pcie-module-device-driver","title":"11.7.1 PCIe Module Device Driver","text":"FPGA devices appear as a PCIe devices. Once enumeration detects a PCIe PF or VF, the Linux OS loads the FPGA PCIe device driver. The device driver performs the following functions:
The PCIe Module Device Driver performs the following functions:
The FME Platform Module Device Driver loads automatically after the PCIe driver creates the FME Platform Module. It provides the following features for FPGA management:
Power and thermal management, error reporting, performance reporting, and other infrastructure functions. You can access these functions via sysfs interfaces the FME driver provides.
Partial Reconfiguration. During PR sub-feature initialization, the FME driver registers the FPGA Manager framework to support PR. When the FME receives the relevant ioctl request from user-space, it invokes the common interface function from the FPGA Manager to reconfigure the AFU using PR.
Port management for virtualization (releasing/assigning port device).
After a port device is released, you can use the PCIe driver SR-IOV interfaces to create/destroy VFs.
For more information, refer to \"FPGA Virtualization\".
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1174-fme-platform-module-device-driver-functions","title":"11.7.4 FME Platform Module Device Driver Functions","text":"The FME Platform Module Device Driver performs the the following functions:
After the PCIe Module Device Driver creates the Port Platform Module device, the FPGA Port and AFU driver are loaded. This module provides an interface for user-space applications to access the individual accelerators, including basic reset control on the Port, AFU MMIO region export, DMA buffer mapping service, and remote debug functions.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1176-port-platform-module-device-driver-functions","title":"11.7.6 Port Platform Module Device Driver Functions","text":"The Port Platform Module Device Driver performs the the following functions:
The user-space interface consists of a sysfs hierarchy and ioctl requests. Most kernel attributes can be accessed/modified via sysfs nodes in this hierarchy. More complex I/O operations are controlled via ioctl requests. The OPAE API implementation, libopae-c, has been designed to use this interface to interact with the OPAE FPGA kernel drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#120-plugin-development","title":"12.0 Plugin Development","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#121-overview","title":"12.1 Overview","text":"Beginning with OPAE C library version 1.2.0, OPAE implements a plugin-centric model. This guide serves as a reference to define the makeup of an OPAE C API plugin and to describe a sequence of steps that one may follow when constructing an OPAE C API plugin.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#122-plugin-required-functions","title":"12.2 Plugin Required Functions","text":"An OPAE C API plugin is a runtime-loadable shared object library, also known as a module. On Linux systems, the dl family of APIs from libdl are used to interact with shared objects. Refer to \"man dlopen\" and \"man dlsym\" for examples of using the libdl API.
An OPAE C API plugin implements one required function. This function is required to have C linkage, so that its name is not mangled.
int opae_plugin_configure(opae_api_adapter_table *table, const char *config);\nDuring initialization, the OPAE plugin manager component loads each plugin, searching for its opae_plugin_configure function. If none is found, then the plugin manager rejects that plugin. When it is found, opae_plugin_configure is called passing a pointer to a freshly-created opae_api_adapter_table and a buffer consisting of configuration data for the plugin.
The job of the opae_plugin_configure function is to populate the given adapter table with each of the plugin's API entry points and to consume and comprehend the given configuration data in preparation for initialization.
The adapter table is a data structure that contains function pointer entry points for each of the OPAE APIs implemented by a plugin. In this way, it adapts the plugin-specific behavior to the more general case of a flat C API. Note that OPAE applications are only required to link with opae-c. In other words, the name of the plugin library should not appear on the linker command line. In this way, plugins are truly decoupled from the OPAE C API, and they are required to adapt to the strict API specification by populating the adapter table only. No other linkage is required nor recommended.
adapter.h contains the definition of the opae_api_adapter_table. An abbreviated version is depicted below, along with supporting type opae_plugin:
typedef struct _opae_plugin {\nchar *path;\nvoid *dl_handle;\n} opae_plugin;\ntypedef struct _opae_api_adapter_table {\nstruct _opae_api_adapater_table *next;\nopae_plugin plugin;\nfpga_result (*fpgaOpen)(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result (*fpgaClose)(fpga_handle handle);\n...\nfpga_result (*fpgaEnumerate)(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n// configuration functions\nint (*initialize)(void);\nint (*finalize)(void);\n// first-level query\nbool (*supports_device)(const char *device_type);\nbool (*supports_host)(const char *hostname);\n} opae_api_adapter_table;\nSome points worth noting are that the adapter tables are organized in memory by adding them to a linked list data structure. This is the use of the next structure member. (The list management is handled by the plugin manager.) The plugin structure member contains the handle to the shared object instance, as created by dlopen. This handle is used in the plugin's opae_plugin_configure to load plugin entry points. A plugin need only implement the portion of the OPAE C API that a target application needs. Any API entry points that are not supported should be left as NULL pointers (the default) in the adapter table. When an OPAE API that has no associated entry point in the adapter table is called, the result for objects associated with that plugin will be FPGA_NOT_SUPPORTED.
The following code illustrates a portion of the opae_plugin_configure for a theoretical OPAE C API plugin libfoo.so:
/* foo_plugin.c */\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\nadapter->fpgaOpen = dlsym(adapter->plugin.dl_handle, \"foo_fpgaOpen\");\nadapter->fpgaClose =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaClose\");\n...\nadapter->fpgaEnumerate =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaEnumerate\");\n...\nreturn 0;\n}\nNotice that the implementations of the API entry points for plugin libfoo.so are prefixed with foo_. This is the recommended practice to avoid name collisions and to enhance the debugability of the application. Upon successful configuration, opae_plugin_configure returns 0 to indicate success. A non-zero return value indicates failure and causes the plugin manager to reject the plugin from futher consideration.
Once the plugin manager loads and configures each plugin, it uses the adapter table to call back into the plugin so that it can be made ready for runtime. This is the job of the opae_plugin_initialize entry point, whose signature is defined as:
int opae_plugin_initialize(void);\nThe function takes no parameters, as the configuration data was already given to the plugin by opae_plugin_configure. opae_plugin_initialize returns 0 if no errors were encountered during initialization. A non-zero return code indicates that plugin initialization failed. A plugin makes its opae_plugin_initialize available to the plugin manager by populating the adapter table's initialize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_initialize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->initialize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_initialize\");\n...\nreturn 0;\n}\nIf a plugin does not implement an opae_plugin_initialize entry point, then the initialize member of the adapter table should be left uninitialized. During plugin initialization, if a plugin has no opae_plugin_initialize entry in its adapter table, the plugin initialization step will be skipped, and the plugin will be considered to have initialized successfully.
Once plugin initialization is complete for all loaded plugins, the system is considered to be running and fully functional.
During teardown, the plugin manager uses the adapter table to call into each plugin's opae_plugin_finalize entry point, whose signature is defined as:
int opae_plugin_finalize(void);\nopae_plugin_finalize returns 0 if no errors were encountered during teardown. A non-zero return code indicates that plugin teardown failed. A plugin makes its opae_plugin_finalize available to the plugin manager by populating the adapter table's finalize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_finalize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->finalize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_finalize\");\n...\nreturn 0;\n}\nIf a plugin does not implement an opae_plugin_finalize entry point, then the finalize member of the adapter table should be left uninitialized. During plugin cleanup, if a plugin has no opae_plugin_finalize entry point in its adapter table, the plugin finalize step will be skipped, and the plugin will be considered to have finalized successfully.
In addition to initialize and finalize, an OPAE C API plugin has two further optional entry points that relate to device enumeration. During enumeration, when a plugin is being considered for a type of device, the plugin may provide input on that decision by exporting an opae_plugin_supports_device entry point in the adapter table:
bool opae_plugin_supports_device(const char *device_type);\nopae_plugin_supports_device returns true if the given device type is supported and false if it is not. A false return value from opae_plugin_supports_device causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_device is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_device(const char *device_type)\n{\nif (/* device_type is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_device =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_device\");\n...\nreturn 0;\n}\n.. note::\n The `opae_plugin_supports_device` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\nSimilarly to determining whether a plugin supports a type of device, a plugin may also answer questions about network host support by populating an opae_plugin_supports_host entry point in the adapter table:
bool opae_plugin_supports_host(const char *hostname);\nopae_plugin_supports_host returns true if the given hostname is supported and false if it is not. A false return value from opae_plugin_supports_host causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_host is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_host(const char *hostname)\n{\nif (/* hostname is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_host =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_host\");\n...\nreturn 0;\n}\n.. note::\n The `opae_plugin_supports_host` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\nThe steps required to implement an OPAE C API plugin, libfoo.so, are:
opae_plugin_configure, opae_plugin_initialize, opae_plugin_finalize, opae_plugin_supports_device, and opae_plugin_supports_host as described in the previous sections. /* foo_plugin.h */\nfpga_result foo_fpgaOpen(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result foo_fpgaClose(fpga_handle handle);\n...\nfpga_result foo_fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n /* foo_types.h */\nstruct _foo_token {\n...\n};\nstruct _foo_handle {\n...\n};\nstruct _foo_event_handle {\n...\n};\nstruct _foo_object {\n...\n};\nfoo_fpgaEnumerate, foo_fpgaCloneToken, and foo_fpgaDestroyToken.foo_fpgaOpen.foo_fpgaClose.foo_fpgaGetProperties, foo_fpgaGetPropertiesFromHandle, foo_fpgaUpdatePropertiesfoo_fpgaMapMMIO, foo_fpgaUnmapMMIO foo_fpgaWriteMMIO64, foo_fpgaReadMMIO64, foo_fpgaWriteMMIO32, foo_fpgaReadMMIO32.foo_fpgaPrepareBuffer, foo_fpgaReleaseBuffer, foo_fpgaGetIOAddress.foo_fpgaReadError, foo_fpgaClearError, foo_fpgaClearAllErrors, foo_fpgaGetErrorInfo.foo_fpgaCreateEventHandle, foo_fpgaDestroyEventHandle, foo_fpgaGetOSObjectFromEventHandle, foo_fpgaRegisterEvent, foo_fpgaUnregisterEvent.foo_fpgaReconfigureSlot.foo_fpgaTokenGetObject, foo_fpgaHandleGetObject, foo_fpgaObjectGetObject, foo_fpgaDestroyObject, foo_fpgaObjectGetSize, foo_fpgaObjectRead, foo_fpgaObjectRead64, foo_fpgaObjectWrite64.foo_fpgaSetUserClock, foo_fpgaGetUserClock.Building a sample application The library source includes code samples. Use these samples to learn how to call functions in the library. Build and run these samples as quick sanity checks to determine if your installation and environment are set up properly.
In this guide, we will build hello_fpga.c. This is the \"Hello World!\" example of using the library. This code searches for a predefined and known AFU called \"Native Loopback Adapter\" on the FPGA. If found, it acquires ownership and then interacts with the AFU by sending it a 2MB message and waiting for the message to be echoed back. This code exercises all major components of the API except for AFU reconfiguration: AFU search, enumeration, access, MMIO, and memory management.
You can also find the source for hello_fpga in the samples directory of the OPAE SDK repository on GitHub.
int main(int argc, char *argv[])\n{\nfpga_properties filter = NULL;\nfpga_token afu_token;\nfpga_handle afu_handle;\nfpga_guid guid;\nuint32_t num_matches;\nvolatile uint64_t *dsm_ptr = NULL;\nvolatile uint64_t *status_ptr = NULL;\nvolatile uint64_t *input_ptr = NULL;\nvolatile uint64_t *output_ptr = NULL;\nuint64_t dsm_wsid;\nuint64_t input_wsid;\nuint64_t output_wsid;\nfpga_result res = FPGA_OK;\nif (uuid_parse(NLB0_AFUID, guid) < 0) {\nfprintf(stderr, \"Error parsing guid '%s'\\n\", NLB0_AFUID);\ngoto out_exit;\n}\n/* Look for accelerator by its \"afu_id\" */\nres = fpgaGetProperties(NULL, &filter);\nON_ERR_GOTO(res, out_exit, \"creating properties object\");\nres = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nON_ERR_GOTO(res, out_destroy_prop, \"setting object type\");\nres = fpgaPropertiesSetGuid(filter, guid);\nON_ERR_GOTO(res, out_destroy_prop, \"setting GUID\");\n/* TODO: Add selection via BDF / device ID */\nres = fpgaEnumerate(&filter, 1, &afu_token, 1, &num_matches);\nON_ERR_GOTO(res, out_destroy_prop, \"enumerating accelerators\");\nif (num_matches < 1) {\nfprintf(stderr, \"accelerator not found.\\n\");\nres = fpgaDestroyProperties(&filter);\nreturn FPGA_INVALID_PARAM;\n}\n/* Open accelerator and map MMIO */\nres = fpgaOpen(afu_token, &afu_handle, 0);\nON_ERR_GOTO(res, out_destroy_tok, \"opening accelerator\");\nres = fpgaMapMMIO(afu_handle, 0, NULL);\nON_ERR_GOTO(res, out_close, \"mapping MMIO space\");\n/* Allocate buffers */\nres = fpgaPrepareBuffer(afu_handle, LPBK1_DSM_SIZE,\n(void **)&dsm_ptr, &dsm_wsid, 0);\nON_ERR_GOTO(res, out_close, \"allocating DSM buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&input_ptr, &input_wsid, 0);\nON_ERR_GOTO(res, out_free_dsm, \"allocating input buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&output_ptr, &output_wsid, 0);\nON_ERR_GOTO(res, out_free_input, \"allocating output buffer\");\nprintf(\"Running Test\\n\");\n/* Initialize buffers */\nmemset((void *)dsm_ptr, 0, LPBK1_DSM_SIZE);\nmemset((void *)input_ptr, 0xAF, LPBK1_BUFFER_SIZE);\nmemset((void *)output_ptr, 0xBE, LPBK1_BUFFER_SIZE);\ncache_line *cl_ptr = (cache_line *)input_ptr;\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE / CL(1); ++i) {\ncl_ptr[i].uint[15] = i+1; /* set the last uint in every cacheline */\n}\n/* Reset accelerator */\nres = fpgaReset(afu_handle);\nON_ERR_GOTO(res, out_free_output, \"resetting accelerator\");\n/* Program DMA addresses */\nuint64_t iova;\nres = fpgaGetIOAddress(afu_handle, dsm_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting DSM IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_AFU_DSM_BASEL, iova);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_AFU_DSM_BASEL\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 0);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 1);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaGetIOAddress(afu_handle, input_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting input IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_SRC_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_SRC_ADDR\");\nres = fpgaGetIOAddress(afu_handle, output_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting output IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_DST_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_DST_ADDR\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_NUM_LINES, LPBK1_BUFFER_SIZE / CL(1));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_NUM_LINES\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CFG, 0x42000);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nstatus_ptr = dsm_ptr + DSM_STATUS_TEST_COMPLETE/8;\n/* Start the test */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 3);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Wait for test completion */\nwhile (0 == ((*status_ptr) & 0x1)) {\nusleep(100);\n}\n/* Stop the device */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 7);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Check output buffer contents */\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE; i++) {\nif (((uint8_t*)output_ptr)[i] != ((uint8_t*)input_ptr)[i]) {\nfprintf(stderr, \"Output does NOT match input \"\n\"at offset %i!\\n\", i);\nbreak;\n}\n}\nprintf(\"Done Running Test\\n\");\n/* Release buffers */\nout_free_output:\nres = fpgaReleaseBuffer(afu_handle, output_wsid);\nON_ERR_GOTO(res, out_free_input, \"releasing output buffer\");\nout_free_input:\nres = fpgaReleaseBuffer(afu_handle, input_wsid);\nON_ERR_GOTO(res, out_free_dsm, \"releasing input buffer\");\nout_free_dsm:\nres = fpgaReleaseBuffer(afu_handle, dsm_wsid);\nON_ERR_GOTO(res, out_unmap, \"releasing DSM buffer\");\n/* Unmap MMIO space */\nout_unmap:\nres = fpgaUnmapMMIO(afu_handle, 0);\nON_ERR_GOTO(res, out_close, \"unmapping MMIO space\");\n/* Release accelerator */\nout_close:\nres = fpgaClose(afu_handle);\nON_ERR_GOTO(res, out_destroy_tok, \"closing accelerator\");\n/* Destroy token */\nout_destroy_tok:\nres = fpgaDestroyToken(&afu_token);\nON_ERR_GOTO(res, out_destroy_prop, \"destroying token\");\n/* Destroy properties object */\nout_destroy_prop:\nres = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(res, out_exit, \"destroying properties object\");\nout_exit:\nreturn res;\n}\nLinking with the OPAE library is straightforward. Code using this library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, the minimalist compile and link line should look like:
gcc -std=c99 hello_fpga.c -I/usr/local/include -L/usr/local/lib -lopae-c -luuid -ljson-c -lpthread -o hello_fpga\nThe API uses some features from the C99 language standard. The -std=c99 switch is required if the compiler does not support C99 by default.
Third-party library dependency: The library internally uses libuuid and libjson-c. But they are not distributed as part of the library. Make sure you have these libraries properly installed. The -c flag may not be necessary depending on the platform being used.
sudo ./hello_fpga -c\nRunning Test\nRunning on bus 0x08.\nAFU NLB0 found @ 28000\nDone Running Test\nsudo ./hello_fpga\nRunning Test\nDone\nThe process of adding a custom device to the DFL framework (backport or not) and OPAE SDK requires changes to files in several locations. In this section we will walk through the additions necessary to instruct the kernel's probe and match function to associate your new device with OPAE, choose the correct OPAE plugin to associate with your board, and change basic descriptors to properly display the name of your new custom platform when using OPAE's management interfaces.
This section does not require useage of an entirely new platform - we will use the Intel N6001's FIM design as our base, and alter only those settings in the PCIe IP necessary to change the following PCIe IDs for both the PF and VF:
This document will not cover the process by which a FIM can be modified to support these new IDs. Refer to the Shell Developer Guide: Agilex\u00ae 7 SoC Attach: Open FPGA Stack or Shell Developer Guide for Open FPGA Stack: Intel\u00ae FPGA SmartNIC N6000-PL / Intel\u00ae FPGA SmartNIC N6001-PL PCIe Attach FIM Developer guide for an overview of this process. The following sections assume you have a FIM that has been configured with new IDs. This FIM can be loaded onto your N6001 board using either the SOF via a USB-BlasterII cable and the Quartus Programmer, or by using the OPAE command fpgasupdate consuming a compatible binary programming file. The following values will be used as our new device IDs.
The only value that differs between PF0 and PF0-VF in this example is the Device ID, and all other values do not currently exist in either the OPAE SDK or linux-dfl drivers. You will need to download the OPAE SDK and linux-dfl sources from GitHub and modify their contents. We will be using a validated Agilex OFS release to pair our OPAE SDK and linux-dfl versions together. Refer to the below table for a list of the exact version we will use for each software component and where you can learn how to build and install. Your versions may not match those in the document.
Software Version Build Instructions OPAE SDK 2.3.0-1 4.0 OPAE Software Development Kit linux-dfl ofs-2022.3-2 3.0 Intel OFS DFL Kernel DriversThe following steps will enable your device to use the OPAE SDK. We will call our new device the \"Intel FPGA Programmable Acceleration Card N6002\". This device is identical to the Intel FPGA Programmable Acceleration Card N6001, and will use the pre-existing plugins and feature ID associated with that device. We will also use the enum value FPGA_HW_DCP_N6002 to describe our new board in the code. These plugins can be customized as you become more familiar with the OPAE SDK software.
opae-sdk.binaries/opae.io/opae/io/config.py. Add a new configuration entry under DEFAULT_OPAE_IO_CONFIG. Save and exit.Example:
(0xff00, 0x1234, 0xff00, 0x5678) : {\n'platform': 'Intel FPGA Programmable Acceleration Card N6002'\n},\nlibraries/libopae-c/cfg-file.c. Add two new entries (one for PF0 and PF0-VF) under STATIC libopae_config_data default_libopae_config_table[]. Add two new entries under STATIC fpgainfo_config_data default_fpgainfo_config_table[]. Save and exit.Example:
STATIC fpgainfo_config_data default_fpgainfo_config_table[] = {\n...\n{ 0xff00, 0x1234, 0xff00, 0x5678, 0x12, \"libboard_n6000.so\", NULL,\n\"Intel FPGA Programmable Acceleration Card N6002\" }, // N6002 PF0\n{ 0xff00, 0x5555, 0xff00, 0x5678, 0x12, \"libboard_n6000.so\", NULL,\n\"Intel FPGA Programmable Acceleration Card N6002\" }, // N6002 PF0-VF\nExample:
STATIC libopae_config_data default_libopae_config_table[] = {\n...\n{ 0xff00, 0x1234, 0xff00, 0x5678, \"libxfpga.so\", \"{}\", 0 } , // N6002 PF0\n{ 0xff00, 0x5555, 0xff00, 0x5678, \"libxfpga.so\", \"{}\", 0 } , // N6002 PF0-VF\nlibraries/plugins/xfpga/metrics/metric_utils.c. Add one entry to the switch case under enum_fpga_metrics(fpga_handle handle). The enum value used should match the enum set in step 6. Add a new condition to the if statement beginning if (((_fpga_enum_metric->hw_type == FPGA_HW_DCP_N3000). Save and exit. Example:
// DCP VC DC\ncase FPGA_HW_DCP_N3000:\ncase FPGA_HW_DCP_D5005:\ncase FPGA_HW_DCP_N6002:\ncase FPGA_HW_DCP_N5010: {\n...\nExample:
if (((_fpga_enum_metric->hw_type == FPGA_HW_DCP_N3000) ||\n(_fpga_enum_metric->hw_type == FPGA_HW_DCP_D5005) ||\n(_fpga_enum_metric->hw_type == FPGA_HW_DCP_N6002) ||\n(_fpga_enum_metric->hw_type == FPGA_HW_DCP_N5010)) &&\nlibraries/plugins/xfpga/sysfs.c. Add a new set of switch cases under enum fpga_hw_type opae_id_to_hw_type(uint16_t vendor_id, uint16_t device_id). The enum value used should match the enum value set in step 6. Save and exit. Example:
if (vendor_id == 0xff00) { switch (device_id) {\ncase 0x1234:\ncase 0x5555:\nhw_type = FPGA_HW_DCP_N6002;\nbreak;\ndefault:\nOPAE_ERR(\"unknown device id: 0x%04x\", device_id);\nlibraries/plugins/xfpga/types_int.h. Add your new enum value under enum fpga_hw_type. Save and exit. Example:
enum fpga_hw_type {\nFPGA_HW_MCP,\nFPGA_HW_DCP_RC,\nFPGA_HW_DCP_D5005,\nFPGA_HW_DCP_N3000,\nFPGA_HW_DCP_N5010,\nFPGA_HW_DCP_N6002,\nFPGA_HW_UNKNOWN\n};\nopae.cfg. Create a new entry for device \"n6002\" by copying the entry for \"n6001,\"\" substituting our new values. Add one new entry under \"configs\" for the name \"n6002.\" Save and exit. Example:
\"n6002\": {\n\"enabled\": true,\n\"platform\": \"Intel Acceleration Development Platform N6002\",\n\"devices\": [\n{ \"name\": \"n6002_pf\", \"id\": [ \"0xff00\", \"0x1234\", \"0xff00\", \"0x5678\" ] },\n{ \"name\": \"n6002_vf\", \"id\": [ \"0xff00\", \"0x5555\", \"0xff00\", \"0x5678\" ] }\n],\n\"opae\": {\n\"plugin\": [\n{\n\"enabled\": true,\n\"module\": \"libxfpga.so\",\n\"devices\": [ \"n6002_pf\" ],\n\"configuration\": {}\n},\n{\n\"enabled\": true,\n\"module\": \"libopae-v.so\",\n\"devices\": [ \"n6002_pf\", \"n6002_vf\" ],\n\"configuration\": {}\n}\n],\n\"fpgainfo\": [\n{\n\"enabled\": true,\n\"module\": \"libboard_n6000.so\",\n\"devices\": [\n{ \"device\": \"n6002_pf\", \"feature_id\": \"0x12\" },\n{ \"device\": \"n6002_vf\", \"feature_id\": \"0x12\" }\n]\n}\n],\n\"fpgad\": [\n{\n\"enabled\": true,\n\"module\": \"libfpgad-vc.so\",\n\"devices\": [ \"n6002_pf\" ],\n\"configuration\": {\n\"cool-down\": 30,\n\"get-aer\": [ \"setpci -s %s ECAP_AER+0x08.L\",\n\"setpci -s %s ECAP_AER+0x14.L\" ],\n\"disable-aer\": [ \"setpci -s %s ECAP_AER+0x08.L=0xffffffff\",\n\"setpci -s %s ECAP_AER+0x14.L=0xffffffff\" ],\n\"set-aer\": [ \"setpci -s %s ECAP_AER+0x08.L=0x%08x\",\n\"setpci -s %s ECAP_AER+0x14.L=0x%08x\" ],\n\"sensor-overrides\": [],\n\"monitor-seu\": false\n}\n}\n],\n\"rsu\": [\n{\n\"enabled\": true,\n\"devices\": [ \"n6002_pf\" ],\n\"fpga_default_sequences\": \"common_rsu_sequences\"\n}\n],\n\"fpgareg\": [\n{\n\"enabled\": true,\n\"devices\": [ \"n6002_pf\", \"n6002_vf\" ]\n}\n],\n\"opae.io\": [\n{\n\"enabled\": true,\n\"devices\": [ \"n6002_pf\", \"n6002_vf\" ]\n}\n]\n}\n},\nExample:
\"configs\": [\n\"mcp\",\n\"a10gx\",\n\"d5005\",\n\"n3000\",\n\"n5010\",\n\"n5013\",\n\"n5014\",\n\"n6000\",\n\"n6001\",\n\"n6002\",\n...\nlinux-dfl 1. Open the file drivers/fpga/dfl-pci.c. Define a list of necessary ID values at the top of the file. Use these values to add two new entries under pci_device_id cci_pcie_id_tbl[], one for PF0 and the other for PF0-VF. Save and exit. Example:
/* N6002 IDS */\n#define PCIE_DEVICE_ID_PF_N6002 0x1234\n#define PCIE_VENDOR_ID_PF_N6002 0xff00\n#define PCIE_SUBDEVICE_ID_PF_N6002 0x5678\n#define PCIE_DEVICE_ID_VF_N6002 0x5555\nExample:
static struct pci_device_id cci_pcie_id_tbl[] = {\n...\n{PCI_DEVICE_SUB(PCIE_VENDOR_ID_PF_N6002, PCIE_DEVICE_ID_PF_N6002,\nPCIE_VENDOR_ID_PF_N6002, PCIE_SUBDEVICE_ID_PF_N6002),}, //N6002 PF0\n{PCI_DEVICE_SUB(PCIE_VENDOR_ID_PF_N6002, PCIE_DEVICE_ID_VF_N6002,\nPCIE_VENDOR_ID_PF_N6002, PCIE_SUBDEVICE_ID_PF_N6002),}, //N6002 PF0-VF\n...\nThis concludes our integration our new platform with the linux-dfl driver set. Build and install the linux-dfl enabled kernel and the OPAE SDK userspace libraries as discussed in their relevant sections in the user guide linked above. If the above conditions have been met, and your N6001 board has been configured with this new \"N6002\" FIM, you should see the following output when running the command \"fpgainfo fme\" (your Bitstream ID, PR Interface ID, and Image Info entries may differ). Check that the board's display name at the top and values for Vendor/Device/SubVendor/Subdevice IDs are correct.
Intel Acceleration Development Platform N6002\nBoard Management Controller NIOS FW version: 3.11.0\nBoard Management Controller Build version: 3.11.0\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0xFF00\nDevice Id : 0x1234\nSubVendor Id : 0xFF00\nSubDevice Id : 0x5678\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010202C8AD72D7\nBitstream Version : 5.0.1\nPr Interface Id : 8df219e3-cf25-5e77-8689-f57102d54435\nBoot Page : user1\nFactory Image Info : a2b5fd0e7afca4ee6d7048f926e75ac2\nUser1 Image Info : 9804075d2e8a71a192ec89602f2f5544\nUser2 Image Info : 9804075d2e8a71a192ec89602f2f5544\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/","title":"oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#10-overview","title":"1.0 Overview","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a reference manual for platform designers wanting to enable oneAPI support on their Open FPGA Stack(OFS) platforms. The document describes essential hardware and software components required for enabling this design flow using OFS.
Please use table below as a quick reference for the contents in this document.
Table 1-1: Document Overview
Concept Description Sections Overview, Prerequisites Covers the flow of the document, important terms and prerequisite knowledge for this manual * 1.0 Overview * 1.1 About this Document * 1.2 Terminology * 1.3 Prerequisites Introduction to the oneAPI FPGA Acceleration Flow using Open FPGA Stack Covers the introduction to accelerating oneAPI applications on an FPGA platform design using Open FPGA Stack. This gives an overview of the complete stack consisting of the oneAPI application and explains the high level tool . it also covers an introduction to what the oneAPI Accelerator Support Package 1.4 Introduction to oneAPI on Open FPGA Stack(OFS) * 1.5 Introduction to oneAPI Accelerator Support Package(ASP) oneAPI Accelerator Support Package Design Fundamentals This covers the information on design elements required to ensure the FPGA hardware design is functional with the oneAPI base toolkit. It covers both the components required in hardware design as well as in the software layer that oneAPI runtime requires * 2.0 XML Files in oneAPI ASP * 2.1 board_spec.xml File * 2.2 board_env.xml File * 3.0 oneAPI ASP Hardware * 3.1 Host to External Memory Interface(EMIF) * 3.2 Host to Kernel Interface * 3.3 Kernel to External Memory Interface * 4.0 oneAPI ASP Software * 4.1 Memory Mapped Device(MMD) Layer * 4.2 Board Utilities oneAPI ASP Implementation on OFS Reference Platforms Provides detailed explanation of the oneAPI ASP design for OFS reference platforms. These sections cover information about the harware as well as software design files in theoneapi-asp repository * 5.0 oneapi-asp Implementation Details * 5.1 oneapi-asp Directory Structure * 5.2 oneapi-asp Build Flow * 5.3 oneapi-asp Hardware Implementation * 5.4 oneapi-asp Memory Mapped Device(MMD) Layer Implementation * 5.5 oneapi-asp Utilities Implementation Appendix This currently has information about additional features provided in the oneapi-asp repository. Appendix A covers debug capabilities in the MMD. Appendix B has information about a new functionality added in the oneapi-asp tag ofs-2024.2-1 called oneAPI ASP Editor, enabling a GUI based approach to design oneAPI ASPs * Appendix A: Memory Mapped Device(MMD) Layer Debug Variables * Appendix B: oneAPI Accelerator Support Package(ASP) Editor Note: Table 1-1 in oneAPI Accelerator Support Package (ASP): Getting Started User Guide lists OFS reference platforms.
For more information about developing application kernels for FPGA using oneAPI Base Toolkit (Base Kit) refer to Intel\u00ae FPGA Add-on for oneAPI Base Toolkit webpage.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#12-terminology","title":"1.2 Terminology","text":"This table defines some of the common terms used when discussing OFS.
Table 1-2: Terminology
Term Abbreviation Description Open FPGA Stack OFS A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. Accelerator Functional Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. FPGA Interface Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. High Level Design HLD For the purpose of this guide, this term refers to designing with High Level Design tools like oneAPI Base Toolkit (Base Kit). oneAPI Accelerator Support Package oneAPI ASP A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and other OFS hardware, software components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Intel\u00ae FPGA Basic Building Blocks BBB Basic Building Blocks (BBB) for Altera\u00ae FPGAs is a suite of application building blocks and shims like Memory Properties Factory (MPF). BBB Memory Properties Factory BBB MPF Intel\u00ae FPGA BBB MPF block provides features like virtual to physical address (VTP), ordering read responses, read/write hazard detection, and masked (partial) writes.oneapi-asp uses MPF VTP feature. Open Programmable Acceleration Engine Software Development Kit OPAE SDK A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. Platform Interface Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Device Feature List DFL A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. Best Known Configuration BKC The exact hardware configuration that the solution has been optimized and validated against. SYCL - SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL\u2122) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Installable Client Driver ICD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports the OpenCL ICD extension from the Khronos Group\u2122. The OpenCL ICD extension allows you to have multiple OpenCL implementations on your system. With the OpenCL ICD Loader Library, you may choose from a list of installed platforms and execute OpenCL API calls that are specific to your OpenCL implementation of choice. FPGA Client Driver FCD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports FPGA Client Driver(FCD) extension. FCD allows the runtime to automatically find and load the oneAPI ASP libraries at host run time Note: oneapi-asp was referred to as ofs-hld-shim in OFS (for Agilex\u00ae 7 FPGA & Stratix\u00ae 10 FPGA) and OpenCL AFU Shim (ofs-opencl-afu-shim) in OFS early access(EA) release (for Stratix\u00ae 10 FPGA with Intel\u00ae FPGA PAC D5005 as reference platform).
The content in this manual requires readers to be familiar with:
In addition to above, developers must be familiar with the following tools & concepts:
The Intel\u00ae oneAPI Base Toolkit (Base Kit) is a core set of tools and libraries for developing high-performance, data-centric applications across diverse architectures (CPUs, GPUs and FPGAs). It features an industry-leading C++ compiler that implements SYCL, an evolution of C++ for heterogeneous computing.
Figure 1-1 shows the high-level representation of oneAPI application developers using FPGAs for acceleration. The runtime flow consists of a host code running on a processor and an application kernel code running on an FPGA. Open FPGA Stack enables vendors to enable support for this flow on their platforms.
Figure 1-1: oneAPI Application on OFS Platforms
oneAPI Base Toolkit (Base Kit) consists of a compiler and runtime environment. The compiler converts a SYCL kernel (FPGA application code) into a hardware circuit. This hardware circuit requires additional logic to communicate with the runtime and FPGA board peripherals. This additional logic is provided by oneAPI Accelerator Support Package(oneAPI ASP). oneAPI ASP consists of hardware components that enable this generated hardware circuit to communicate with the host processor as well as software components that enable the runtime to identify and communicate with the kernel.
Figure 1-2 shows the workload design steps and steps in which the oneAPI Base Toolkit (Base Kit) requires oneAPI ASP as input. For more information about workload development and how workload developers target a specific platform during compilation, refer to Intel oneAPI Programming Guide and Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs. The next section introduces oneAPI ASP.
Figure 1-2: High Level Design Flow for FPGAs with oneAPI Base Toolkit (Base Kit)
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#15-introduction-to-oneapi-accelerator-support-packageasp","title":"1.5 Introduction to oneAPI Accelerator Support Package(ASP)","text":"As mentioned in previous section, oneAPI ASP is a collection of hardware and software components that interface with the hardware circuit generated by the oneAPI compiler. The hardware circuit generated by the oneAPI compiler from a oneAPI kernel is referred to as the kernel system. While the kernel system consists of logic controlled by the workload developer's specifications, the kernel system interfaces are generated by the oneAPI compiler based on specifications provided by the oneAPI ASP designer. These specifications are input to the compiler using XML files (discussed in section 2.0).
Note: All the interfaces generated by the oneAPI compiler are Avalon\u00ae Interfaces.
Figure 1-3: Kernel System Interfaces
Figure 1-3 shows a high-level representation of an OFS hardware design and interfaces to/from kernel_system. The numbered arrows depict the following:
oneAPI ASP hardware components can be divided into 3 categories:
In addition to the hardware components, a software layer is required for handling I/O operations between oneAPI runtime and the board. The oneAPI ASP software layer can be divided into 2 categories:
The MMD uses API provided by OPAE SDK to communicate with the device. The FPGA driver is provided by the linux-DFL kernel driver.
Figure 1-4 shows how the above oneAPI ASP components tie into Open FPGA Stack.
Figure 1-4: Open FPGA Stack (OFS) components
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#20-xml-files-in-oneapi-asp","title":"2.0 XML Files in oneAPI ASP","text":"The kernel system interfaces generated by the oneAPI compiler are based on specifications provided by oneAPI ASP developer. An XML file, called board_spec.xml is used to pass the specifications to the oneAPI compiler. oneAPI ASP developers must create this XML file for their boards.
In addition to board_spec.xml, the oneAPI Base Toolkit (Base Kit) relies on another XML file called board_env.xml to get information about the board environment. The board_env.xml file helps the runtime setup board installation.
The next section explains the contents of board_spec.xml. Section 2.2 covers contents of board_env.xml file.
board_spec.xml File","text":"A typical board_spec.xml structure is shown in Fig 2-1. In addition to kernel system interfaces, the board_spec.xml file is also used to specify other compilation details like Quartus\u00ae Prime Pro Edition Software version used in platform design, compilation scripts to help control Quartus\u00ae software compile flows, FPGA resource utilization details.
Elements of board_spec.xml file are summarized in table 2-1. Each element has additional attributes and parameters. Details are covered in respective sections for each element.
Figure 2-1: board_spec.xml File Structure
Table 2-1: Elements of board_spec.xml
Element Use of Element Attributes board Used to specify a name for the board and the version of Quartus\u00ae Prime Pro Edition Software used to develop the platform design. This board name is used to identify the board design to be compiled and must match the name of the directory in which board_spec.xml resides. version, name compile Used to describe different compile flows project, revision, qsys_file, generic_kernel, generate_cmd, synthesize_cmd, auto_migrate device Used to specify the FPGA device model file for the FPGA part on the board. device_model, used_resources global_mem Different attributes in this element are used to provide details about the external memory used as the global memory for the FPGA oneAPI kernel/application. name, max_bandwidth, interleaved_bytes, config_addr, default, interface host Used to specify the offset at which the kernel resides. kernel_config interfaces Used to specify control signals to oneAPI kernels interface, kernel_clk_reset channels Used to describe interface to stream data directly between kernels and I/O interfaceThe compiler expects a separate board_spec.xml file for every board variant a platform supports. Board variants are different hardware design implementations for the same platform, a oneAPI ASP can have multiple board variants. A oneAPI kernel developer can select the board variant suitable for their application at compile time.
A board_spec.xml file must be located at the top most level of each board variant's hardware directory (the hardware directory is specified by board_env.xml, please refer to section 2.2 for details on hardware element). For example, a separate board_spec.xml file for each board variant for OFS reference platforms is located in oneapi-asp/Platform-Name/hardware/Board-Variant/ directory, where Platform-Name is n6001, fseries-dk, iseries-dk for OFS targeting Agilex\u00ae 7 FPGA and d5005 for OFS targeting Stratix\u00ae 10 FPGA.
The board element of the board_spec.xml file provides the Quartus\u00ae Prime Pro Edition Software version and the name of the board.
Table 2-2: Attributes for the board Element
Example below shows the board element populated for a board designed with Quartus\u00ae Prime Pro Edition Software version 22.3 and board variant named \"Agilex_brd1\".
Note: A board variant name is different from a platform directory name. Please see Note in section 2.2 for more information on board variants.
Figure 2-2: board Element
Depending on the application requirements, the design may have different compilation flows and different design settings for each flow (for example, there can be a flat flow without partial reconfiguration support or a flow with partitions in the design to enable partial reconfiguration). Designers can control the flow and its settings using scripts. To allow selection of compile flow during application compile & to describe control of Quartus\u00ae software compilation as well as registration, automigration, the compile element of the board_spec.xml file and its associated attributes and parameters are used.
Table 2-3: Attributes for compile Element
-Xsbsp-flow option. project Name of the Quartus\u00ae software project file (.qpf) that the Quartus\u00ae Prime Pro Edition Software intends to compile. revision Name of the revision within the Quartus\u00ae software project that the Quartus\u00ae Prime Pro Edition Software compiles to generate the final bitstream. qsys_file Name of the Platform Designer file into which the oneAPI kernel is embedded. You have the option to assign a value of \"none\" to qsys_file if you do not require the Quartus\u00ae Prime Pro Edition Software to create a top-level .qsys file for your design. In this scenario, oneAPI compiler adds a .qip file into the Quartus\u00ae software project. In this case, the custom oneAPI ASP must manually instantiate the generated HDL entity (generated entity is in the kernel_system.v file). generic_kernel Set this value to 1 if you want the offline compiler to generate a common Verilog interface for all compilations. This setting is necessary in situations where you must set up design partitions around the kernel, such as in the Configuration via Protocol (CvP) flow. generate_cmd Command required to prepare for full compilation, such as to generate the Verilog files for the Platform Designer system into which the oneAPI kernel is embedded. synthesize_cmd Command required to generate the fpga.bin file from the Custom Platform. Usually, this command instructs the Quartus\u00ae Prime Pro Edition Software to perform a full compilation. auto_migrate *platform_type\u2014Choose this value based on the value referenced in the Reference Platform from which you derive your Custom Platform. Valid values are a10_ref, s10_ref, and none. *include fixes\u2014Comma-separated list of named fixes that you want to apply to the Custom Platform. *exclude fixes\u2014Comma-separated list of named fixes that you do not want to apply to the Custom Platform. Example below shows a populated compile element for a sample Quartus\u00ae software Project called ofs.qpf, the Quartus\u00ae software revision to be compiled is called asp (asp.qsf). In this example, the compiler generates the kernel system (entity is called kernel_system) and this entity is instantiated manually in the Quartus\u00ae software project (e.g. in a file called kernel_wrapper.v), hence qsys_file is set to \"none\". The synthesize_cmd points to a script \"compile.tcl\" located in the same directory as the board_spec.xml, compile script performs all necessary system generation and compile steps for generation of final bitstream. The project directory snippet below is for demonstration only. The compile flow is named \"demo_flow\".
There can be multiple compile elements for the different compilation flows that a platform designer wishes to enable in their platform (e.g. different revisions with different Quartus\u00ae software settings or a PR revision).
Figure 2-3: compile Element
A device model(DM) file is an XML file that has the total resources on the device (i.e. ALMs, FFs, DSPs, RAMs). This is required for any FPGA part used in a oneAPI design. Most device model files are provided as part of the oneAPI Base Toolkit (Base Kit) installation ($INTELFPGAOCLSDKROOT/share/models/dm, where INTELFPGAOCLSDKROOT is set by the setvars.sh environment setup script provided by oneAPI toolkit). If the device model file for your part number is not included in $INTELFPGAOCLSDKROOT/share/models/dm, it must be created and placed in the same folder as board_spec.xml. A new device model file can be created using existing files as reference.
The device model file name must be specified in the device_model attribute of device element. The used_resources attribute is used to specify the resources being utilized by the oneAPI ASP and peripheral IPs. The utilization by non-kernel logic is calculated during platform design. The compiler utilizes the total resources from device model file and utilized resources in used_resources section to estimate the available resources for application kernel.
Table 2-4: Attributes for device Element
Example below shows the device element added for a Agilex\u00ae 7 FPGA based platform with device model file named \"agfb014r24a2e2vr0_dm.xml\". The number of used_resources are for demonstration purposes and are not to be used by oneAPI ASP developers.
Figure 2-4: device Element
Note: This is different from the interfaces element discussed in upcoming sections. In the board_spec.xml file, each global memory, channel or kernel connection is comprised of individual interfaces. For the global_mem, channels, and interfaces XML elements, an interface attribute must be included to specify the corresponding parameters for each connection.
Table 2-5: Parameters for interface attribute
interface attribute or as a separate element in interface attribute. See section 2.1.4.1 for more information. All type * For global_mem: set to agent. * For channels: - Set to streamsource for a stream source that provides data to the kernel.- Set to streamsink for a stream sink interface that consumes data from the kernel.* For interfaces: set to either host, irq, or streamsource. All width * For global_mem: width of the memory interface in bits.* For channels: number of bits in the channel interface.* For interfaces: width of the kernel interface in bits. All waitrequest_allowance * For global_mem: [Optional] Amount of Avalon\u00ae-MM waitrequest allowance supported on the agent interface (that is, kernel-facing interface) of the clock-crossing bridge that spans between the memory and the kernel clock domains.* For kernel_cra: [Optional] Amount of Avalon\u00ae-MM waitrequest allowance that the kernel_cra agent interface must support.This parameter defaults to 0 if you do not specify it in the board_spec.xml file. A value of 0 indicates that this waitrequest allowance feature is disabled. All maxburst Maximum burst size for the agent interface.Attention: The value of width \u00f7 8 x maxburst must be less than the value of interleaved_bytes. global_mem address Starting address of the memory interface that corresponds to the host interface-side address.For example, address 0 should correspond to the bank1 memory host from the Memory Bank Divider. In addition, any non-zero starting address must abut the end address of the previous memory. global_mem size Size of the memory interface in bytes.* For global_mem: The sizes of all memory interfaces should be equal.* For interfaces: interface can have variable sizes. global_meminterfaces > Note: Support for size parameter for interface attribute is available in oneAPI Base Toolkit (Base Kit) version 2024.0 and beyond. latency_type If the memory interface has variable latency, set this parameter to average to signify that the specified latency is considered the average case. If the complete kernel-to-memory path has a guaranteed fixed latency, set this parameter to fixed. global_mem chan_id A string used to identify the channel interface. The string may have up to 128 characters. channels clock For the streamsource kernel interface type, the parameter specifies the name of the clock that the snoop stream uses. Usually, this clock is the kernel clock. interfaces Example for how the interface attribute is used in global_mem and interfaces elements is covered in section for these elements respectively.
As mentioned in Table 2-5, port parameter can be defined either inline in interface attribute or as a separate element in interface attribute. The definition method to use depends on the direction of the port.
read or write, it must be a separate element in interface attribute. readwrite, port must be inline with the port name in interface attribute. No direction specification is required. Table below shows the port element attributes.
Table 2-6: port parameter
Kernel Interface Platform Designer component. For example, kernel_cra is the Avalon\u00ae-MM interface, and kernel_irq is an interrupt. direction Direction of the port. Valid values are: * \"r\" : Indicates read * \"w\" : Indicates write Snippet below shows the inline and separate element definitions of port parameter.
Note: Direction specification for port is available in oneAPI Base Toolkit (Base Kit) versions 2024.0 and beyond. For versions prior to oneAPI Base Toolkit (Base Kit) version 2024.0, only the default inline definition of port parameter is supported.
Examples for global_mem and interfaces elements in sections below use the inline definition of port.
The global_mem element of the board_spec.xml file is used to provide information on the memory interfaces that connect to the kernel.
Note: For each global memory that the kernel accesses, you must include one interface element that describes its characteristics. The different attributes for global_mem element are discussed in table below.
Table 2-7: Attributes for global_mem Element
Memory Bank Divider must match the exported kernel agent interfaces in this respect (refer to section 3.1.1 for information about Memory Bank Divider) For block RAM, interleaved_bytes equals the width of the interface in bytes. config_addr The address of the ACL Mem Organization Control Platform Designer component (mem_org_mode) that the host software uses to configure memory. You may omit this attribute if your board has homogeneous memory; the software uses the default address (0x18) for this component. If your board has heterogeneous memory, there is a mem_org_mode component in the board system for each memory type. Enter the config_addr attribute and set it to the value of the base address of the mem_org_mode component(s). default Include this optional attribute and assign a value of 1 to set the global memory as the default memory interface. The default memory must start at address 0x0. If you do not implement this attribute, the first memory type defined in the board_spec.xml file becomes the default memory interface. interface See the interface section above for the parameters you must specify for each interface. allocation_type A list that specifies which USM allocator is used to allocate from the global_mem element. Values allowed in this list are host, shared, and device. The following conditions apply: If there are multiple global_mem elements with the same allocation_type attribute, the first allocation_type attribute in the board_spec.xml is assumed to be the one used by the specified allocator. If there is a single global_mem element with multiple allocation_type attributes, this indicates that allocations of the specified types use this global_mem interface. [Legacy support] If you have not specified the allocation_type attribute, it is assumed that all global memory interfaces have the device allocation_type. Example below shows a global_mem element configuration for a kernel system connected to four 4GB DDR4 memory banks. The DDR4 interface is 64 bit operating at 1200MHz. Note that the name of the platform designer system name is board.qsys. As mentioned in description for interleaved_bytes in table above, the Memory Bank Divider configuration ensures that the host interface matches the interleaved_bytes setting (i.e. 512 bits x 64 burst size = 4096 bytes). For information on waitrequest_allowance, refer to section 2.1.4 on interface attribute.
Note: More details on the Memory Bank Divider and the Clock Crossing Bridges is covered in section 3.0
Figure 2-6: Memory Connection Example Block Diagram and Corresponding global_mem Element in board_spec.xml
A board can have a single memory bank, multiple memory banks of the same type (e.g. 4 banks of DDR4) or different banks of different types.
The partitioning of memory for oneAPI kernel developers is explained in the Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs. The global memory configuration required by an application kernel must match the configuration in board_spec.xml as the compiler uses this information to generate a suitable architecture for the application. The different memory configurations are
For boards with multiple memory banks of the same type, designers can configure these as a single contiguous global memory region. This is done by specifying each memory interface within a single global_mem element. Figure 2-6 showed 4 DDR4 memory banks configured as a single global memory region.
With this configuration, FPGA application developers have the option to use contiguous memory region in an interleaved or a non-interleaved fashion. Even with contiguous memory regions, kernel developers can partition data buffers across the banks/memory channels. Please refer to Global Memory Access Optimization section in Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs for more details on these partitioning techniques.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#21512-heterogeneous-memory","title":"2.1.5.1.2 Heterogeneous Memory","text":"For boards with different memory technologies, designers must specify each type of memory that the kernel needs to access as a separate global memory.
Figure 2-7 shows heterogeneous configurations and the global_mem element structure for two different types of memories (QDR, DDR4). The global_mem element in example below also demonstrates use of the default attribute. It is set to \"1\" for the DDR4 memory banks, indicating to the oneAPI compiler that the default global memory for the kernel is DDR4.
Figure 2-7: Heterogeneous Memory Example Block Diagram and Corresponding global_mem Element in board_spec.xml
For applications that require USM support, the board_spec.xml must specify host and device memories in a heterogeneous manner. The allocation_type must be host for global memory region on the host processor. The allocation_type must be set to device for global memory on the FPGA board. Example below extends the board_spec.xml snippet in figure 2-6 to add a global_mem element for the kernel system to host processor memory interface.
Figure 2-8: global_mem Element Example for Unified Shared Memory(USM)
The host element of the board_spec.xml file provides information on the interface from the host to the kernel. Figure below shows an example of host element.
Figure 2-9: host Element Example
Table 2-8: Attributes for the host Element
kernel_cra host on the kernel_interface module.* start: the starting address of the kernel. Normally, this attribute has a value of 0 because the kernel_cra host should not host anything except kernels.* size: keep this parameter at the default value of 0x0100000."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#217-interfaces-element","title":"2.1.7 interfaces Element","text":"The interfaces element of the board_spec.xml file describes the kernel interfaces that connect to application kernels and control kernel behavior. For this element, include one of each interface of types host, irq and streamsource. Refer to the interface section for the parameters you must specify for each interface. In addition to the host, irq, and streamsource interfaces, if your design includes a separate Platform Designer subsystem containing the board logic, the kernel clock and reset interfaces exported from it are also part of the interfaces element. Specify these interfaces with the kernel_clk_reset attribute and its corresponding parameters.
Figure below shows example of interfaces element.
Figure 2-10: interfaces Element Example
Table 2-9: Parameters for the kernel_clk_reset Attribute
Note: Name the kernel clock and reset interfaces in the Platform Designer connection format (that is, .). For example: board.kernel_clk"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#218-channels-element","title":"2.1.8 channels Element","text":"
The channels element provides channels for streaming data directly between kernel and I/O. Each channel (implemented using Avalon-ST specification) must be connected to the kernel via the interface attribute. The channel interface only supports data, and valid and ready Avalon-ST signals. The I/O channel defaults to 8-bit symbols and big-endian ordering at the interface level.
Figure below shows an example of channels element for a single channel with a width of 64 bits. The chan_id attribute identified helps identify the port in the generated kernel_system. Refer to section 2.1.4 for more information about the interface attribute parameters. Additional interface attributes can be added for additional channels.
Figure 2-11: channels Element Example
For more information about kernel development using channels, refer to I/O Pipes section in Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#22-board_envxml-file","title":"2.2board_env.xml File","text":"The board_env.xml file is used by the oneAPI toolkit to set up the board installation that enables the compiler to target a specific accelerator platform. The board_env.xml file must be located in the top most level of the oneAPI ASP for each platform. For example, the board_env.xml for oneAPI ASP for OFS reference platforms is located in the oneapi-asp/Platform-Name folder, where Platform-Name is n6001, fseries-dk, iseries-dk for OFS targeting Agilex\u00ae 7 FPGA and d5005 for OFS targeting Stratix\u00ae 10 FPGA.
A sample board_env.xml file is shown below. Table 2-10 explains the elements of this file.
Figure 2-12: board_env.xml File Structure
Table 2-10: Specifications of XML Elements and Attributes in the board_env.xml File
platform element must be specified for each supported OS for the oneAPI ASP platform mmdlib A string that specifies the path to the MMD library of your oneAPI ASP. To load multiple libraries, specify them in an ordered, comma-separated list. The host application will load the libraries in the order that they appear in the list> Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if MMD library is located inside linux64/lib folder in oneAPI ASP, the path would be %b/linux64/lib/libintel_opae_mmd.so linkflags A string that specifies the linker flags necessary for linking with the MMD layer available with the board> Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if MMD library is located inside linux64/lib folder in oneAPI ASP, the path would be %b/linux64/lib. linklibs A string that specifies the libraries the oneAPI runtime must link against to use the MMD layer available with the board utilbindir Directory in which the runtime expects to locate board utility executables (i.e. install, uninstall, program, diagnose, flash) > Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if the utilities are located in linux64/libexec folder in oneAPI ASP, the path would be %b/linux64/libexec"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#30-oneapi-asp-hardware","title":"3.0 oneAPI ASP Hardware","text":"The oneAPI compiler generates the kernel system interfaces based on specifications provided by the oneAPI ASP developer in the board_spec.xml file. The kernel system interfaces with the rest of the oneAPI ASP RTL as shown in figure 1-3.
Figure 1-3 shows 5 different paths, summarized below:
Please note that the kernel system generated by oneAPI compiler has Avalon\u00ae interfaces. OFS FIM has AXI interfaces. Additional logic blocks from Platform Interface Manager are used to handle protocol conversions. Please refer to section 5.3.1 for more details on PIM. The next few sections cover some of the important IP components required to enable kernel communications with host and board peripherals. More design implementation details are covered in section 5.0.
The host to EMIF datapath consists of a PCIe Subsytem(SS), EMIF Subsystem located in the FIM and a Direct Memory Access(DMA) engine in the oneAPI ASP.
PCIe Subsystem(SS) has the PCIe IP and additional logic to handle PCIe packet format and routing. FIM handles routing signals received from host to the user design located in a region referred to as Accelerator Functional Unit(AFU) (the Kernel system resides in the AFU).
Note: For more information about the PCIe SS, please refer to Intel FPGA IP Subsystem for PCI Express IP User Guide
The External Memory Interface Subsystem (EMIF SS) consists of EMIF IP and additional logic for handling transfers between AFU and on-board memories.
Note: For more information about the EMIF SS, please refer to Memory Subsystem Intel FPGA IP User Guide
Large buffers of data are usually transferred between host and on-board memory in oneAPI applications. This necessitates a Direct Memory Access(DMA) Engine between host and on-board memory. In oneAPI ASP designs for OFS reference platform, this DMA engine is placed in the AFU region.
As described in section 2.1.5.1, there are different configurations for memories on board. In addition to above, figure 1-3 also shows an additional IP in the host to memory datapath, called Memory Bank Divider. This IP is used for handling one of the most commonly used configurations, i.e. configuring multiple memory banks of same type as a contiguous memory region. In this case, the kernel has a contiguous view of the memory and data can be interleaved across the different memory channels. The host must also have the same view of the memory in order to ensure read and write transactions from correct addresses.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#311-memory-bank-divider","title":"3.1.1 Memory Bank Divider","text":"The Memory Bank Divider is a oneAPI ASP IP component that takes an incoming request from the host interface on the Avalon\u00ae-MM agent port and routes it to the appropriate bank host port. This component must reside on the path between the host and the global memory interfaces. In addition, it must reside outside of the path between the kernel and the global memory interfaces.
The following image shows the IP interfaces. Table 3-2 provides more information on the interface signals.
Figure 3-1: Memory Bank Divider IP
This IP is currently made available as part of the oneapi-asp located in oneapi-asp/common/hardware/common/build/ip folder (memory_bank_divider_hw.tcl). The IP can be instantiated in a top level system by passing values to the parameters as shown in snippet below. Table 3-1 provides more information on the parameters. The oneAPI ASP for OFS reference platforms instantiates this IP in a similar way. Refer to section 5.0 for more information on implementation details for oneAPI ASP for OFS reference platforms.
\n add_instance memory_bank_divider memory_bank_divider <version>\n set_instance_parameter_value memory_bank_divider {NUM_BANKS} {number-of-memory-banks}\n set_instance_parameter_value memory_bank_divider {SEPARATE_RW_PORTS} {true/false-value}\n set_instance_parameter_value memory_bank_divider {PIPELINE_OUTPUTS} {true/false-value}\n set_instance_parameter_value memory_bank_divider {SPLIT_ON_BURSTBOUNDARY} {true/false-value}\n set_instance_parameter_value memory_bank_divider {DATA_WIDTH} {data-width}\n set_instance_parameter_value memory_bank_divider {ADDRESS_WIDTH} {total-addressable-width}\n set_instance_parameter_value memory_bank_divider {BURST_SIZE} {burst-size}\n set_instance_parameter_value memory_bank_divider {MAX_PENDING_READS} {max-pending-reads}\n set_instance_parameter_value memory_bank_divider {ASYNC_RESET} {value}\n set_instance_parameter_value memory_bank_divider {SYNCHRONIZE_RESET} {value}\n Table 3-1: Parameter Settings for the Memory Bank Divider Component
Parameter Name in IP TCL Parameter Description NUM_BANKS Number of banks Number of memory banks for each of the global memory types included in your board system. A single memory bank divider has the following allowed values: 1,2,4,8 for this parameter. SEPARATE_RW_PORTS Separate read/write ports Enable this parameter so that each bank has one port for read operation and one for write operation. PIPELINE_OUTPUTS Add pipeline stage to output Enable this parameter to allow for potential timing improvements. DATA_WIDTH Data Width Width of the data bus to the memory in bits. ADDRESS_WIDTH Address Width (total addressable) Total number of address bits necessary to address all global memory. BURST_SIZE Burst size (maximum) Set to a value equal to interleaved_bytes/(width/8), where interleaved_bytes and width are defined in the interface attribute of the global_mem element in theboard_spec.xml file. MAX_PENDING_READS Maximum Pending Reads Maximum number of pending read transfers the component can process without asserting a waitrequest signal. Recommended value is 64 if ASP has two global memory banks or fewer and 128 if ASP has four or more global memory banks. CAUTION: A high Maximum Pending Reads value causes Platform Designer to insert a deep response FIFO buffer, between the component's host and agent, that consumes a lot of device resources. It also increases the achievable bandwidth between host and memory interfaces. SPLIT_ON_BURSTBOUNDARY Split read/write bursts on burst word boundary Enable splitting of read and write bursts on burst word boundary. Enable this parameter if the Number of banks parameter value is greater than 1, and the burst reads and writes that the host controller sends to the agent port crosses burst word boundary. Table 3-2: Signals and Ports for the Memory Bank Divider Component
Signal or Port Description clk The bank divider logic uses this clock input. If the IP of your host and memory interfaces have different clocks, ensure that clk clock rate is not slower than the slowest of the two IP clocks. reset The reset input that connects to the board power-on reset. s The agent port that connects to the host interface controller. kernel_clk The kernel_clk drives this clock input kernel_reset The kernel_reset output from theKernel Interface IP drives this reset input. acl_asp_snoop Export this Avalon\u00ae Streaming (Avalon\u00ae-ST) source. In the board_spec.xml file, under interfaces, describe only the snoop interface for the default memory (acl_internal_snoop). If you have a heterogeneous memory design, perform these tasks only for the Memory Bank Divider component associated with the default memory. Important: The memory system you build in HW tcl (e.g. ddr_board_hw.tcl in oneapi-asp for OFS reference platforms. Refer to section 5 for more details on ddr_board_hw.tcl) alters the width of acl_asp_snoop. You must update the width of the streamsource interface within the channels element in the board_spec.xml file to match the width of acl_asp_snoop. In the board_spec.xml file, update the width of the snoop interface (acl_internal_snoop) specified with the streamsource kernel interface within the interfaces element. Updating the width ensures that the global_mem interface entries in board_spec.xml match the characteristics of the bankN Avalon\u00ae-MM hosts from corresponding Memory Bank Divider component for the default memory. acl_asp_memorg_host This conduit connects to the acl_asp_memorg_host interface of the Kernel Interface IP.> Note: Signal present if Number of banks > 1. bank1, bank2, ..., bank8 The number of memory hosts available in the Memory Bank Divider depends on the number of memory banks that were included when the unit was instantiated. Connect each bank with each memory interface in the same order as the starting address for the corresponding kernel memory interface specified in the board_spec.xml file. For example, global_mem interface that begins at address 0 must correspond to the memory host in bank1 from the Memory Bank Divider."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#32-host-to-kernel-interface","title":"3.2 Host to Kernel Interface","text":"The host exchanges control signals with kernel with the help of an additional oneAPI ASP IP . The control signals coming from the host are on a different clock domain (PCIe clock) while the kernel runs on different clock frequency . The Kernel Interface IP handles the clock domain crossing for these control signals as well as handles communication with kernel CSR, interrupts and generates the reset for oneAPI kernel. All oneAPI ASP designs must instantiate Kernel Interface IPs to ensure the kernel functions correctly.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#321-kernel-interface","title":"3.2.1 Kernel Interface","text":"The Kernel Interface is an oneAPI ASP component that allows the host interface to access and control the oneAPI kernel.
The following image shows the IP interfaces. Table 3-4 provides more information on the interface signals.
Figure 3-2: Kernel Interface IP
This IP is currently made available as part of the oneapi-asp located in oneapi-asp/common/hardware/common/build/ip folder (kernel_interface_hw.tcl). The IP can be instantiated in a top level system by passing values to the parameters as shown in snippet below. Table 3-3 provides more information on the parameter. The oneAPI ASP for OFS reference platforms instantiates this IP in a similar way. Refer to section 5.0 for more information on implementation details for oneAPI ASP for OFS reference platforms.
\n add_instance kernel_interface kernel_interface <version>\n set_instance_parameter_value kernel_interface {NUM_GLOBAL_MEMS} {value}\n Table 3-3: Parameter Settings for the Kernel Interface Component
Parameter Name in IP TCL Parameter Description NUM_GLOBAL_MEMS Number of global memory systems Number of global memory types in your board design.Table 3-4: Signals and Ports for the Kernel Interface Component
Signal or Port Description clk The clock input used for the host control interface. The clock rate of clk can be slow. reset This reset input resets the control interface. It also triggers the kernel_reset signal, which resets all kernel logic. ctrl Use this agent port to connect to the host interface. This interface is a low-speed interface with which you set kernel arguments and start the kernel's execution. kernel_clk kernel clock drives this clock input. kernel_cra This Avalon\u00ae-MM host interface communicates directly with the kernels generated by the oneAPI compiler. Export the Avalon\u00ae-MM interface to the Kernel Interface and name it in the board_spec.xml file. sw_reset_in When necessary, the host interface resets the kernel via the ctrl interface. If the board design requires a kernel reset, it can do so via this reset input. Otherwise, connect the interface to a global power-on reset. kernel_reset Use this reset output to reset the kernel and any other hardware that communicates with the kernel. Warning: This reset occurs between the MMD open and close calls. Therefore, it must not reset anything necessary for the operation of your MMD. sw_reset_export This reset output is the same as kernel_reset, but it is synchronized to the clk interface. Use this output to reset logic that is not in the kernel_clk clock domain but should be reset whenever the kernel resets. acl_asp_memorg_host The memory interfaces use these signals. Based on the number of global memory systems you specify in theKernel Interface component parameter editor, the Quartus\u00ae Prime Pro Edition Software creates the corresponding number of copies of this signal, each with a different hexadecimal suffix. Connect each signal to the Memory Bank Divider component associated with each global memory system (for example, DDR). Then, list the hexadecimal suffix in the config_addr attribute of the global_mem element in the board_spec.xml file. kernel_irq_from_kernel An interrupt input from the kernel. This signal is exported and named in the board_spec.xml file. kernel_irq_to_host An interrupt output from the kernel. This signal connects to the host interface."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#33-kernel-to-external-memory-interface","title":"3.3 Kernel to External Memory Interface","text":"The kernel system masters the interface from kernel to external memory. oneAPI compiler generates kernel system memory interface logic (e.g. Load-Store Unit) according to the global memory configuration and interface specifications in board_spec.xml file. The kernel system operates at kernel clock, hence, oneAPI ASP developers must handle clock domain crossing from kernel to EMIF clock domain.
For implementation details for all datapaths discussed above, please refer to section 5.3.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#40-oneapi-asp-software","title":"4.0 oneAPI ASP Software","text":"The software components of oneAPI ASP consist of the Memory Mapped Device(MMD) layer and the board utility routine required by runtime.
Section 4.1 introduces MMD layer and section 4.2 explains board utilities.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#41-memory-mapped-devicemmd-layer","title":"4.1 Memory Mapped Device(MMD) Layer","text":"The oneAPI ASP Memory Mapped Device (MMD) layer sits in between the oneAPI runtime and OPAE SDK and provides a set of API for device communication and control. The runtime calls into the MMD API for various operations like opening a handle to the device, allocating memory etc.
Note: For more information about the FPGA runtime, please refer to FPGA Runtime documentation here.
A header file, called aocl_mmd.h, has the list of MMD API calls that must be implemented by oneAPI ASPs. From the perspective of the caller, below is typical MMD API lifecycle:
Table below summarizes all APIs listed in aocl_mmd.h.
Table 4-1: Summary of MMD API from aocl_mmd.h
aocl_mmd_open() call aocl_mmd_get_info Obtain information about the board specified in the requested_info_id argument (refer to section 4.1.2 for more information) aocl_mmd_open Open and initialize the specified device aocl_mmd_close Close an opened device via its handle aocl_mmd_set_interrupt_handler Set the interrupt handler for the opened device aocl_mmd_set_device_interrupt_handler Sets the device interrupt handler for opened device, interrupt handler is called to notify runtime of any exceptions aocl_mmd_set_status_handler Set the operation status handler for the opened device aocl_mmd_yield The aocl_mmd_yield function is called when the host interface is idle. The host interface might be idle because it is waiting for the device to process certain events aocl_mmd_read Read operation on a single interface aocl_mmd_write Write operation on a single interface aocl_mmd_copy Copy operation on a single interface aocl_mmd_hostchannel_create Creates a channel interface aocl_mmd_hostchannel_destroy Destroys channel interface aocl_mmd_hostchannel_get_buffer Provides host with pointer used to read/write from channel interface aocl_mmd_hostchannel_ack_buffer Acknowledges read/write from channel aocl_mmd_program Reprogram operation for the specified device aocl_mmd_host_alloc Provide memory that is allocated on the host. Host allocations are accessible by the host and one or more devices aocl_mmd_free Free memory that has been allocated by MMD aocl_mmd_device_alloc Allocate memory that is owned by the device aocl_mmd_shared_alloc Allocate shared memory between the host and the FPGA aocl_mmd_shared_migrate Handle migration of non-concurrent shared allocations any time the accessor of the allocation changes Sections below cover more details for each API (expected arguments, return values). Section 5.4 discusses more about the implementation of the MMD layer APIs in oneAPI ASPs for OFS reference platforms.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#411-aocl_mmd_get_offline_info","title":"4.1.1aocl_mmd_get_offline_info","text":"The aocl_mmd_get_offline_info function obtains offline information about the board specified in the requested_info_id argument. This function is offline because it is device-independent and does not require a handle from the aocl_mmd_open() call.
Syntax
\n\n int aocl_mmd_get_offline_info (\n aocl_mmd_offline_info_t requested_info_id,\n size_t param_value_size,\n void* param_value,\n size_t* param_size_ret )\n\n
Function Arguments
requested_info_id: An enum value of type aocl_mmd_offline_info_t that indicates the offline device information returning to the caller.Table 4-2: Possible Enum Values for the requested_info_id Argument
aocl_mmd_yield function while it waits for events to complete.CAUTION: Setting AOCL_MMD_USES_YIELD to 1 might cause high CPU utilization if the aocl_mmd_yield function does not suspend the current thread. int param_value_size: Size of the param_value field in bytes. This size_t value should match the size of the expected return type that the enum definition indicates. For example, if AOCL_MMD_NUM_BOARDS returns a value of type int, set the param_value_size to sizeof (int). You should see the same number of bytes returned in the param_size_ret argument.
param_value: A void* pointer to the variable that receives the returned information.
param_size_ret: A pointer argument of type size_t* that receives the number of bytes of returned data.
Return Value
A negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#412-aocl_mmd_get_info","title":"4.1.2aocl_mmd_get_info","text":"The aocl_mmd_get_info function obtains information about the board specified in the requested_info_id argument.
Syntax
\n\n int aocl_mmd_get_info (\n int handle,\n aocl_mmd_info_t requested_info_id,\n size_t param_value_size,\n void* param_value,\n size_t* param_size_ret )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
requested_info_id: An enum value of type aocl_mmd_info_t that indicates the device information returning to the caller.
Table 4-3: Possible Enum Values for the requested_info_id Argument
aocl_mmd_host_alloc() function unsigned int AOCL_MMD_SHARED_MEM_CAPABILITIES Capabilities of aocl_mmd_shared_alloc() function unsigned int AOCL_MMD_DEVICE_MEM_CAPABILITIES Capabilities of aocl_mmd_device_alloc() function unsigned int AOCL_MMD_HOST_MEM_CONCURRENT_GRANULARITY Granularity of concurrent host accesses size_t AOCL_MMD_SHARED_MEM_CONCURRENT_GRANULARITY Granularity of concurrent shared accesses size_t AOCL_MMD_DEVICE_MEM_CONCURRENT_GRANULARITY Granularity of concurrent device accesses size_t param_value_size: Size of the param_value field in bytes. This size_t value should match the size of the expected return type that the enum definition indicates. For example, if AOCL_MMD_TEMPERATURE returns a value of type float, set the param_value_size to sizeof (float). You should see the same number of bytes returned in the param_size_ret argument.
param_value: A void* pointer to the variable that receives the returned information.
param_size_ret: A pointer argument of type size_t* that receives the number of bytes of returned data.
Capability Values
Table 4-4: Capability Values for aocl_mmd_get_info Function
Return Value
A negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#413-aocl_mmd_open","title":"4.1.3aocl_mmd_open","text":"The aocl_mmd_open function opens and initializes the specified device.
Syntax
\n\n int aocl_mmd_open (const char *name)\n\n
Function Arguments
name: The function opens the board with a name that matches this const char* string. The name typically matches the one specified by the AOCL_MMD_BOARD_NAMES offline information.The runtime first queries the AOCL_MMD_BOARD_NAMES offline information to identify the boards that it might be able to open. Then it attempts to open all possible devices by calling aocl_mmd_open and using each of the board names as argument.
Note: The name must be a C-style NULL-terminated ASCII string.
Return Value
If aocl_mmd_open() executes successfully, the return value is a positive integer that acts as a handle to the board.
If aocl_mmd_open() fails to execute, a negative return value indicates an error. In the event of an error, the runtime proceeds to open other known devices. Therefore, it is imperative that the MMD layer does not exit the application if an open call fails.
aocl_mmd_close","text":"The aocl_mmd_close function closes an opened device via its handle.
Syntax
\n\n int aocl_mmd_close (int handle)\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.Return Value
If the aocl_mmd_close() executes successfully, the return value is 0.
If aocl_mmd_close() fails to execute, a negative return value indicates an error.
aocl_mmd_set_interrupt_handler","text":"The aocl_mmd_set_interrupt_handler function sets the interrupt handler for the opened device. When the device internals identify an asynchronous kernel event (for example, a kernel completion), the interrupt handler is called to notify the runtime of the event.
Note: Ignore the interrupts from the kernel until this handler is set.
Syntax
\n\n int aocl_mmd_set_interrupt_handler (\n int handle, \n aocl_mmd_interrupt_handler_fn fn,\n void* user_data )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
fn: The callback function to invoke when a kernel interrupt occurs. The fn argument is of type aocl_mmd_interrupt_handler_fn, which is defined as follows:
\n typedef void (*aocl_mmd_interrupt_handler_fn)( int handle, void* user_data );\n
user_data: The void* type user-provided data that passes to fn when it is called.Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#416-aocl_mmd_set_device_interrupt_handler","title":"4.1.6aocl_mmd_set_device_interrupt_handler","text":"The aocl_mmd_set_device_interrupt_handler function sets the device interrupt handler for the opened device. When the device internals identify an asynchronous exception event (for example, a bit correction event), the device interrupt handler is called to notify the runtime of the event.
Note: Ignore the interrupts from the device until this handler is set.
Syntax
\n\n int aocl_mmd_set_device_interrupt_handler (\n int handle, \n aocl_mmd_device_interrupt_handler_fn fn,\n void* user_data )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
fn: The callback function to invoke when a kernel interrupt occurs. The fn argument is of type aocl_mmd_device_interrupt_handler_fn, which is defined as follows:
\n typedef void (*aocl_mmd_device_interrupt_handler_fn)( int handle, aocl_mmd_interrupt_info* data_in, void* user_data );\n
aocl_mmd_interrupt_info is defined as:
\n\n typedef struct {\n unsigned long long int exception_type;\n void *user_private_info;\n size_t user_cb;\n } aocl_mmd_interrupt_info;\n\n Where:
exception_type acts as a bitfield that contains exactly one bit, corresponding to an exception number.user_private_info and user_cb represent pointers to binary data that the OpenCL implementation return. These pointers log additional information that is helpful for debugging the error.user_data: The void* type user-provided data that passes to fn when it is called.Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#417-aocl_mmd_set_status_handler","title":"4.1.7aocl_mmd_set_status_handler","text":"The aocl_mmd_set_status_handler function sets the operation status handler for the opened device. The operation status handler is called under the following circumstances:
Syntax
\n\n int aocl_mmd_set_status_handler (\n int handle,\n aocl_mmd_status_handler_fn fn,\n void* user_data )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
fn: The callback function to invoke when a status update occurs. The fn argument is of type aocl_mmd_status_handler_fn, which is defined as follows:
\n typedef void (*aocl_mmd_status_handler_fn)( int handle, void* user_data, aocl_mmd_op_t op, int status );\n
user_data: The void* type user-provided data that passes to fn when it is called.Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#418-aocl_mmd_yield","title":"4.1.8aocl_mmd_yield","text":"The aocl_mmd_yield function is called when the host interface is idle. The host interface might be idle because it is waiting for the device to process certain events.
Syntax
\n\n int aocl_mmd_yield (int handle)\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.Return Value
A nonzero return value indicates that the yield function performed work necessary for proper device functioning such as processing direct memory access (DMA) transactions.
A return value of 0 indicates that the yield function did not perform work necessary for proper device functioning.
Note: The yield function might be called continuously if it reports that it has necessary work to perform.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#419-aocl_mmd_read","title":"4.1.9aocl_mmd_read","text":"The aocl_mmd_read function is the read operation on a single interface.
Syntax
\n\n int aocl_mmd_read (\n int handle,\n aocl_mmd_op_t op,\n size_t len,\n void* dst,\n int mmd_interface, size_t offset )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.
Note: aocl_mmd_op_t is defined as follows:
\n typedef void* aocl_mmd_op_t;\n
len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
dst: The host buffer, of type void*, to which data is written.
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
offset: The size_t byte offset within the interface at which the data transfer begins.
Return Value
If the read operation is successful, the return value is 0.
If the read operation fails, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4110-aocl_mmd_write","title":"4.1.10aocl_mmd_write","text":"The aocl_mmd_write function is the write operation on a single interface.
Syntax
\n\n int aocl_mmd_write (\n int handle,\n aocl_mmd_op_t op,\n size_t len,\n const void* src,\n int mmd_interface, size_t offset )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.
Note: aocl_mmd_op_t is defined as follows:
\n typedef void* aocl_mmd_op_t;\n
len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
src: The host buffer, of type const void*, from which data is read.
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
offset: The size_t byte offset within the interface at which the data transfer begins.
Return Value
If the write operation is successful, the return value is 0.
If the write operation fails, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4111-aocl_mmd_copy","title":"4.1.11aocl_mmd_copy","text":"The aocl_mmd_copy function is the copy operation on a single interface.
Syntax
\n\n int aocl_mmd_copy (\n int handle,\n aocl_mmd_op_t op,\n size_t len,\n int mmd_interface, size_t src_offset, size_t dst_offset )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.
Note: aocl_mmd_op_t is defined as follows:
\n typedef void* aocl_mmd_op_t;\n
len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
src_offset: The size_t byte offset within the source interface at which the data transfer begins.
dst_offset: The size_t byte offset within the destination interface at which the data transfer begins
Return Value
If the copy operation is successful, the return value is 0.
If the copy operation fails, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4112-aocl_mmd_hostchannel_create","title":"4.1.12aocl_mmd_hostchannel_create","text":"The aocl_mmd_hostchannel_create function creates a channel interface.
Syntax
\n\n int aocl_mmd_hostchannel_create (\n int handle,\n char *channel_name,\n size_t queue_depth,\n int direction )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel_name: Name of the channel to be initialized. The channel name is same as that used in the board_spec.xml file.
queue_depth: The size of pinned internal buffer in bytes. Pointer to the internal buffer is provided when the user calls the aocl_mmd_hostchannel_get_buffer() function.
direction: The direction of the channel.
Return Value
If the function executes successfully, the return value is positive and is handle to the channel.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4113-aocl_mmd_hostchannel_destroy","title":"4.1.13aocl_mmd_hostchannel_destroy","text":"The aocl_mmd_hostchannel_destroy function destroys the channel interface.
Syntax
\n\n int aocl_mmd_hostchannel_destroy (\n int handle,\n int channel )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4114-aocl_mmd_hostchannel_get_buffer","title":"4.1.14aocl_mmd_hostchannel_get_buffer","text":"The aocl_mmd_hostchannel_get_buffer function provides a host with a pointer to the buffer they can access to write or read from the channel interface, along with the space or data available in the buffer, in bytes.
Syntax
\n\n void *aocl_mmd_hostchannel_get_buffer (\n int handle,\n int channel,\n size_t *buffer_size,\n int *status )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
buffer_size: A pointer to size_t that the function writes available buffer space or size to.
status: A pointer to int that the function writes result of the call to.
Return Value
If the function executes successfully, int pointed to by the status pointer is 0. Returned void* may still be NULL, in which case size_t pointed by the buffer_size is 0.
If the function fails to execute, int pointed by the status pointer is a negative value.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4115-aocl_mmd_hostchannel_ack_buffer","title":"4.1.15aocl_mmd_hostchannel_ack_buffer","text":"You can acknowledge write or read from the channel by calling aocl_mmd_hostchannel_ack_buffer.
Syntax
\n\n size_t aocl_mmd_hostchannel_ack_buffer (\n int handle,\n int channel,\n size_t send_size,\n int *status )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
send_size: The size in bytes that the user is acknowledging.
status: A pointer to int that the function writes result of the call to.
Return Value
If the function executes successfully, int pointed to by status pointer is 0. Also, there is no guarantee that the user's send_size is the actual size that gets acknowledged. The returned size_t is the amount of bytes that was actually acknowledged.
If the function fails to execute, int pointed by status pointer is a negative value.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4116-aocl_mmd_program","title":"4.1.16aocl_mmd_program","text":"The aocl_mmd_program function is the program operation for the specified device. The host must guarantee that no other operations are executing on the device during the program operation.
During aocl_mmd_program execution, the kernels are idle and no read, write, or copy operation can occur.
Disable interrupts and program the FPGA with the data from user_data, which has a size specified by the size argument. The host then calls aocl_mmd_set_status_handler and aocl_mmd_set_interrupt_handler again, which enable the interrupts. If events such as interrupts occur during aocl_mmd_program execution, race conditions or data corruption might occur.
Syntax
\n\n int aocl_mmd_program (\n int handle,\n void * user_data,\n size_t size,\n aocl_mmd_program_mode_t program_mode )\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
user_data: The void* type binary contents of the fpga.bin file that is created during kernel compilation.
size: The size of user_data in bytes. The size argument is of size_t.
program_mode: The bit-field that specifies the mode of device programming.
Table 4-5: Possible Values for the program_mode Argument
program_mode Argument Value Description AOCL_MMD_PROGRAM_PRESERVE_GLOBAL_MEMORY This flag specifies that during programming the global memory on the devices are preserved. Return Value
If aocl_mmd_program executes successfully, the return value is the pointer value that the host uses to access shared memory.
aocl_mmd_host_alloc","text":"Host allocations provide memory that is allocated on the host. This memory must be deallocated with the aocl_mmd_free function. Host allocations are accessible by the host and one or more devices. The same pointer to a host allocation may be used on the host and all supported devices. They have address equivalence.
Syntax
Once the device has signaled completion through the aocl_mmd_interrupt_handler_fn function, the host can assume it has access to the latest contents of the memory, allocated by the aocl_mmd_host_alloc function call.
\n\n void* aocl_mmd_host_alloc (\n int* handles,\n size_t num_devices,\n size_t size,\n size_t alignment,\n aocl_mmd_mem_properties_t *properties,\n int* error )\n\n
Function Arguments
handles: Handles for devices that needs access to this memory.
num_devices: Number of devices in the handles.
size: The size of the memory region.
alignment: The alignment (in bytes) of the allocation.
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
error: The error code defined by AOCL_MMD_ERROR*:
Return Value
If the aocl_mmd_host_alloc function executes successfully, the return value is a valid pointer value. Otherwise, the return value is NULL.
aocl_mmd_free","text":"Releases memory that was allocated by MMD.
Syntax
\n\n int aocl_mmd_free (void* mem)\n\n
Function Arguments
mem: The pointer to the memory region. Must be a pointer that is allocated by the MMD.Return Value
Returns one of the following error code:
aocl_mmd_device_alloc","text":"Allocate memory that is owned by the device. This pointer can only be accessed by the kernel. It cannot be accessed by the host. The host is able to manipulate the pointer (for example, increment it) and not just access the underlying data. This memory must be deallocated by the aocl_mmd_free() function.
Syntax
\n\n void * aocl_mmd_device_alloc (\n int handle, \n size_t size, \n size_t alignment, \n aocl_mmd_mem_properties_t *properties, \n int* error )\n\n
Function Arguments
handle: Device that has access to this memory.
size: The size of the memory region.
alignment: The alignment (in bytes) of the memory region.
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
Return Value
Returns one of the following error code:
aocl_mmd_shared_alloc","text":"Shared allocations can migrate between the host and one or more associated device. The same pointer to a shared allocation can be used on the host and the supported device. They have address equivalence.
Syntax
\n\n void * aocl_mmd_shared_alloc (\n int handle, \n size_t size, \n size_t alignment, \n aocl_mmd_mem_properties_t* properties, \n int* error )\n\n
Function Arguments
handle: Device that needs access to this memory.
size: The size of the memory region.
alignment: The alignment (in bytes) of the allocation.
properties: Specifies additional information about the allocated memory described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
error: The error code defined by AOCL_MMD_ERROR*.
Return Value
If the aocl_mmd_shared_alloc function executes successfully, the return value is a valid pointer value. Otherwise, the return value is NULL.
aocl_mmd_shared_migrate","text":"A call to the aocl_mmd_shared_migrate() function must be made for non-concurrent shared allocations any time the accessor of the allocation changes. For example, the aocl_mmd_shared_migrate() function should be called indicating that the allocation should be migrated to the device before a kernel accessing the allocation is launched on the device. Similarly, the aocl_mmd_shared_migrate() function should be called indicating that the allocation is migrated to the host before the host accesses the memory after kernel completion. For concurrent allocations, this call may be used as a performance hint, but it is not strictly required for functionality.
Syntax
\n\n int aocl_mmd_shared_migrate (\n int handle, \n void* shared_ptr, \n size_t size, \n aocl_mmd_migrate_t destination )\n\n
Function Arguments
handle: Device that has access to this memory.
shared_ptr: Pointer allocated by the aocl_mmd_shared_alloc() function.
size: Size (in bytes) of the migration. Must be a multiple of a page boundary that the oneAPI ASP supports.
destination: The destination of the migration.
Return Value
Returns one of the following error code:
oneAPI runtime provides a set of options for the aocl utility.
Note: aocl is an utility available in the oneAPI runtime environment, please use aocl help command for more information on this.
Table 4-6 shows the subcommands that aocl utility provides for FPGA platforms.
Table 4-6: aocl Board Utilities
path-to-FPGA-platform-oneapi-asp uninstall Uninstall board from the host system. Removes FCD. aocl uninstall path-to-FPGA-platform-oneapi-asp initialize Configure a default FPGA image onto the board. For more information about initialization refer to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs Two methods are available to initalize the board: 1. aocl initialize device-name board-variant 2. aocl initialize device-name oneAPI fat binary > Note: The second option requires oneAPI Base Toolkit (Base Kit) version 2024.0 & above as well as 2023.3 OFS Release for oneAPI Accelerator Support Package and above program Configure a new FPGA image onto the board aocl program device-name aocx file diagnose Runs ICD and FCD diagnostics followed by querying devices in installed platforms. If a device-name is specified in the call, it run board vendor's test program for the FPGA platform * aocl diagnose : This queries the devices in FPGA platform and supplies a list of valid strings assigned to the list of devices * aocl diagnose device-name : This runs full diagnostic test on the FPGA platform The runtime expects the routine for each of this utilities to be defined in the oneAPI ASP. It looks for the routine executables in the location specified by the utilbinder element in the board_env.xml file.
install
oneAPI runtime uses the information in board_env.xml file to create a FCD file. The FCD file name matches name attribute of board_env element and the FCD contents are the platform libraries specified in mmdlib element. Refer to section 2.2 for more information about board_env.xml file. The runtime adds the installed platform to the list of installed packages(file used by runtime to track installed platforms) and then invokes the install routine from oneAPI ASP.
uninstall
oneAPI runtime removes the FCD file and removes the platform from list of installed packages. It then invokes the uninstall routine for oneAPI ASP.
initialize
oneAPI runtime invokes initialize routine provided by oneAPI ASPs for installed platforms.
program
oneAPI runtime loads the programming file on the FPGA by invoking program routine provided by the oneAPI ASPs for the installed platform.
diagnose
oneAPI runtime runs ICD and FCD diagnostics to check the ICD and FCD files installed on the system. It then queries for available boards in the installed platform and lists boards matching every installed platform. If a device-name is specified in the call, runtime invokes the diagnostic routine provided in oneAPI ASP.
For more information about the implementation of these routines in oneAPI ASPs for OFS reference platforms, please refer to section 5.5.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#50-oneapi-asp-implementation-details","title":"5.0oneapi-asp Implementation Details","text":"oneapi-asp in the OFS has four reference platform releases, one is based on Stratix\u00ae 10 FPGA and the other three are based on Agilex\u00ae 7 FPGA. This chapter aims to explain the architecture and current implementation details of oneapi-asp for these platforms. The oneapi-asp repository is located here.
The next section explains the oneapi-asp directory structure, followed by sections on hardware and MMD layers.
In addition to the information covered in the sections 5.1 to 5.5, there is a new functionality added to the oneapi-asp repository called the oneAPI ASP Editor. The editor enables easy parameterization and generation of the oneAPI ASP using IP Parameter Editor GUI in Quartus Prime.
If you want to customize oneAPI ASP implementation, you can use one of the two approaches:
Follow the design implementation in section 5.1 to 5.5 as a reference to manually make updates to your oneAPI ASP design or
Use the oneAPI ASP Editor to parameterize your ASP design and let the editor handle ASP directory & design files creation, information about the ASP editor is covered in Appendix B.
Using approach (1) allows fine grained control over design, allowing you to tweak all parameters, interfaces, code blocks in the ASP. The ASP editor (approach 2) abrstracts away some of this details, allowing easy parameter and interface settings, hence reducing design time. There are also limits on some parameters in the editor. Please refer to Appendix B for more information about the parameters in the editor.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#51-oneapi-asp-directory-structure","title":"5.1oneapi-asp Directory Structure","text":"As described in section 2.0, oneAPI compiler & runtime use the board_env.xml and board_spec.xml files to get information about the FPGA platform. The compiler expects the board_env.xml file to be at the topmost level in the platform directory. The board_env.xml file describes the location of the hardware files & platform libraries.
\n\n <?xml version=\"1.0\"?>\n <board_env version=\"Compiler-version\" name=\"ofs_Platform-name/FCD-name\">\n <hardware dir=\"hardware\" default=\"Board-Variant-1\"></hardware>\n <platform name=\"linux64\">\n <mmdlib>libopae-c.so,%b/linux64/lib/libMPF.so,%b/linux64/lib/libintel_opae_mmd.so</mmdlib>\n <linkflags>-L%a/linux64/lib -L%b/linux64/lib</linkflags>\n <linklibs>-lintel_opae_mmd -lrt -lMP</linklibs>\n <utilbindir>%b/linux64/libexec</utilbindir>\n </platform>\n\n </board_env>\n\n
Snippet above shows a sample board_env.xml file, the corresponding oneAPI ASP directory structure must match the following format. Table 5-1 provides details on each folder.
In addition to below folders, oneAPI ASP for OFS reference platforms have another folder called commmon (oneapi-asp/common) containing the hardware files common across all reference platforms (oneapi-asp/common/hardware) and source code for the software layer, i.e. MMD & board utilities ( oneapi-asp/common/source). This is because the software layer is compatible with both Stratix\u00ae 10 FPGA and Agilex\u00ae 7 FPGA PCIe attach reference platform ASPs.
\n oneapi-asp/Platform-Name/\n |--hardware/\n |--|--Board-Variant-1/\n |--|--Board-Variant-2/\n |--linux64/\n |--board_env.xml\n
Note: In addition to above folders, oneAPI ASPs for OFS reference platforms have additional directories called scripts (in oneapi-asp/common and oneapi-asp/Platform-Name) which contains helper scripts for platform generation and oneapi-asp/common/bringup folder which contains a sample for board bring up. Please refer to the README for each reference platform in the oneASP-asp repository for more information on these additional folders. * README for oneapi-asp targeting Intel\u00ae FPGA PAC D5005 reference platform: README * README for oneapi-asp targeting Intel\u00ae FPGA SmartNIC N6001-PL reference platform: README * README for oneapi-asp targeting Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):README * README for oneapi-asp targeting Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile): README
The Platform-Name is used for identifying the platform and can be alphanumeric value decided by the platform developer. For example, Platform-Name d5005 is used for oneapi-asp for Stratix\u00ae 10 FPGA as the reference platform is Intel\u00ae FPGA PAC D5005.
Table 5-1: Directory Contents
Files/Folder (path relative tooneapi-asp/Platform-Name/) Description hardware Contains hardware files (RTL, platform designer files, SDCs, compilation scripts, floorplan settings) and the board_spec.xml files for each board variants. See table 5-2 for more details ../common/source Source code for MMD layer as well as oneapi-asp board utilities linux64 Location for FPGA platform libraries and executables for oneapi-asp board utilities > Note: The linux64 folder does not exist in the Platform-Name directory, it is added after the oneapi-asp build flow is complete board_env.xml Contains platform installation information. Please refer to section 2.2 for more details on board_env.xml elements Tables 5-2 to 5-4 give more details on each of these folders for oneAPI ASPs for OFS reference platforms (before oneapi-asp build flow unless noted otherwise, for more information about oneapi-asp build flow refer to section 5.2).
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#511-hardware-folder","title":"5.1.1 hardware Folder","text":"\noneapi-asp/\n|--common\n|--|--hardware/\n|--Platform-Name\n|--|--hardware/\n|--|--|--common/\n|--|--|--|--build\n|--|--|--board-variant-1/\n|--|--|--|--build\n|--|--|--|--board_spec.xml\n|--|--|--|--part-number_dm.xml (Please see note for this file in table 5-2)\n
Table 5-2: hardware Folder Contents
oneapi-asp/) Description common/hardware Contains hardware design files common to all reference platforms including Quartus\u00ae software Settings Files (*.qsf), IP files (*.v, *_hw.tcl), common RTL code (*.v, *.sv) as well as scripts to control compile flow Platform-Name/hardware/common/build Contains platform specific hardware design files (common to each board variant for a platform) including *.sv, *.sdc, Quartus\u00ae software ini settings file as well as scripts to control compile flow Platform-Name/hardware/board-variant-1/build Contains board variant specific hardware design files like variant specific Verilog header file (*.vh), oneapi_afu.json, *.tcl Platform-Name/hardware/board-variant-1/board_spec.xml Defines compile flow, global memory, kernel interfaces. Please see section 2.1 for more information about board_spec.xml file Platform-Name/hardware/board-variant-1/part-number_dm.xml Device Model file for Altera\u00ae FPGA part on the target platform. The name must be of the format part-number_dm.xml. This file has the total resources available on the device. > Note: The device model files provided as part of the oneAPI Base Toolkit (Base Kit) installation are located in $INTELFPGAOCLSDKROOT/share/models/dm, where INTELFPGAOCLSDKROOT is set by the setvars.sh environment setup script provided by oneAPI toolkit. If the device model file for your part number is not included in $INTELFPGAOCLSDKROOT/share/models/dm, it must be created and placed in the same folder as board_spec.xml. In oneapi-asp for reference platforms, this file only exists for platforms whose target device model files are not provided by oneAPI Base Toolkit (Base Kit) installation."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#512-commonsource-folder","title":"5.1.2 common/source Folder","text":"\noneapi-asp/common/source/\n|--cmake/\n|--|--modules\n|--extra/\n|--|--intel-fpga-bbb\n|--host\n|--include\n|--util\n|--CMakeLists.txt\n
Table 5-3: oneapi-asp/common/source Folder Contents
cmake/modules Contains Find***.cmake files to find packages required for building MMD library extra/intel-fpga-bbb oneAPI ASP for OFS reference platforms uses a library provided as part of Intel\u00ae FPGA Basic Building Blocks (BBB) repository. The oneapi-asp build scripts clone this repository in extra directory by default host Source code for MMD API include Contains MMD header files util Contains source code for oneapi-asp reference platform routines for diagnose and program utilities CMakeLists.txt Top-level CMakeLists.txt file for building MMD and other libraries required by oneapi-asp as well as the executables for diagnose and program utilities"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#513-linux64-folder","title":"5.1.3 linux64 Folder","text":"\noneapi-asp/Platform-Name/linux64/\n|--include (see note below)\n|--lib\n|--libexec\n
Note: The linux64 folder does not exist in the Platform-Name directory, it is present in the oneapi-asp/common folder as the utilities are common for all reference platforms. The include and lib folders also do not exist in the oneapi-asp repository. These along with the utilities are added to the oneapi-asp/Platform-Name folder after the oneapi-asp build flow is complete.
Table 5-4: linux64 Folder Contents
include Contains header files from Intel\u00ae FPGA BBB required by MMD (see section 5.4 for more information on use of MPF from Intel\u00ae FPGA BBB in MMD) lib Contains MMD and Intel\u00ae FPGA BBB libraries libexec Contains executables/scripts for oneapi-asp board utilities"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#52-oneapi-asp-build-flow","title":"5.2 oneapi-asp Build Flow","text":"Figure 1-3 shows a high level overview of the hardware design and Figure 1-4 shows oneAPI ASP as part of the Open FPGA Stack. As shown in these images, all oneapi-asp hardware components reside in the AFU region in the PR slot.
Note: The architecture of the FIM with PR slot is explained in the FIM technical reference manuals, please refer to section 1.3 for links to the manuals.
The oneapi-asp repository contains source files for components that reside in the AFU region for each reference platform. oneapi-asp expects a compiled FIM netlist and a corresponding PR tree. The FIM database is copied to the oneapi-asp during ASP build flow (oneapi-asp/Platform-Name/scripts/build-asp.sh) and ASP compile scripts import the FIM database during oneAPI kernel compilation.
Notes: 1. FIM developer guide outlines steps to compile a FIM and generate PR tree, please refer to section 1.3 for links to FIM developer guides 2. The steps to build oneapi-asp using PR tree and build-asp.sh script are covered in the oneAPI Accelerator Support Package (ASP): Getting Started User Guide
The following figure shows the oneapi-asp build process.
Figure 5-1: oneapi-asp Build Flow
Table 5-5 Environment Variables used in Build Flow
Variable Number Environment Variable Description 1 OFS_PLATFORM_AFU_BBB Should point to location where ofs-platform-afu-bbb repository is cloned, if this variable is not set,build-asp.sh script clones the repository 2 OPAE_PLATFORM_ROOT Must point to the PR tree generated during FIM build, this is a required variable and build flow fails without this 3 LIBOPAE_C_ROOT Should point to the installation location for OPAE libraries (please see oneAPI Accelerator Support Package (ASP): Getting Started User Guide for more information on this variable setting), build-opae.sh script is used to clone & build OPAE library if this variable is not set. 4 OPAE_SDK_REPO_BRANCH If LIBOPAE_C_ROOT is not set, it is recommended to set this variable to indicate the OPAE SDK branch to be used for building OPAE libraries. > Note: build-opae.sh always clones the master branch if OPAE_SDK_REPO_BRANCH environment variable is not set. If you do not have the OPAE SDK installed, please ensure the OPAE_SDK_REPO_BRANCH is set to point to the OPAE release tag matching the release notes for the OFS release you are using. Please refer to oneAPI Accelerator Support Package (ASP): Getting Started User Guide for more information on environment variable and build steps The scripts required for oneapi-asp build that are common to all OFS reference platforms are located in oneapi-asp/common/scripts and build scripts specific to individual platforms are located in oneapi-asp/Platform-Name/scripts folders in the oneapi-asp repository, where Platform-Name is:
n6001 for Intel\u00ae FPGA SmartNIC N6001-PL reference platformd5005 for Intel\u00ae FPGA PAC D5005 reference platformfseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)iseries-dkfor Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)The build flow copies the scripts from oneapi-asp/common/scripts into the oneapi-asp/Platform-Name/scripts directory and generates the complete hardware directories for all board variants in oneapi-asp.
Note: For more information about the build scripts, please refer to README in oneapi-asp/common/scripts directory.
oneapi-asp Hardware Implementation","text":"This section goes deeper into the current hardware architecture of the oneapi-asp.
Figure 1-3 shows a high level overview of the hardware design. OFS reference platforms have different board variants enabling the different paths shown in Figure 1-3. Table below summarizes the board variants and paths enabled in the oneAPI ASP reference design for each. The Path numbers in the table match the ones in Figure 1-3. Figure 5-2 to 5-5 show the detailed diagram for oneapi-asp components in each of these board variants.
Note: The design files have the option to enable multiple global memories (oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems), however, for the purpose of readability, the figures 5-2 to 5-5 show the design with one global memory. For more information about multiple global memories, refer to section 5.3.3.
Table 5-6: OFS Reference Platform Board Variants
# Device Board Variant Host to EMIF with DMA Engine (Path 1) Host to Kernel Interface (Path 2) Kernel to EMIF (Path 3) Kernel to Unified Shared Memory (Path 4) Kernel to HSSI (Path 5) Figure # 1 Agilex\u00ae 7 FPGA Stratix\u00ae 10 FPGAoneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001 * ofs_fseries-dk * ofs_iseries-dk oneapi-asp for Stratix\u00ae 10 FPGA : * ofs_d5005 Yes Yes Yes No No 5-2 2 Agilex\u00ae 7 FPGA Stratix\u00ae 10 FPGA oneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001_usm * ofs_fseries-dk_usm * ofs_iseries-dk_usm oneapi-asp for Stratix\u00ae 10 FPGA : * ofs_d5005_usm Yes Yes Yes Yes No 5-3 3 Agilex\u00ae 7 FPGA oneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001_iopies Yes Yes Yes No Yes 5-4 4 Agilex\u00ae 7 FPGA oneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001_usm_iopipes Yes Yes Yes Yes Yes 5-5 Note: Please see oneapi-asp/Platform-Name/hardware folder for the board variants for each OFS reference platform.
Figure 5-2: oneapi-asp Reference Platform Hardware Design - Board Variant #1
Figure 5-3: oneapi-asp Reference Platform with USM Hardware Design - Board Variant #2
Figure 5-4: oneapi-asp Reference Platform with IO Pipes Hardware Design - Board Variant #3
Figure 5-5: oneapi-asp Reference Platform with IO Pipes and USM Hardware Design - Board Variant #4
The ofs_plat_afu.sv is the top level entity for the oneapi-asp.
Hardware design files that are common for all OFS reference platforms (for the components inside ofs_plat_afu module) are located in oneapi-asp/common/hardware/common/build and hardware design file specific to the individual platforms are located in the oneapi-asp/Platform-Name/hardware/Board-Variant/build directory. The common files are copied to the oneapi-asp/Platform-Name/hardware/Board-Variant folder during oneapi-asp build flow. The FIM database files imported during oneapi-asp build (section 5.2) are located in oneapi-asp/Platform-Name/hardware/Board-Variant/fim_platform directory.
Tables 5-7 give a brief description of the important design files in all board variants, the location for these files shown in table 5-7 is post oneapi-asp build flow.
Table 5-7: Hardware Design Files in oneapi-asp/Platform-Name/hardware/Board-Variant/build
afu_flat.qsf) mpf_vtp.qsf Adds IP components from Intel\u00ae FPGA BBB (see section 5.3.2 below for information about use of MPF blocks from Intel\u00ae FPGA BBB) repository used in the design asp_design_files.tcl Adds design files to the project, this file is source in afu_ip.qsf oneapi_afu.json Accelerator Description File that describes the metadata associated with a oneAPI ASP. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration > Note: For more information about use of a JSON file for storing meadata for an AFU, refer to Accelerator Description File section in PIM Based AFU Developer Guide ip Contains the HW TCL and Verilog source code files for IP components used in the hardware design scripts Contains scripts to control oneAPI kernel and asp Quartus\u00ae software compile flow (see section 5.3.5 for more information on compile flow) rtl Contains the SystemVerilog and Verilog design files shown in figures 5-2 to 5-5 above rtl/ofs_plat_afu.sv Top level wrapper file for oneapi-asp. Contains the Platform Interface Manager (PIM) blocks for protocol translation & connecting FIM ports to oneapi-asp ports. For more information about PIM, see the PIM subtopic below. rtl/afu.sv Contains the PIM blocks for VTP address translation and instantiates the kernel_wrapper entity. For more information about PIM, see the PIM subtopic below. rtl/kernel_wrapper.v Contains the kernel_system instantiation rtl/asp_logic.sv Contains the DMA engine, board system instantiation, please see information about board_hw.tcl, ddr_board_hw.tcl and ddr_channel_hw.tcl below rtl/ofs_asp.vh Contains the macros to control instantiation of logic on host to memory path, logic required to enable Unified Shared Memory(USM) support and IO pipes support rtl/ofs_asp_pkg.sv Contains Verilog parameters used in controlling values of number of channels (memory/IO pipes), address widths, pipeline stages etc. for logic in the oneapi-asp board_hw.tcl Contains the Kernel Interface IP instantiation, register to store board AFU ID and ddr_board system instantiation. Please refer to ddr_board_hw.tcl and ddr_channel_hw.tcl below. > Note: The board AFU ID is used by the MMD layer during board discovery ddr_board_hw.tcl Instantiated in the board_hw.tcl, contains IP components in the host to EMIF and kernel to EMIF datapath. Memory Bank Divider is instantiated in this file. Please refer to section 3.1.1 for more details on Memory Bank Divider ddr_channel_hw.tcl Instantiated in ddr_board_hw.tcl, contains bridges required for clock domain crossings between PCIe and EMIF clocks as well as kernel and EMIF clock ofs_asp.sdc Contains clock group constraints for all clocks in oneAPI ASP ../fim_platform Contains Platform Interface Manager(PIM) and FIM database used in the design. See PIM subtopic below The hardware implementation diagrams show a PF/VF Mux/De-mux module in the AFU region. The PF/VF mux routes packets to AFU component based on pf_num and vf_num information in PCIe TLP header. For more information about the PF/VF mux, please refer to the PF/VF Mapping details in FPGA Interface Manager Technical Reference Manual for your target device (links are available in section 1.3).
The oneapi-asp resides inside the AFU region and, depending on the FIM configuration, connects to the lowest PF or lowest VF number that routes into the PR slot. Table below summazires the PF/VF mapping in oneapi-asp for some of the different FIM configurations.
Table 5-8: PF/VF Mapping in oneapi-asp
oneapi-asp Stratix\u00ae 10 FPGA Default PF0-VF1 Agilex\u00ae 7 FPGA Default PF0-VF0 Agilex\u00ae 7 FPGA Compiled using <n6001/iseries-dk>_1pf_1vf.ofss, i.e. PCIe Subsystem configuration is set to PF0 with 1 VF PF0-VF0 Agilex\u00ae 7 FPGA Compiled using n6001_2pf.ofss, i.e. PCIe Subsystem configuration is set to two physical functions PF0 and PF1 PF1 *Note: For more information on different FIM configurations & how to compile these, please refer to FPGA Interface Manager(FIM) Developer Guides for Open FPGA Stack for your target device (links are available in section 1.3).
Sections below provide some more information on some blocks in the hardware design block diagram shown above. Refer to section 3.1 for more information about Memory Bank Divider and to section 3.2 for information about Kernel Interface.
In addition to above files/folders, an important part of the design are the ofs_plat_* modules provided by Platform Interface Manager (PIM). oneAPI kernel system and oneapi-asp have Avalon\u00ae interfaces. FIM components have AXI interfaces. The oneAPI compiler generated kernel_system with Avalon interfaces. The PIM modules are used for protocol translation.
In addition to this, PIM blocks are also used for Virtual to Physical (VTP) address translation. The Memory Mapped Device(MMD) layer provides virtual addresses which are converted to physical addresses by the VTP blocks to allow Unified Shared Memory(USM) and DMA transfers to/from host memory.
Note: For more information about PIM, please refer to PIM README here.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#532-direct-memory-accessdma-module","title":"5.3.2 Direct Memory Access(DMA) Module","text":"The Direct Memory Access(DMA) module is located in the host to EMIF datapath in the oneapi-asp and provides a controller to execute transfers from host to DDR on the board and vice versa. The source files for the DMA module used in oneapi-asp for OFS reference platforms are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl/dma directory (location post oneapi-asp build flow). Figure below shows the DMA module interface.
Figure 5-7: DMA Controller Block Diagram
Figure 5-2 shows the instantiation of this dma_top module as dma_controller_inst. Table 5-9 shows the signal names and descriptions. Section 5-5 covers the complete compilation flow.
Table 5-9: DMA Module Signal Descriptions
Signal or Port Description mmio64_if Control signal from host, connects to DMA Control Status Registers(CSR) host_mem_rd_avmm_if Avalon\u00ae memory-mapped interface to read from host memory host_mem_wr_avmm_if Avalon\u00ae memory-mapped interface to write to host memory local_mem_rd_avmm_if Avalon\u00ae memory-mapped interface to read from on-board DDR4 memory local_mem_wr_avmm_if Avalon\u00ae memory-mapped interface to write to on-board DDR4 memory dma_irq_host2fpga Interrupt to indicate completion of host to FPGA DMA transaction (read from host, write to DDR4) dma_irq_fpga2host Interrupt to indicate completion of FPGA DDR4 to host DMA transaction (read from DDR4, write to host)> Note: Theoneapi-asp for OFS reference platforms uses a host memory write to indicate completion of FPGA to host transaction The data transfer module manages data movement from source to destination addresses.
To start a data transfer, the data transfer module requires following information, this is set by the dma_dispatcher:
oneapi-asp for OFS reference platforms uses virtual addresses in the DMA controller for data transfer. The Memory Properties Factory(MPF) Virtual to Physical(VTP) blocks (i.e.mpf_vtp_* modules) shown in figure 5-2 translate virtual addresses to appropriate host addresses.
Note: Memory Properties Factory(MPF) is a part of Intel\u00ae FPGA Basic Building Blocks. Please refer to Intel\u00ae FPGA BBB repository for more information.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#533-board_hwtcl-file","title":"5.3.3board_hw.tcl File","text":"As described in table 5-7, the board_hw.tcl contains the Kernel Interface IP instantiation, register to store board AFU ID and ddr_board system instantiation. The board_hw.tcl component has a few parameters that can be used to control the internal composition. Snippet below shows the portion of the TCL script adding these parameters. These are described in table below.
board_hw.tcl sources another file called parameters.tcl, this file is located in oneapi-asp/Platform-Name/hardware/Board-Variant/build directory and contains different values for the parameters based on the board variant. All the parameters in the board_hw.tcl derive their value from respective values in parameters.tcl(all variables starting with p_<parameter-name>) for each board variant when the oneapi-asp is built.
\n\n +-----------------------------------\n | parameters\n |\nsource parameters.tcl\n\nadd_parameter AFU_ID_H STD_LOGIC_VECTOR $p_AFU_ID_H\nset_parameter_property AFU_ID_H DEFAULT_VALUE $p_AFU_ID_H\nset_parameter_property AFU_ID_H DISPLAY_NAME \"AFU ID H\"\nset_parameter_property AFU_ID_H AFFECTS_ELABORATION true\n\nadd_parameter AFU_ID_L STD_LOGIC_VECTOR $p_AFU_ID_L\nset_parameter_property AFU_ID_L DEFAULT_VALUE $p_AFU_ID_L\nset_parameter_property AFU_ID_L DISPLAY_NAME \"AFU ID L\"\nset_parameter_property AFU_ID_L AFFECTS_ELABORATION true\n\nadd_parameter IOPIPE_SUPPORT BOOLEAN $p_IOPIPE_SUPPORT\nset_parameter_property IOPIPE_SUPPORT DEFAULT_VALUE $p_IOPIPE_SUPPORT\nset_parameter_property IOPIPE_SUPPORT DISPLAY_NAME \"IO Pipe Support\"\nset_parameter_property IOPIPE_SUPPORT AFFECTS_ELABORATION true\n\nadd_parameter NUMBER_OF_DMA_CHANNELS INTEGER $p_NUMBER_OF_DMA_CHANNELS\nset_parameter_property NUMBER_OF_DMA_CHANNELS DEFAULT_VALUE $p_NUMBER_OF_DMA_CHANNELS\nset_parameter_property NUMBER_OF_DMA_CHANNELS DISPLAY_NAME \"Number of DMA Channels\"\nset_parameter_property NUMBER_OF_DMA_CHANNELS AFFECTS_ELABORATION true\n\nadd_parameter SNOOP_PORT_ENABLE BOOLEAN $p_SNOOP_PORT_ENABLE\nset_parameter_property SNOOP_PORT_ENABLE DEFAULT_VALUE $p_SNOOP_PORT_ENABLE\nset_parameter_property SNOOP_PORT_ENABLE DISPLAY_NAME \"Enable Snoop Port\"\nset_parameter_property SNOOP_PORT_ENABLE AFFECTS_ELABORATION true\n\nadd_parameter NUMBER_OF_GLOBAL_MEMORY_SYSTEMS INTEGER $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS\nset_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS DEFAULT_VALUE $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS\nset_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS DISPLAY_NAME \"Number of Global Memory Systems\"\nset_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS AFFECTS_ELABORATION true\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 0 } {\n add_parameter MEM_0_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_0_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_0_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 0 Banks\"\n set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_0_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_0_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_0_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 0 Memory Bank Address Width\"\n set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_0_DATA_WIDTH INTEGER $p_MEM_0_DATA_WIDTH\n set_parameter_property MEM_0_DATA_WIDTH DEFAULT_VALUE $p_MEM_0_DATA_WIDTH\n set_parameter_property MEM_0_DATA_WIDTH DISPLAY_NAME \"Global Memory 0 Data Width\"\n set_parameter_property MEM_0_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_0_MAX_BURST_SIZE INTEGER $p_MEM_0_MAX_BURST_SIZE\n set_parameter_property MEM_0_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_0_MAX_BURST_SIZE\n set_parameter_property MEM_0_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 0 Maximum Burst Size\"\n set_parameter_property MEM_0_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 0 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_0_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_0_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_0_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 0 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 1 } {\n add_parameter MEM_1_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_1_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_1_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 1 Banks\"\n set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_1_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_1_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_1_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 1 Memory Bank Address Width\"\n set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_1_DATA_WIDTH INTEGER $p_MEM_1_DATA_WIDTH\n set_parameter_property MEM_1_DATA_WIDTH DEFAULT_VALUE $p_MEM_1_DATA_WIDTH\n set_parameter_property MEM_1_DATA_WIDTH DISPLAY_NAME \"Global Memory 1 Data Width\"\n set_parameter_property MEM_1_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_1_MAX_BURST_SIZE INTEGER $p_MEM_1_MAX_BURST_SIZE\n set_parameter_property MEM_1_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_1_MAX_BURST_SIZE\n set_parameter_property MEM_1_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 1 Maximum Burst Size\"\n set_parameter_property MEM_1_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 1 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_1_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_1_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_1_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 1 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 2 } {\n add_parameter MEM_2_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_2_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_2_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 2 Banks\"\n set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_2_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_2_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_2_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 2 Memory Bank Address Width\"\n set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_2_DATA_WIDTH INTEGER $p_MEM_2_DATA_WIDTH\n set_parameter_property MEM_2_DATA_WIDTH DEFAULT_VALUE $p_MEM_2_DATA_WIDTH\n set_parameter_property MEM_2_DATA_WIDTH DISPLAY_NAME \"Global Memory 2 Data Width\"\n set_parameter_property MEM_2_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_2_MAX_BURST_SIZE INTEGER $p_MEM_2_MAX_BURST_SIZE\n set_parameter_property MEM_2_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_2_MAX_BURST_SIZE\n set_parameter_property MEM_2_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 2 Maximum Burst Size\"\n set_parameter_property MEM_2_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 2 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_2_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_2_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_2_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 2 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 3 } {\n add_parameter MEM_3_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_3_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_3_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 3 Banks\"\n set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_3_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_3_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_3_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 3 Memory Bank Address Width\"\n set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_3_DATA_WIDTH INTEGER $p_MEM_3_DATA_WIDTH\n set_parameter_property MEM_3_DATA_WIDTH DEFAULT_VALUE $p_MEM_3_DATA_WIDTH\n set_parameter_property MEM_3_DATA_WIDTH DISPLAY_NAME \"Global Memory 3 Data Width\"\n set_parameter_property MEM_3_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_3_MAX_BURST_SIZE INTEGER $p_MEM_3_MAX_BURST_SIZE\n set_parameter_property MEM_3_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_3_MAX_BURST_SIZE\n set_parameter_property MEM_3_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 3 Maximum Burst Size\"\n set_parameter_property MEM_3_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 3 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_3_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_3_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_3_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 3 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\n Table 5-10 board_hw.tcl Parameters
oneapi-asp has IO pipes support. This must be set to true for oneapi-asp board variant that has the channels element defined in the board_spec.xml. Table 5-6 shows the OFS reference platform board variants that have IO pipes enabled. To ensure correct functionality, the HSSI susbsystem in the FIM must support the settings in the oneapi-asp. For more information about channels element refer to section 2.1.8 NUMBER_OF_DMA_CHANNELS Number of DMA Channels Reserved for future use > Note: oneapi-asp tag ofs-2024.2-1 supports only 1 DMA channel SNOOP_PORT_ENABLE Enable Snoop Port Used to enable the acl_asp_snoop port for Memory Bank Divider, for more information about acl_asp_snoop port, refer to Table 3-2 NUMBER_OF_GLOBAL_MEMORY_SYSTEMS* Please see notes below the table for additional design details Number of Global Memory Systems Number of global memories connected to the oneAPI kernel > Note: oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems MEM_<id>_NUMBER_OF_MEMORY_BANKS Number of Global Memory <id> Banks The number of homogenous memory banks in <id>th global memory system, where id can be 0, 1, 2 or 3 (oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems). Used for parameterizing the ddr_board system. The ddr_board system in oneapi-asp for OFS reference platforms has a single Memory Bank Divider instantiated for each global memory. The number of memory banks in a single global memory can be 1, 2, 4 or 8 (Memory Bank Divider IP requirement). To expand number of banks in a global memory beyond this range, additional Memory Bank Didvider IP will need to be instantiated and connected in ddr_board_hw.tcl. For more information about Memory Bank Divider, please refer to section 3.1.1 MEM_<id>_MEMORY_BANK_ADDRESS_WIDTH Global Memory <id> Memory Bank Address Width This is the total addressable width for a single memory bank/interface in <id>th global memory. For example, if global memory 0 (id) consists of 32 GB on-board DDR4 memory with four 8 GB memory banks, then this memory bank address width parameter must be set to 33 (single 8 GB channel is considered) for MEM_0_MEMORY_BANK_ADDRESS_WIDTH. This value is used for parameterizing the ddr_board system. MEM_<id>_DATA_WIDTH Global Memory <id> Data Width This is the width of the data bus for a single single memory bank/interface in the <id>th global memory to the oneAPI kernel MEM_<id>_MAX_BURST_SIZE Global Memory <id> Maximum Burst Size Maximum burst size for the pipeline bridges in the host to memory as well as kernel to memory data path for <id>th global memory MEM_<id>_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE Global Memory <id> Kernel to global memory waitrequest allowance This is the waitrequest_allowance for the clock crossing bridge in the kernel to memory data path for <id>th global memory > Note: For more information about waitrequest_allowance refer to section 2.1.4 on interface attribute MEM_<id>_MBD_TO_MEMORY_PIPE_STAGES Global Memory <id> MBD to Memory Pipeline Stages Value is used to set the number of Avalon-MM pipeline bridges between Memory Bank Divider and the PCIe clock domain to EMIF clock domain clock crossing bridge for <id>th global memory * > Notes (about Number of Global Memory): 1. If the number of global memory systems in more than 1, a pipeline bridge is added in the host to memory data path ensuring the host sees a continuous address space for the total addressable memory. The kernel gets added as a single memory space using pipeline bridge, the kernel connected to each global memory system. 2. If you are using the oneAPI ASP Editor (described in Appendix B), then the RTL design parameterization and board_spec.xml file generaration is handled by the editor. However if you are manually making changes to the ASP implementation, then the ofs_asp_pkg.sv file must be updated to ensure it has all the ASP_GLOBAL_MEM_<id>_* parameters for all global memories in your system. The default file (in oneapi-asp/common/hardware/common/build/rtl if oneapi-asp build flow is not run, else update the file in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl) contains the parameters for Global memory 0 only. The board_spec.xml file must also be edited to add global memory interfaces. For more information about board_spec.xml and different global memory configurations, refer to secton 2.1.5.
The internal component instantiations and connections are done in the composition callback (compose procedure) in board_hw.tcl.
For more information about the different component commands and callback used in board_hw.tcl, ddr_board_hw.tcl and ddr_channel.tcl, please refer to the Quartus\u00ae Prime Pro Edition User Guide: Platform Designer(Component Interface Tcl Reference section)
I/O pipes allow kernel to stream data directly using HSSI. To demonstrate this functionality, reference design in oneapi-asp repository (refer to Figure 5-4 and 5-5) has a UDP protocol engine to allow transmitting UDP/IP packets over HSSI..
Figure below shows a simple block diagram of the UDP engine.
Figure 5-8: UDP Offload Engine
The UDP engine consists of a separate receive (rx) and trasmit (tx) path. The following functionalilty is performed by this reference design engine:
The source files for UDP engine used in oneapi-asp for OFS reference platform are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl/udp_offload_engine directory.
Note: The same engine is used in the board variant with USM shown in Figure 5-5.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#535-hardware-compile-flow","title":"5.3.5 Hardware Compile Flow","text":"Figure below shows the compile flow overview; the oneAPI compiler generated hardware circuit is compiled by Quartus\u00ae software along with design files for oneapi-asp.
Figure 5-9: oneAPI Compile Flow Overview
The oneAPI compiler uses the board_spec.xml to get more information about the oneapi-asp configuration. board_spec.xml file has a compile element to allow control of the Quartus\u00ae software compilation flow. The attributes of this element are discussed in section 2.1.2. The oneapi-asp uses tcl scripts to control the Quartus\u00ae software compilation flow. Figure 5-10 shows the flow and scripts used. All compilation scripts are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/scripts folder (location post oneapi-asp build flow).
Figure 5-10: Compilation Scripts in oneapi-asp
Table 5-11 summarizes notes for reference numbers 1-2 marked in figure above.
Table 5-11: Notes for Reference Numbers in Figure 5-10
Reference Number Note 1 revision_name isafu_flat for oneapi-asps for OFS reference platforms 2 gen-asp-quartus-report.tcl script generates a report (acl_quartus_report.txt) containing resource utilization and kernel clock frequency summary. See Note below about the acl_quartus_report.txt file Note: The acl_quartus_report.txt is required by the oneAPI compiler to generate the Quartus (Static) summary in FPGA optimization reports successfully. The oneAPI compiler expects the format to be as follows (order of metrics can be different, but the names of these FPGA metrics should match the format given here):
\nALUTs: <value>\nRegisters: <value>\nLogic utilization: <value>\nI/O pins: <value>\nDSP blocks: <value>\nMemory bits: <value>\nRAM blocks: <value>\nActual clock freq: <value>\nKernel fmax: <value>\n1x clock fmax: <value>\n2x clock fmax: <value>\nHighest non-global fanout: <value>\n
If acl_quartus_report.txt is missing, the oneAPI compiler will fail to generate the Quartus(Static) summary in FPGA optimization reports successfully. If you are creating a custom platform, please ensure the acl_quartus_report.txt report is generated. The gen-asp-quartus-report.tcl can be used as a reference to implement the generation of this report in your custom platform
oneapi-asp Memory Mapped Device(MMD) Layer Implementation","text":"As discussed in section 4.1, the MMD provides a set of API that allow the runtime to control the device and communicate with it.
The source code for MMD layer is located in oneapi-asp/common/source/host folder. aocl_mmd.h is the header file for the implemented API and is located in oneapi-asp/common/source/include folder. Table below summarizes the APIs that have been implemented in oneapi-asp for OFS reference platforms.
Note: For more details about the API, its arguments and enums please refer to the aocl_mmd.h file and to section 4.1.
Table 5-12: MMD API Implemented in oneapi-asp for OFS Reference Platforms
The implementation of these APIs is in oneapi-asp/common/source/host/mmd.cpp. The functions used in the implementation are distributed across various source files. Table below provides details on source code files.
Table 5-13: MMD Source Code Files
Files/Folder Description mmd.cpp This file has the implementation for all MMD API calls listed in table 5-12 fpgaconf.hfpgaconf.c Contains bitstream reconfiguration function declaration(.h) & definition(.c) kernel_interrupt.h ContainsKernelInterrupt class declaration; the class consists of functions to handle kernel interrupts kernel_interrupt.cpp Contains KernelInterrupt class constructor and function definitions mmd_device.h Contains Device class declaration, which stores device data and has functions to interact with the device mmd_device.cpp Contains Device class constructor and function definitions mmd_dma.hmmd_dma.cpp Contain DMA functions declaration(.h) & definition(.cpp) mmd_iopipes.h Contains the iopipes class declaration(.h) mmd_iopipes.cpp Contains function definitions, these functions include iopipes class constructor as well as fucntions to detect and setup CSR space for IO pipes feature in board variants that support IO pipes zlib_inflate.hzlib_inflate.c Function declaration(.h) and definition(.c) for decompressing bitstream data CMakeLists.txt CMakeLists.txt file for building MMD source code The build flow scripts build the MMD library, i.e. libintel_opae_mmd, and place them in oneapi-asp/Platform-Name/linux64/lib folder. The MMD library is specified as part of mmdlib, linklibs element in board_env.xml and used at runtime (refer to section 5-1 for sample board_env.xml file and section 2.2 for more information about board_env.xml elements).
Use of OPAE library in MMD
The MMD layer uses API from OPAE SDK for various device operations. Hence, the MMD layers requires OPAE library to be loaded to execute successfully. The mmdlib element specifies the libopae-c library to be loaded before the MMD library (demonstrated in sample board_env.xml in section 5-1).
Note: Please refer to Software Reference Manual: Open FPGA Stack for more information about OPAE SDK API. The document also has information about linux-dfl driver.
Use of Memory Properties Factory(MPF) library in MMD
In addition to OPAE, the MMD also uses API from Memory Properties Factory(MPF) software for memory operations. The libMPF library is compiled as part of oneapi-asp build flow and placed in oneapi-asp/Platform-Name/linux64/lib. This library is also loaded before the MMD library by specifying before libintel_opae_mmd in mmdlib element in board_env.xml.
Note: Memory Properties Factory(MPF) is a part of Intel\u00ae FPGA BBB. Please refer to Intel\u00ae FPGA BBB wiki for more information about MPF.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#55-oneapi-asp-utilities-implementation","title":"5.5oneapi-asp Utilities Implementation","text":"This section covers the implementation of board utilities (refer to section 4.2 for more information on board utilities) in oneapi-asp for OFS reference platforms.
Table below shows the source code/script locations for the utilities.
Table 5-14: oneapi-asp Utilities Source Code Locations
oneapi-asp/common/source/util/diagnostic program oneapi-asp/common/source/util/reprogram installuninstallinitialize> Note: These are executable scripts oneapi-asp/common/linux64/libexec/ (copied to oneapi-asp/Platform-Name/linux64/libexec/ during oneapi-asp build flow, for more information about oneapi-asp build flow, refer to section 5.2) diagnose and program are compiled as part of the oneapi-asp build flow and executables are placed in oneapi-asp/Platform-Name/linux64/libexec/. A CMakeLists.txt file is provided for building the utilities, located in oneapi-asp/common/source/util directory.
The path to all of the above utility executables is used in utilbinder element in board_env.xml (demonstrated in sample board_env.xml in section 5-1). The runtime uses this when the corresponding aocl utility is invoked.
Brief descriptions for the source code files are given in table below.
Table 5-15: Brief Descriptions of oneapi-asp Utility Routines for OFS Reference Platforms
install, initialize routines install install routine invokes the setup_permissions.sh script after the FPGA Client Driver (FCD) is setup by the runtime uninstall uninstall routine reverts the port permission, memory locking and hugepage setting changes performed by install routine and is invoked by runtime after the FCD is removed by runtime initialize initialize routine performs the following steps: * looks for the initialization binary for the board variant to be initialized * extracts the FPGA hardware configuration file from the oneAPI fat binary using clang-offload-extract command provided by oneAPI Base Toolkit (Base Kit) version 2024.0 and beyond * invokes the setup_permissions.sh script to set correct device permissions * performs partial reconfiguration of the FPGA device by invoking program routine with the initialization bitstream as an argument >Note: For more information about how initialize utility extracts FPGA hardware configuration file from oneAPI fat binary, refer to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs program program routine allocates memory and loads the supplied initialization bitstream in memory followed by a call to reprogramming function provided by oneapi-asp's MMD library. The MMD library uses fpgaReconfigureSlot API provided by OPAE library to perform device reconfiguration> Note: Please refer to Software Reference Manual: Open FPGA Stack for more information about OPAE SDK API diagnose diagnose routine scans for the available devices for the installed platform and performs DMA transactions between host & device. It also reports the DMA transfer bandwidth. diagnose routine uses functions provided by the MMD library for scanning & opening connection to available devices"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#appendix","title":"Appendix","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#appendix-a-debug-variables-and-commands","title":"Appendix A: Debug Variables and Commands","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#a1-memory-mapped-devicemmd-layer-debug-variables","title":"A.1 Memory Mapped Device(MMD) Layer Debug Variables","text":"The Memory Mapped Device(MMD) layer provides debug capability for custom ASP and oneAPI application developers. Snippet below shows how to set the variable and table below documents the environment variables that can be used to get debug information from the MMD layer at runtime.
\n\nexport MMD_<DMA/ENABLE/PROGRAM>_DEBUG = 1\n.<oneapi-binary> # Execute sample with MMD debug environment variable set \n\n
Table A-1: Environment Variable to Enable Debug Information in MMD
Environment Variable Set to Value Description MMD_DMA_DEBUG 1 Set this environment variable to see debug information for Direct Memory Access (DMA) transactions MMD_PROGRAM_DEBUG 1 Set this environment variable to see debug information when the MMD reprograms the PR region MMD_ENABLE_DEBUG 1 Set this to see debug information for all steps in the MMD layer. Maximum debug information is printed when this environment variable is set<>"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#a2-runtime-debug-variables","title":"A.2 Runtime Debug Variables","text":"In addition to the debug variables provided by the MMD layer, there are debug variables provided by the Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology used in oneAPI base toolkit. For a full list of runtime debug variables, please refer to the Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology README.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#a3-extracting-oneapi-binary-informtion-using-aocl-binedit-command","title":"A.3 Extracting oneAPI Binary Informtion usingaocl binedit Command","text":"The aocl binedit utility allows you to extract the following useful information about the compiled binary:
* oneAPI compiler version
* Quartus\u00ae Prime software version
* Compiler command used
board_spec.xml from the BSP used for compilingYou can use the aocl binedit utility with the following command:
\naocl binedit <oneapi-binary> <option:list/get/print/exists> <section_name> <output_file>\n
Table A-2: Environment Variable to Enable Debug Information in MMD
| aocl binedit Command Option | Description | | list | Lists all available sections in the given binary | | print | Writes contents of the existing named section to the standard output stream for each package file in the binary | | get | Writes contents of the existing named section to the output file | | exists | Verifies if the section exists in the package files in the binary. The non-zero exit code indicates the section does not exist |
Example
You can first use the option list to see all the available sections of the oneAPI binary:
\naocl binedit <oneapi-binary> list\n
Command below shows the sections available of a binary generated targeting a oneapi-asp board variant, in this case, the sample compiled is board_test.
\n\n $ aocl binedit board_test.fpga list\n AOCX File: binedit/aocx.0\n Sections in package file:\n .acl.board, 9 bytes\n .acl.board_package, 38 bytes\n .acl.compilation_env, 2528 bytes\n .acl.rand_hash, 40 bytes\n .acl.quartus_input_hash, 163 bytes\n .acl.compileoptions, 0 bytes\n .acl.version, 49 bytes\n .acl.autodiscovery, 2048 bytes\n .acl.board_spec.xml, 2360 bytes\n .acl.kernel_arg_info.xml, 4681 bytes\n .acl.target, 4 bytes\n .acl.rtl_hash, 566 bytes\n .acl.fpga.bin, 11948560 bytes\n .acl.quartus_report, 329 bytes\n .acl.quartus_json, 347 bytes\n\n
After knowing the sections available in the oneAPI binary, you can explore the value of each section with the printoption or save the information into a file using the get option.
Command below shows an example of the use of aocl binedit to know the board variant from which the sample was created.
\naocl binedit oneapi-binary print .acl.board\n
Sample output for a board_test.fpga binary that was created with an ofs_n6001 board variant.
\n$ aocl binedit board_test.fpga print .acl.board\nAOCX File: binedit/aocx.0\nofs_n6001\n"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#appendix-b-oneapi-accelerator-support-packageasp-editor","title":"Appendix B: oneAPI Accelerator Support Package(ASP) Editor","text":"
Note: This section requires readers to be faimilar with the Prerequisites (section 1.3) and the main chapters in this reference manual.
The oneapi-asp tag ofs-2024.2-1 adds a preliminary version of an editor for enabling easy parametrization of oneAPI Accelerator Support Pacakges in OFS based platforms. The features provided by this editor are same as the oneapi-asp for OFS reference platforms described in the main chapters in this manual including support for on-board global memory, Unified Shared Memory (USM), Direct Memory Access(DMA) engine, VTP (Virtual to Physical) Address Translation, I/O Pipes, User Datagram Protocol(UDP ) offload engine. In additon to this the editor handles generation of oneAPI directory structure for your board variant, generation of XML files (board_spec.xml, board_env.xml) and JSON file(oneapi_afu.json) as well as setting parameters for the oneAPI ASP hardware design. The editor is meant to abstract away some of the design steps and reduce design time, it can be used to parameterize oneAPI ASP using IP Parameter Editor GUI in Quartus Prime.
The following sections explain functionality, parameterization and user flow for the oneAPI ASP Editor. The oneAPI Accelerator Support Package (ASP): Getting Started User Guide covers the steps to demonstrate use of this editor for OFS reference platforms.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#b1-oneapi-asp-editor-overview","title":"B.1 oneAPI ASP Editor Overview","text":"The following diagram shows GUI view and summarizes the functionality provided by the editor.
Figure B-1: oneAPI ASP Editor GUI
Table below explains the functionality marked in the figure.
Table B-1 oneAPI ASP Editor GUI Functionality
# Editor Function Description 1 Parameters Window The oneAPI ASP interface settings can be set using the parameters window. Users can also derive these settings from the FIM interfaces using presets created from the FIM settings (this requiresOPAE_PLATFORM_ROOT environment variable to be set, for more information, refer to the user flow in section B.2). Details of each parameter is provided in section B.3 2 Presets Window Quartus Prime provides an option to create, modify, and save parameter values as a preset file. You can then apply the parameter values in the preset file to the current component that you are parameterizing. The presets window shows the OFS reference platform presets provided in the oneapi-asp. Table B-3 summarizes the presets provided. You can also save your own preset file , e.g. to saving a custom board variant settings. Note: For more information about applying presets, please refer to Quartus\u00ae Prime Pro Edition User Guide: Getting Started 3 Generate HDL The Generate HDL button creates the directory structure for building the oneAPI ASP for selected board variant with the parameters set in the parameter window. It performs the following tasks: * Creates the directory structure and copies required platform files for the selected board variant & device * Generates the Verilog header and package files (ofs_asp.vh and ofs_asp_pkg.sv) for a board variant based on parameters set in the oneAPI ASP Editor * Generates the board_env.xml, board_spec.xml, oneapi_afu.json > Note: For more information about this XML files, please refer to section 2.0. For more information about oneapi_afu.json, refer to table 5-7 in section 5.3 * Generates parameters.tcl file used by the board_hw.tcl file > Note: For more information about the board_hw.tcl file refer to section 5.3.3 The design files for the oneAPI ASP Editor are located in oneapi-asp/oneapi_asp_editor/ directory. The directory structure of the oneapi_asp_editor folder is shown below.
\noneapi-asp/\n|--common/\n|--|--bringup/\n|--|--hardware/\n|--|--linux64/\n|--|--scripts/\n|--|--source/\n|--Platform-Directories (example : n6001, d5005, fseries-dk, iseries-dk)\n|--oneapi_asp_editor\n|--|--oneapi_asp_editor_hw.tcl\n|--|--oneapi_asp_editor.qpf\n|--|--oneapi_asp_editor.qsf\n|--|--scripts/\n|--|--ip/\n|--|--|--presets/\n|--|--|--oneapi_asp_editor.ip\n
Table below provides more information about the files in the oneapi-asp/oneapi_asp_editor directory.
Table B-2: oneapi_asp_editor Directory Contents
oneapi_asp_editor_hw.tcl to built the oneapi-asp oneapi_asp_editor.qpfoneapi_asp_editor.qsf Dummy Quartus project files to allow parameterization of the editor using IP Parameter Editor GUI ip Contains the oneapi_asp_editor.ip file and presets folder ip/oneapi_asp_editor.ip This is the default .ip file for the oneAPI ASP editor which is used to parameterize and generate oneapi-asp design files ip/presets This contains the preset files for OFS reference platforms. Table B-3 summarizes the preset files Table B-3: Presets
Device OFS Reference Platform Plaform Name inoneapi-asp Board Variant Preset File Agilex\u00ae 7 FPGA Intel\u00ae FPGA SmartNIC N6001-PL n6001 ofs_n6001 ofs_n6001_usm ofs_n6001_iopipes ofs_n6001_usm_iopipes ofs_n6001.qprs ofs_n6001_usm.qprs ofs_n6001_iopipes.qprs ofs_n6001_usm_iopipes.qprs Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) fseries-dk ofs_fseries-dk ofs_fseries-dk_usm ofs_fseries-dk.qprs ofs_fseries-dk_usm.qprs Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) iseries-dk ofs_iseries-dk ofs_Iseries-dk_usm ofs_iseries-dk.qprs ofs_iseries-dk_usm.qprs Stratix\u00ae 10 FPGA Intel\u00ae FPGA PAC D5005 d5005 ofs_d5005 ofs_d5005_usm ofs_d5005.qprs ofs_d5005_usm.qprs Note: For more information about the board variants, please refer to Table 5-6 in section 5.3.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#b2-oneapi-asp-editor-user-flow","title":"B.2 oneAPI ASP Editor User Flow","text":"The expected user flow with the oneAPI ASP Editor is shown in figure below.
Figure B-2: oneAPI ASP Editor User Flow
The oneAPI ASP Editor can be opened in IP Parameter Editor GUI in Quartus Prime. Building oneAPI ASPs requires FIM compilation to be complete successfully and a PR tree to be generated. The OPAE_PLATFORM_ROOT environment variable must be set to point to the PR tree generated during FIM build.
You can use the oneAPI ASP Editor GUI to set parameters for your oneAPI ASP. Based on your FIM and target platform, you can use one of the options below:
oneapi-asp for one of the OFS reference platforms, presets are provided in the oneapi-asp/oneapi_asp_editor/ip/presets directory. These presets should show up in the Presets window as shown in figure B-1. Table B-3 summarizes the available presets. Select and apply the preset matching the reference platform you are targeting. Save the settings and click on \"Generate HDL\".Notes: 1. For more information about how to apply presets, refer to Quartus\u00ae Prime Pro Edition User Guide: Getting Started 2. The oneAPI ASP editor generates only the selected board variant, if you would like to generate all board variants for an OFS reference platform, you will have to apply each preset and click \"Generate HDL\" for each one sequentially.
ofs_fim.qprs or set all parameters for the oneAPI ASP yourself. ofs_fim.qprs file is created from presets generated during FIM compilation flow. This file gets created when the oneAPI ASP editor is opened in the IP Parameter Editor GUI if the OPAE_PLATFORM_ROOT environment variable is set to point to the PR tree generated during FIM build. The ofs_fim.qprs covers some of the interface settings for the oneAPI ASP, there are other parameters required by the ASP that must be set by you when using the ofs_fim.qprs. For more information about which parameters are set by the ofs_fim.qprs, please refer to section B.3.Once the parameters are set, you can save optionally save your own preset file (if cutomizing editor parameters); then save the .ip (File -> Save) and click Generate HDL. For more information on the steps automated by Generate HDL, refer to table B-1.
Note: For more information about saving custom presets, refer to Quartus\u00ae Prime Pro Edition User Guide: Getting Started
For more information about each parameter, refer to section B.3. For a step-by-step example of using the oneAPI ASP editor, refer to the oneAPI Accelerator Support Package (ASP): Getting Started User Guide.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#b3-oneapi-asp-editor-parameters","title":"B.3 oneAPI ASP Editor Parameters","text":"Figure and table below summarizes the oneAPI ASP Editor parameter settings. Figures B-3 to B-8 along with Tables B-5 to B-9 provide details on the parameters under each tab of the editor Parameter window.
The tables B-3 to B-8 have an additional column Derived from FIM settings, the parameters marked yes in this are derived from the preset file generated from the FIM as shown in figure B-2. If you have applied this preset (ofs_fim.qprs) then the parameters marked Yes under Derived from FIM settings column are correctly set to support the interface provided by FIM, you only need to provide settings marked No under this column.
Figure B-3: oneAPI ASP Editor Parameters Summary
Table B-4: oneAPI ASP Editor Parameters Summary
Parameter/Tab Description AFU ID The AFU ID is stored in a register (board_afu_id register shown in Figures 5-2 to 5-5) in the oneAPI ASP hardware design. The board AFU ID is used by the MMD layer during board discovery. * 32-bit AFU ID MSB: Upper 32 bits of the board AFU ID * 32-bit AFU ID LSB: Lower 32 bits of the board AFU ID > Note: The AFU ID is not derived from ofs_fim.qprs, if you are using the ofs_fim.qprs file, this value must be set manually. Ensure the value in MMD (*_ASP_AFU_ID in oneapi-asp/common/source/CMakeLists.txt matches the value set in this register before building the oneapi-asp using build-asp.sh) Global Memory(On-board) Tab Contains the parameter settings for configuring the global memory for the oneAPI kernel. The settings from this tab are used to generate the global memory interface in the board_spec.xml as well as parameterize the global memory interface in the oneAPI ASP hardware design > Note: For more information about global_mem element in board_spec.xml, please refer to section 2.1.5 in this manual Unified Shared Memory(USM) Tab Contains the parameter settings for configuring the Unified Shared Memory interface. The settings from this tab are used to generate the global memory interface with allocation_type=host, shared in board_spec.xml to support USM. These parameters are also used in the oneAPI ASP hardware design for the USM datapath. > Note: For more information about the USM interface support in board_spec.xml, please refer to the section 2.1.5.1.3 in this manual Direct Memory Access(DMA) Tab This tab is reserved for future use. Please do not edit the DMA settings. Keep the Number of DMA Channels set to 1 Channels Tab Contains the parameter settings for configuring I/O Pipes to/from oneAPI kernel. The settings from this tab are used to generate the channels settings in board_spec.xml as well as parameterize the I/O pipes interface in oneAPI ASP hardware design > Note: For more information about channels, please refer to section 2.1.8 in this manual Project Settings Tab In addition to interface settings, oneAPI ASP design (hardware as well as the board_spec.xml requires some more details (e.g. project names, resources) to create oneAPI ASP directory sucessfully. This tab contains some of the additional settings required Figure B-4: oneAPI ASP Editor Global Memory(On-board) Parameters
Table B-5: oneAPI ASP Editor Global Memory(On-board) Parameters
Note: All parameters in the table below except Number of Global Memory Systems are for different for each global memory in the oneAPI ASP. If the Number of Global Memory Systems is increased, a new tab (titled Global Memory <id>, where <id> starts at 0 for the lowest global memory) gets added for each global memory and you must set the parameters for each additional global memory under its tab.
ofs_fim.qprs is applied) Description Number of Global Memory Systems -(see Note in description) No Set the numbr of global memory systems (for on-board memories) > Note: oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems Name No No The name of global memory (not editable, set to device), this value is used as the value for name attribute for global_mem element in board_spec.xml > Note: For more information about global_mem element attributes, please refer to table 2-7 in section 2.1.5 in this manual Maximum theoretical bandwidth Yes Yes The maximum theoretical bandwidth for the global memory, this is used as the value for max_bandwidth attribute for global_mem element in board_spec.xml > Note: For more information about calculating max_bandwidth, please refer to table 2-7 in section 2.1.5 in this manual Number of memory banks Yes Yes Number of memory banks in a single global memory (e.g. if your board has four DDR4 banks in a single global memory, this value must be set to 4). This value is used in the oneAPI kernel interface design in the oneAPI ASP hardware design files as well as to calculate number of interface for each global memory in the board_spec.xml Permitted Values: 1, 2, 4, 8 Data width Yes Yes Width of the data bus from oneAPI kernel to on-board memory bank (interface width to individual memory bank in a global memory). This is used as the value for width of interface in global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization > Note: For more information about interface, please refer to in section 2.1.4 in this manual Address width per bank Yes Yes The address width for each individual memory bank in the global memory. This is used to calculate the size and address attributes of each interface of global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization Burst size Yes No The burst size for each individual memory bank interface in the global memory. This is used to calculate the maxburst value of interface in global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization Pipeline stages (kernel to clock crosser) Yes No Number of pipeline stages in kernel to global memory data path (controls pipe depth of pipeline bridge on this data path in kernel_wrapper.v) Waitrequest allowance No No The value is not editable through the Editor and is calculated from number of Pipeline stages (kernel to clock crosser). This is used to set the waitrequest_allowance for interface in global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization Pipeline stages (MBD to global memory) Yes No Number of pipeline stages in the data path from Memory Bank Divider (MBD) to the global memory interface. This value is used in oneAPI ASP hardware design parameterization AVMM write acknowledge Yes No This is used in controlling the value for bsp_avmm_write_ack for interface in global_mem element in board_spec.xml as well as in deciding the addition of avmm_wr_ack_gen_inst block in the oneAPI ASP hardware design (refer to Figure 5-2 to 5-6) Configuration address No No This value is not editable and is calculated automatically by the oneapi_asp_editor_hw.tcl for all global memories. The value is used for the config_addr attribute of global_mem element in board_spec.xml > Note: For more information about config_addr, refer to table 2-7 in section 2.1.5 Interleaved bytes No No This value is not editable and is set to 4096 in oneapi-asp tag ofs-2024.2-1, this is a known issue, please refer to release notes for more information on the workaround. This value is used for the interleaved_bytes attribute of global_mem element in board_spec.xml > Note: For more information about the calculation of interleaved_bytes refer to table 2-7 in section 2.1.5 Latency (for oneAPI compiler) Yes No This value is used for latency attribute of global_mem element in board_spec.xml Figure B-5: oneAPI ASP Editor Unified Shared Memory Parameters
Table B-6: oneAPI ASP Editor Unified Shared Memory Parameters
Parameter/Tab Editable Derived from FIM Settings(only ifofs_fim.qprs is applied) Description Unified Shared Memory Interface Yes No Used to enable or disable Unified Shared Memory(USM) interface in the oneAPI ASP design. When this box is selected, a global_mem element with allocation_type=host, shared is added to the board_spec.xml and the data path for USM is added to the oneAPI ASP hardware design (Figures 5-3, 5-5 show the implementaion of the USM data path) Name No No The name of global memory interface for USM (not editable, set to host), this value is used as the value for name attribute for global_mem element for USM interface in board_spec.xml > Notes: * For more information about global_mem element attributes, please refer to table 2-7 in section 2.1.5 in this manual * For more information about Unified Shared Memory interface, refer to section 2.1.5.1.3 Maximum theoretical bandwidth Yes No Maximum bandwidth for the USM interface. This is used as the value for max_bandwidth attribute for global_mem element in board_spec.xml Number of interfaces No No This is not editable, set to 1 for USM interface Data width Yes No The data width for the USM interface, this value is used for the width of interface in global_mem element for USM in `board_spec.xml Address width Yes No The address width for Unified Shared Memory. This is used to calculate the size attribute for interface of global_mem element for USM in board_spec.xml Burst size Yes No The burst size for USM interface. This is used to calculate the maxburst value for interface in global_mem element for USM in board_spec.xml as well as to calculate the burst count setting for this interface in oneAPI ASP hardware design Pipeline stages Yes No Number of pipeline stages in USM data path (controls pipe depth of pipeline bridge on this data path in kernel_wrapper.v) Waitrequest allowance No No The value is not editable through the Editor and is calculated from number of Pipeline stages setting. This is used to set the waitrequest_allowance for interface in global_mem element for USM in board_spec.xml Interleaved bytes No No This value is not editable and is set to 1024, this value is used for the interleaved_bytes attribute of global_mem element for USM in board_spec.xml Latency (for oneAPI compiler) Yes No This value is used for latency attribute of global_mem element for USM in board_spec.xml Figure B-6: oneAPI ASP Editor Direct Memory Access(DMA) Parameters
Table B-7: oneAPI ASP Editor Direct Memory Access(DMA) Parameters
Parameter/Tab Editable Derived from FIM Settings(only ifofs_fim.qprs is applied) Description Number of DMA Channels -(see Note in description) No Reserved for future use - This value is used in the oneAPI ASP hardware design in implementation of the data path from host to DMA engine (see Figure 5-2 to 5-5 for the data path diagram) > Note: oneapi-asp tag ofs-2024.2-1 supports only 1 DMA channel Figure B-7: oneAPI ASP Editor Channels Parameters
Table B-8: oneAPI ASP Editor Channels Parameters
Parameter/Tab Editable Derived from FIM Settings(only ifofs_fim.qprs is applied) Description Number of I/O Channels Yes Yes Number of I/O pipes to/from oneAPI kernel. This value is used in generating the channels interface in board_spec.xml as well adding the I/O pipes interface in the oneAPI ASP hardware design (Figures 5-4 to 5-5 show the I/O pipes interface from oneAPI kernel to HSSI Subsystem) Data Width for I/O Channel <id> Yes Yes The width of each individual I/O pipe interface to/from oneAPI kernel. The <id> value starts at 0 for the first channel Figure B-8: oneAPI ASP Editor Project Settings Parameters
Table B-9: oneAPI ASP Editor Project Settings Parameters
Parameter/Tab Editable Derived from FIM Settings(only ifofs_fim.qprs is applied) Description Device Family Yes Yes Set to target device family for your platform, options are: * Stratix\u00ae 10 FPGA * Agilex\u00ae 7 FPGA Platform name Yes No Sets the name of the top-level directory that stores oneAPI ASP design files for your target platform and board variant Board variant name Yes No Sets the name for the board variant directory Quartus version Yes Yes Sets the Quartus version for the oneAPI ASP OneAPI compiler version Yes No Sets the oneAPI compiler version, this is used as the value for version attribute in board_env.xml > Note: For more information about board_env.xml attributes, refer to section 2.2 in this manual Snoop port enable Yes No When this is enabled, it enables the snoop port in the Memory Bank Divider (acl_asp_snoop_port) > Note: For more information about Memory Bank Divider, refer to section 3.1.1 Quartus project name Yes No Sets the value of project attribute of compile element in board_spec.xml > Note: For more information about the compile element attributes, refer to section 2.1.2 in this manual Quartus revision name Yes No Sets the value of revision attribute of compile element in board_spec.xml > Note: For more information about the compile element attributes, refer to section 2.1.2 in this manual Device model Yes No Specify the name of the device model file. This value is used to set the file name for device_model attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 ALMs Yes Yes Adaptive Logic Modules(ALMs) that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 FFs Yes Yes Flip-flops(FFs) that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 DSPs Yes Yes Digital Signal Processing(DSPs) blocks that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 M20Ks Yes Yes RAM (M20Ks) blocks that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 Kernel CRA data width No No This is not editable and is set to 64. This parameter sets the width of Kenel CRA interface and is used as the width value of kernel_cra port in interfaces element in board_spec.xml > Notes: * For more information about kernel_cra, please refer to table 3-4 in section 3.2.1 * For more information about interfaces element, refer to section 2.1.7 * For more information about interface attribute, refer to section 2.1.4 Kernel CRA pipeline stages Yes No Number of pipeline stages in host to kernel CRA Avalon\u00ae interface (controls pipe depth of pipeline bridge on this control path in kernel_wrapper.v) > Note: For more information about kernel_cra, please refer to table 3-4 in section 3.2.1 Kernel CRA waitrequest allowance No No The value is not editable through the Editor and is calculated from number of Kernel CRA pipeline stages setting. This is used to set the waitrequest_allowance of kernel_cra port in interfaces element in board_spec.xml Range for low clock No No Sets the lower threshold for the user clock frequency (kernel clock) range. This value is used in the creation of oneapi_afu.json file. This is not editable and is set to the following values based on target device: * Stratix\u00ae 10 FPGA: auto-400 * Agilex\u00ae 7 FPGA: auto-600 > Note: For more information about oneapi_afu.json, refer to table 5-7 in section 5.3 Range for high clock No No Sets the upper threshold for the user clock frequency (kernel clock) range. This value is used in the creation of oneapi_afu.json file. This is not editable and is set to the following values based on target device: * Stratix\u00ae 10 FPGA: auto-800 * Agilex\u00ae 7 FPGA: auto-1200 > Note: For more information about oneapi_afu.json, refer to table 5-7 in section 5.3"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#document-revision-history","title":"Document Revision History","text":"Date Release Changes May 26, 2023 2023.1 First release of oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack on https://ofs.github.io/ September 15, 2023 2023.2 1. Section 1: * Updated Figure 1-3 to add HSSI path 2. Section 2: * Added channels element in figure 2-1 and table 2-1, added information about board variants below this table * Added Section 2.1.8 on channels element 3. Section 4: * Added information about MMD API (table 4-1) and new sections 4.1.1 to 4.1.21 4. Section 5: * Moved oneapi-asp build flow information into new section 5.2 * Added new table 5-6 with oneAPI ASP board variants information * Added hardware design diagrams and information about new board variants with I/O pipes support (Hardware Design with IO Pipes and Hardware Design with IO Pipes and USM) * Updated hardware design diagrams to show PF/VF Mux/De-mux and added information about PF/VF mapping in section 5.3 * Added new section on UDP engine (section 5.3.4) * Updated figure 5-10 to remove import_opencl_kernel.tcl and add_bbb_to_pr_project.tcl * Updated table 5-13 to add mmd_iopipes.h and mmd_iopipes.cpp files Dec 13, 2023 2023.3-1 1. Section 2: * Update table 2-5 to add information about port parameter * Added new section 2.1.4.1 on port parameter * Update table & figure numbers * Changed \"master\" and \"slave\" ports to \"host\" & \"agent\" ports in figures 2. Section 4: * Added new executable call supported by initialize utility 3. Section 5: * Updated initialize utility information to add clang-offload-extractcommand usage * Updated table with hardware design files information, added information about ofs_asp.sdc * Updated compile flow diagram, added information about new gen-asp-quartus-report.tcl script * Updated hardware implementation diagrams for signal name and IP name changes, replaced mem_if_vtp block with host_mem_if_vtp * Updated OpenCL Memory Bank Divider to Memory Bank Divider * Updated OpenCL Kernel Interface to Kernel Interface Feb 2, 2024 2023.3-2 1. Section 5: * Updated Table 5-8 to add PF/VF mapping for different FIM configurations Mar 19, 2024 2024.1 1. Section 1: * Added links for FIM developer guides for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) * Added link to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs * Updated figure 1-3 to rename OpenCL Memory Bank Divider to Memory Bank Divider 2. Section 2: * Replaced links to FPGA Optimization Guide for Intel\u00ae oneAPI Toolkits with links to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs * Fixed format for table 2-6 * Updated figure 2-6 to rename OpenCL Memory Bank Divider to Memory Bank Divider * Replaced broken link for USM (Unified Shared Memory) 3. Section 3: * Updated Memory Bank Dividersection to add information about memory_bank_divider_hw.tcl. Replaced figure showing GUI for the IP with parameters in memory_bank_divider_hw.tcl and added information about these parameters in table 3-1 * Updated port name acl_bsp_snoop to acl_asp_snoopand acl_bsp_memorg_host to acl_asp_memorg_host in table 3-2 * Updated Kernel Interface section to add information about kernel_interface_hw.tcl. Replaced figure showing GUI for the IP with parameters in kernel_interface_hw.tcl and added information about these parameters in table 3-3 4. Section 5: * Replaced board_env.xml figure with code snippet * Added links to README for fseries-dk & iseries-dk oneapi-asp folders in section 5.1 * Updated directory structure in section 5.1 and added information about design files common to oneapi-asp for OFS reference platforms * Update file paths in table 5-2 * Update section for script file name changes from build-bsp.sh to build-asp.sh, setup-bsp.py to setup-asp.py, build-bsp-sw.sh to build-asp-sw.sh * Updated figure 5-2 to add a step in build-asp.sh script to copy common files to platform directory * Added Note on OPAE_SDK_REPO_BRANCH environment variable in table 5-5 * Added new OFS reference platforms (fseries-dk and iseries-dk) in table 5-6 * Update section for design file name changes from board.qsys to board_hw.tcl, ddr_board.qsys to ddr_board_hw.tcl, ddr_channel.qsys to ddr_channel_hw.tcl, bsp_logic.sv to asp_logic.sv, bsp_design_files.tcl to asp_design_files.tcl * Updated signal names bsp_mem_if to asp_mem_if, acl_bsp_memorg_host to acl_asp_memorg_host, host_mem_if_pa_bsp to host_mem_if_pa_asp * Updated hardware implementation diagrams (figure 5-2 to 5-5) for new file, signal names * Added information about RTL files to table 5-7 * Added iseries-dk_1pf_1vf.ofss in table 5-8 * Fixed section 5.3.1 header and added more information about PIM blocks in section 5.3.1 * Added new section 5.3.3 on board_hw.tcl * Updated figure 5-10 to add information about ip-deploy and qsys-generate for board_hw.tcl 5. Appendix: * Added appendix section with MMD debug variable information July 15, 2024 2024.2 1. Section 1: * Added document overview table (Table 1-1) 2. Section 2: * Added a section number 2.1.5.1.3 to Unified Shared Memory Section 3. Section 5: * Added information about oneapi_afu.json to table 5-7 * Added information about new parameters in board_hw.tcl in table 5-10(NUMBER_OF_DMA_CHANNELS, DATA_WIDTH, MAX_BURST_SIZE, KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE) and parameters.tcl * Added Note about acl_quartus_report.txt in section 5.3.5 4. Appendix A: * Modified appendix to split into sections explaining debug variables for MMD and runtime * Added new section for Extracting oneAPI Binary Informtion usingaocl bineditCommand * Updated table 5-10 for parameters for multiple global memories (NUMBER_OF_GLOBAL_MEMORY_SYSTEMS) 5. Appendix B: * Added Appendix B to describe new GUI based flow using oneAPI ASP Editor"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/","title":"Software Installation Guide: Open FPGA Stack for PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the PCIe Attach software stack on the host. This document will not cover the process of board installation or platform bring-up. After reviewing this document, a user shall be able to:
The information in this document is intended for customers evaluating a PCIe Attach shell. The PCIe Attach shell design is supported on a number of board offerings, including the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile), Intel\u00ae FPGA SmartNIC N6000/1-PL, and Intel\u00ae FPGA PAC D5005.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-2-software-and-component-version-summary-for-ofs-pcie-attach","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach","text":"The OFS PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are where the source code resides for each of the components discussed in this document.
Component Version Download Link Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 link OPAE SDK 2.13.0-3 2.13.0-3 Linux DFL intel-1.11.0-2 intel-1.11.0-2"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-3-release-pages-for-each-pcie-attach-platform","title":"Table 3: Release Page(s) for each PCIe Attach Platform","text":"This is a comprehensive list of the platform(s) whose software build and installation steps are covered in this document.
Platform Release Page Link Stratix\u00ae 10 FPGA https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Intel\u00ae FPGA SmartNIC N6001-PL https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#12-server-requirements","title":"1.2 Server Requirements","text":""},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#121-host-bios","title":"1.2.1 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#122-host-server-kernel-and-grub-configuration","title":"1.2.2 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack, on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions, and currently only supported the PCIe Attach release. Its usage is detailed in the relevant Quick Start Demonstration Guideline for your platform and will not be covered here.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#31-ofs-dfl-backport-kernel-driver-installation-environment-setup","title":"3.1 OFS DFL Backport Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL Backport GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.2.2 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 3.3 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
It is recommended you lock your Red Hat release version to 8.10 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\nInstall the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n\nsudo dnf install kernel-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-headers-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-devel-4.18.0-553.5.1.el8_10.x86_64\nNow you have the choice to either follow the steps in section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source or 3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#32-building-and-installing-the-ofs-dfl-backport-kernel-drivers-from-source","title":"3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Select the 4.18.0-553.5.1.el8_10 kernel as your next boot target on the build machine, then reboot.
# For multiple boots (until overwritten), use the following\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\n# Or select the kernel you want during boot time\nsudo reboot\n1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nintel-1.11.0-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n3. Build the kernel.
```bash\ncd /home/OFS/linux-dfl-backport\nmake && make rpm\n```\n4. Install the newly compiled RPM package and reboot.
```bash\nsudo rpm -i intel-fpga-dfl-dkms-1.11.0-2.2024.07.25.g276007e.noarch.rpm\n\nsudo reboot\n```\n5. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/4.18.0-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\nIf an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n6. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n7. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n8. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n9. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-4.18.0-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\nA list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#33-installing-the-ofs-dfl-backport-kernel-drivers-from-pre-built-packages","title":"3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page under the Artifacts tab. The name will resemble kernel-*.tar.gz. For the backport repository you can also choose to download packages under the GitHub action Build dkms packages, although your version will differ from the release version of the software.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\ntar xf kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\nsudo rpm -i kernel-4.18.0_dfl_2024.06.14_276007e/intel-fpga-dfl-dkms-1.11.0-2.2024.06.14.g276007e.noarch.rpm\n\n# Set this kernel as your new default boot target\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\nsudo reboot\nContinue from step 5 of Section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 4.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#41-opae-sdk-installation-environment-setup","title":"4.1 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package.Remove any currently installed OPAE packages.
sudo dnf remove opae*\nInitialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\nVerify that the correct tag/branch have been checkout out.
git describe --tags\n2.13.0-3\nSet up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\nThe following packages will be built in the same directory as create:
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\nCheck that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\nYou can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.13.0-3.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.13.0-3.x86_64-*.tar.gz\nFor a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/","title":"Software Installation Guide: Intel Agilex 7 SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the OFS SoC Attach software stack on the host and platform. After reviewing this document, a user shall be able to:
This document does not cover first time setup of the IPU Platform F2000X-PL platform.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating an SoC Attach release. This document will cover key topics related to initial bring up of the IPU Platform F2000X-PL software stack, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#table-2-software-component-version-summary-for-soc-attach","title":"Table 2: Software Component Version Summary for SoC Attach","text":"Name Location META-OFS https://github.com/OFS/meta-ofs, tag: ofs-2024.1-2 Host Operating System Ubuntu 22.04 LTS Host OPAE SDK 2.12.0-5 SoC OS meta-intel-ese Reference Distro 1.0-ESE (kirkstone) SoC Kernel Version 6.1.78-dfl SoC OPAE SDK 2.12.0-5 SoC Linux DFL ofs-2024.1-6.1-2Not all components shown in Table 2 will have an update available upon release. The OPAE SDK and Linux DFL software stacks are incorporated into a Yocto image and do not need to be downloaded separately.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#20-updating-the-ipu-platform-f2000x-pl","title":"2.0 Updating the IPU Platform F2000X-PL","text":"Every IPU Platform F2000X-PL ships with pre-programmed firmware for the FPGA user1, user2, and factory images, the Cyclone 10 BMC RTL and FW, the SoC NVMe, and the SoC BIOS. In this software installation guide, we will only be focusing on the building and loading of a new SoC NVMe image. Board setup and configuration for the IPU Platform F2000X-PL is discussed in the Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#30-compiling-a-custom-yocto-soc-image","title":"3.0 Compiling a Custom Yocto SoC Image","text":"Current Yocto image architecture for SoC Attach is based off of the IOTG Yoct-based ESE BSP, with the addition of the Linux DFL kernel including the latest DFL drivers for FPGA devices alongside the OPAE SDK user space software. The image targets x86_64 SoC FPGA devices but should boot on most UEFI-based machines. The source code and documentation for this image is hosted on the meta-ofs repository.
Build requirements exceed 100 GiB of disk space, depending on which image is built. As a reference point, on a system with two Intel(R) Xeon(R) E5-2699 v4 for a total of 44 CPU cores, the initial, non-incremental build takes less than an hour of wall time.
The repo tool is needed to clone the various Yocto layer repositories used in this example.
Note: If you are behind a firewall that prevents you from accessing references using the git:// protocol, you can use the following to redirect Git to use the corresponding https repositories for Yocto only: git config --global url.https://git.yoctoproject.org/.insteadOf git://git.yoctoproject.org/.
To compile the image as-is, use the following steps (as provided in meta-ofs):
Create and initialize the source directory.
mkdir ofs-yocto && cd ofs-yocto\ngit clone --recurse-submodules --shallow-submodules https://github.com/OFS/meta-ofs\ncd meta-ofs\ngit checkout tags/ofs-2024.1-2\nBuild packages and create an image.
cd examples/iotg-yocto-ese\nTEMPLATECONF=$PWD/conf source openembedded-core/oe-init-build-env build\nbitbake mc:x86-2022-minimal:core-image-full-cmdline\nThe resulting GPT disk image is available in uncompressed (.wic) and compressed form (.wic.gz) in meta-ofs/examples/iotg-yocto-ese/build/tmp-x86-2021-minimal-glibc/deploy/images/intel-corei7-64/. With no changes the uncompressed image size is ~21 GB.
The image type core-image-full-cmdline includes the familiar GNU core utilities, as opposed to core-image-minimal which uses BusyBox instead.
The example build configuration files under build/conf/ are symlinked from examples/iotg-yocto-ese/. To customize the image, start by modifying local.conf and bblayers.conf.
The uncompressed Yocto image can be loaded onto a flash drive as discussed in section 1.5.5 Creating a Bootable USB Flash Drive for the SoC and written to NVMe as the default boot target for the SoC as demonstrated in section 2.1 Updating the F2000X-PL ICX-D SoC NVMe.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#40-verifying-the-icx-d-soc-software-stack","title":"4.0 Verifying the ICX-D SoC Software Stack","text":"The reference SoC Attach FIM and unaltered FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Full supported functionality of the HEMs is documented in this platform's associated Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs and will not be covered here.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#50-setting-up-the-host","title":"5.0 Setting up the Host","text":"External SoC Attach supports testing Host to FPGA latency, MMIO latency, and MMIO bandwidth. This testing is accomplished using the utility host_exerciser on the host, which is included as a part of OPAE. This section will cover the installation and verification flow for a host interacting with the SoC Attach workload.
Review Section 1.2 Server Requirements of the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL for a list of changes required on the host to support an IPU Platform F2000X-PL and for a list of supported OS distributions. Installation will require an active internet connection to resolve dependencies.
The following software checks may be run on the host to verify the FPGA has been detected and has auto-negotiated the correct PCIe link width/speed. These commands do not require any packages to be installed. We are using PCIe BDF b1:00.0 as an example address.
# Check that the board has enumerated successfully.\n# Your PCIe BDF may differ from what is shown below.\n$ lspci | grep accel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce b1:00.1 Processing accelerators: Intel Corporation Device bcce\n\n# Check PCIe link status and speed. Width should be x16, and speed whould be 16GT/s\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\nThe OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit the opae.io page, and the Software Reference Manual
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access. If you wish to install pre-built artifacts instead of building the release yourself, skip steps 3 and 4.
Before Installing the newest version of OPAE you must remove any prior OPAE framework installations.
$ sudo apt-get remove opae*\nThe following system and Python3 package dependencies must be installed before OPAE may be built.
$ sudo apt-get install bison flex git ssh pandoc devscripts debhelper cmake python3-dev libjson-c-dev uuid-dev libhwloc-dev doxygen libtbb-dev libncurses-dev libspdlog-dev libspdlog1 python3-pip libedit-dev pkg-config libcli11-dev libssl-dev dkms libelf-dev gawk openssl libudev-dev libpci-dev libiberty-dev autoconf llvm\n\n$ python3 -m pip install setuptools pybind11 jsonschema\nClone the OPAE SDK repo. In this example we will use the top level directory OFS for our package installs.
$ mkdir OFS && cd OFS\n$ git init\n$ git clone https://github.com/OFS/opae-sdk\n$ cd opae-sdk\n$ git checkout tags/2.12.0-5\n\n# Verifying we are on the correct release tag\n$ git describe --tags\n2.12.0-5\nNavigate to the automatic DEB package build script location and execute.
$ cd OFS/opae-sdk/packaging/opae/deb $ ./create\n\n# Verify all packages are present\n$ ls | grep opae.*.deb\nopae_2.12.0-5_amd64.deb\nopae-dbgsym_2.12.0-5_amd64.ddeb\nopae-devel_2.12.0-5_amd64.deb\nopae-devel-dbgsym_2.12.0-5_amd64.ddeb\nopae-extra-tools_2.12.0-5_amd64.deb\nopae-extra-tools-dbgsym_2.12.0-5_amd64.ddeb\nInstall your newly built OPAE SDK packages.
$ cd OFS/opae-sdk/packaging/opae/deb\n$ sudo dpkg -i opae*.deb\nThe OPAE SDK version installed the host are identical to those installed on the SoC. A set of pre-compiled OPAE SDK artifacts are included in this release. These can be downloaded from ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell and installed without building/configuring.
$ tar xf opae-*.tar.gz\n$ sudo dpkg -i opae*.deb\nEnable iommu=on, pcie=realloc, and set hugepages as host kernel parameters.
# Check if parameters are already enabled\n$ cat /proc/cmdline\nIf you do not see intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200, then add them manually.
$ sudo vim /etc/default/grub\n\n# Edit the value for GRUB_CMDLINE_LINUX, add the values at the end of the variable inside of the double quotes. Example: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/rhel00-swap rd.lvm.lv=rhel00/root rd.lvm.lv=rhel00/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\"\n# Save your changes, then apply them with the following\n$ sudo grub2-mkconfig\n$ sudo reboot now\nAfter rebooting, check that proc/cmdline reflects your changes.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\nHere is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\nIn main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\nMapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\nThe below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\nThe below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\nIn main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\nThe host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\nWhen the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\nA Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\nTo run the host application, you will need to:
See the associated AFU Developer Guide for details.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/","title":"AFU Developer Guide: OFS for Agilex\u00ae 7 FPGA PCIe Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#1-introduction","title":"1. Introduction","text":"This document is a design guide for the creation of an Accelerator Functional Unit (AFU) using Open FPGA Stack (OFS) for Agilex\u00ae 7 FPGAs PCIe Attach. The AFU concept consists of separating out the FPGA design development process into two parts, the construction of the foundational FPGA Interface Manager (FIM), and the development of the Acceleration Function Unit (AFU), as shown in the diagram below.
This diagram shows the separation of FPGA board interface development from the internal FPGA workload creation. This separation starts with the FPGA Interface Manager (FIM) which consists of the external interfaces and board management functions. The FIM is the base system layer and is typically provided by board vendors. The FIM interface is specific to a particular physical platform. The AFU makes use of the external interfaces with user defined logic to perform a specific application. By separating out the lengthy and complicated process of developing and integrating external interfaces for an FPGA into a board allows the AFU developer to focus on the needs of their workload. OFS for Agilex\u00ae 7 FPGAs PCIe Attach provides the following tools for rapid AFU development:
Please notice in the above block diagram that the AFU region consists of static and partial reconfiguration (PR) regions where the PR region can be dynamically reconfigured while the remaining FPGA design continues to function. Creating AFU logic for the static region is described in Shell Developer Guide for the associated platform. This guide covers logic in the AFU Main region.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#11-document-organization","title":"1.1. Document Organization","text":"This document is organized as follows:
This guide provides theory followed by tutorial steps to solidify your AFU development knowledge.
NOTE:
This guide uses the Intel\u00ae FPGA SmartNIC N6001-PL as the platform for all tutorial steps. Additionally, this guide and the tutorial steps can be used with other platforms, such as the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile).
Some of the document links in this guide are specific to the Intel\u00ae FPGA SmartNIC N6001-PL. If using a different platform, please use the associated documentation for your platform as there could be differences in building the FIM and downloading FIM images.
If you have worked with previous Altera\u00ae Programmable Acceleration products, you will find out that OFS for Agilex\u00ae 7 FPGAs PCIe Attach is similar. However, there are differences and you are advised to carefully read and follow the tutorial steps to fully understand the design tools and flow.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#12-prerequisite","title":"1.2. Prerequisite","text":"This guide assumes you have the following FPGA logic design-related knowledge and skills:
You are strongly encouraged to review the Shell Developer Guide for the associated platform.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#13-acceleration-functional-unit-afu-development-flow","title":"1.3. Acceleration Functional Unit (AFU) Development Flow","text":"The AFU development flow is shown below:
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#131-understanding-platform-capabilities","title":"1.3.1. Understanding Platform Capabilities","text":"The block diagram of the N6001 Board is shown below:
The N6001 FIM provided with this release is shown below:
This release FIM provides the following features:
The OFS high level data flow is shown below:
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#133-considerations-for-pim-usage","title":"1.3.3. Considerations for PIM Usage","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
The following resources are available to assist in creating an AFU:
PIM Core Concepts provides details on using the PIM and its capabilities.
PIM Based AFU Developer Guide provides details on interfacing your AFU to the FIM using the PIM.
Multi-PCIe Link AFUs provides details on encapsulation of multiple FPGA device connections as a single OPAE handle.
The examples-afu repo provides several AFU examples:
Example Description PIM-based Hybrid Native clocks Example AFU using user configurable clocks. X copy_engine Example AFU moving data between host channel and a data engine. X dma Example AFU moving data between host channel and local memory with a DMA. X hello_world Example AFU sending \"Hello World!\" to host channel. X X X local_memory Example AFU reading and writing local memory. X XThese examples can be run with the current OFS FIM package. There are three AFU types of examples provided (PIM based, hybrid and native). Each example provides the following:
The figure below shows the interfaces available to the AFU in this architecture. It also shows the design hierarchy with module names from the fim (top.sv) to the PR region AFU (afu_main.sv). One of the main differences from the Stratix 10 PAC OFS architecture to this one is the presence of the static port gasket region (port_gasket.sv) that has components to facilitate the AFU and also consists of the PR region (afu_main.sv) via the PR slot. The Port Gasket contains all the PR specific modules and logic, e.g., PR slot reset/freeze control, user clock, remote STP etc. Architecturally, a Port Gasket can have multiple PR slots where user workload can be programmed into. However, only one PR slot is supported for OFS Release for Agilex\u00ae. Everything in the Port Gasket until the PR slot should be provided by the FIM developer. The task of the AFU developer is to add their desired application in the afu_main.sv module by stripping out unwanted logic and instantiating the target accelerator. As shown in the figure below, here are the interfaces connected to the AFU (highlighted in green) via the PCIe Attach FIM:
This section covers the setup of the AFU development environment.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#21-afu-development-environment-overview","title":"2.1. AFU development environment overview","text":"A typical development and hardware test environment consists of a development server or workstation with FPGA development tools installed and a separate server with the target OFS compatible FPGA PCIe card installed. The typical usage and flow of data between these two servers is shown below:
Note: both development and hardware testing can be performed on the same server if desired.
This guide uses Intel\u00ae FPGA SmartNIC N6001-PL as the target OFS compatible FPGA PCIe card for demonstration steps. The Intel\u00ae FPGA SmartNIC N6001-PL must be fully installed following the Board Installation Guide: OFS for Agilex\u00ae 7 PCIe Attach Development Kits. If using a different OFS FPGA PCIe card, contact your supplier for instructions on how to install and operate user developed AFUs.
The following is a summary of the steps to set up for AFU development:
Building AFUs with OFS for Agilex\u00ae requires the build machine to have at least 64 GB of RAM.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#22-installation-of-quartus-and-required-patches","title":"2.2. Installation of Quartus and required patches","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\nCreate the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
Download your required Quartus Prime Pro Linux version here.
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\nFor example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\nVerify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\nMake sure support tools are installed and meet version requirements.
The OFS provided Quartus build scripts require the following tools. Verify these are installed in your development environment.
Item Version Python 3.6.8 GCC 8.5.0 cmake 3.15 git 1.8.3.1 perl 5.8.8"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#24-installation-of-opae-sdk","title":"2.4. Installation of OPAE SDK","text":"Working with the Intel\u00ae FPGA SmartNIC N6001-PL card requires opae-2.13.0-3. Follow the instructions in the Follow the instructions in the Software Installation Guide: OFS for PCIe Attach FPGAs to build and install the required OPAE SDK for the Intel\u00ae FPGA SmartNIC N6001-PL. Make sure to check out the cloned repository to tag 2.13.0-3 and branch release/2.13.0.
$ git checkout tags/2.13.0-3 -b release/2.13.0\nNote: The tutorial steps provided in the next sections assume the OPAE SDK is installed in default system locations, under the directory, /usr. In most system configurations, this will allow the OS and tools to automatically locate the OPAE binaries, scripts, libraries and include files required for the compilation and simulation of the FIM and AFUs.
The ofs-platform-afu-bbb repository contains the PIM files as well as example PIM-based AFUs that can be used for testing and demonstration purposes. This guide will use the host_chan_mmio AFU example in the ofs-platform-afu-bbb repository and the hello_world sample accompanying the examples-afu repository to demonstrate how to synthesize, load, simulate, and test a PIM-based AFU using the Intel\u00ae FPGA SmartNIC N6001-PL card with the PCIe Attach FIM.
Execute the next commands to clone the BBB repository.
# Create top level directory for AFU development\n$ mkdir OFS_BUILD_ROOT\n$ cd OFS_BUILD_ROOT\n$ export OFS_BUILD_ROOT=$PWD\n# Clone the ofs-platform-afu-bbb repository.\n$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/ofs-platform-afu-bbb.git\n$ cd ofs-platform-afu-bbb\n$ git checkout tags/ofs-2024.2-1\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n# Verify retrieval\n$ ls\nLICENSE plat_if_develop plat_if_release plat_if_tests README.md\nThe documentation in the ofs-platform-afu-bbb repository further addresses - The PIM concept. - The structure of the PIM-based AFU examples. - How to generate a release and configure the PIM. - How to connect an AFU to an FIM.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#26-build-or-download-the-relocatable-pr-build-tree","title":"2.6. Build or download the relocatable PR build tree","text":"A relocatable PR build tree is needed to build the AFU partial reconfiguration area for the intended FIM. The tree is relocatable and may be copied to a new location. It does not depend on files in the original FIM build.
You can use the Intel\u00ae FPGA SmartNIC N6001-PL release package and download the PR build tree and FIM images, to develop your AFU. These are located at OFS-N6001 release
Or you can build your own FIM and generate the PR build tree during the process.
To download and untar the pr_build_template:
$ cd $OFS_BUILD_ROOT\n$ wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/n6001-images_ofs-2024-2-1.tar.gz\n$ tar -zxvf n6001-images_ofs-2024-2-1.tar.gz\n$ cd n6001-images_ofs-2024-2-1/\n$ cd pr_build_template\n$ export OPAE_PLATFORM_ROOT=$PWD\nTo build your own FIM and generate the PR build tree for the Intel\u00ae FPGA SmartNIC N6001-PL platform, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs and follow the Out-of-Tree PR FIM build flow. If you are using a different platform, refer to the Shell Developer Guide for your platform and follow the Out-of-Tree PR FIM build flow.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#27-download-fim-to-fpga","title":"2.7. Download FIM to FPGA","text":"The AFU requires that the FIM from which the AFU is derived be loaded onto the FPGA.
If you are using the Intel\u00ae FPGA SmartNIC N6001-PL release package downloaded in the previous section:
$ cd $OFS_BUILD_ROOT/n6001-images_ofs-${{ env.COMMON_OFS_RELEASE_TAR }}/\nIf you are generating your own FIM, use the unsigned FPGA binary images from your FIM build.
Downlaod the FIM to the Intel\u00ae FPGA SmartNIC N6001-PL platform. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.
$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin <N6001 SKU2 PCIe b:d.f>\n$ sudo fpgasupdate ofs_top_page2_unsigned_user2.bin <N6001 SKU2 PCIe b:d.f>\n$ sudo rsu fpga --page=user1 <N6001 SKU2 PCIe b:d.f>\nIf you are using a different platform, refer to the documentation for your platform to download the FIM images onto your Agilex\u00ae PCIe Attach Platform.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#28-set-up-required-environment-variables","title":"2.8. Set up required Environment Variables","text":"Set the required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks. You can create a simple script to set these variables and save time going forward.
# If not already done, export OFS_BUILD_ROOT to the top level directory for AFU development\n$ export OFS_BUILD_ROOT=<path to ofs build directory>\n\n# If not already done, export OPAE_PLATFORM_ROOT to the PR build tree directory\n$ export OPAE_PLATFORM_ROOT=<path to pr build tree>\n\n# If not already done, export OFS_PLATFORM_AFU_BBB to the clone of ofs-platform-afu-bbb repository which contains PIM files and AFU examples.\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu-bbb>\n\n# Quartus Tools\n# Note, QUARTUS_HOME is your Quartus installation directory, e.g. $QUARTUS_HOME/bin contains Quartus executable.\n$ export QUARTUS_HOME=<user_path>/intelFPGA_pro/24.1/quartus\n$ export QUARTUS_ROOTDIR=$QUARTUS_HOME\n$ export QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\n$ export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n$ export IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys\n$ export PATH=$QUARTUS_HOME/bin:$QSYS_ROOTDIR/bin:$QUARTUS_HOME/../sopc_builder/bin/:$PATH\n# OPAE SDK release\n$ export OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# OPAE and MPF libraries must either be on the default linker search paths or on both LIBRARY_PATH and LD_LIBRARY_PATH. \n$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\nIn this section, you will use the relocatable PR build tree created in the previous steps from the FIM to compile an example PIM-based AFU. This section will be developed around the host_chan_mmio and hello_world AFU examples to showcase the synthesis of a PIM-based AFU.
The build steps presented below demonstrate the ease in building and running an actual AFU on the board. To successfully execute the instructions in this section, you must have set up your development environment and have a relocateable PR Build tree as instructed in section 2 of this document.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#31-creating-the-afu-synthesis-environment","title":"3.1. Creating the AFU Synthesis Environment","text":"The PIM flow provides the script afu_synth_setup to create the synthesis environment to build the AFU examples. See how to use it below.
usage: afu_synth_setup [-h] -s SOURCES [-p PLATFORM] [-l LIB] [-f] dst\n\nGenerate a Quartus build environment for an AFU. A build environment is\ninstantiated from a release and then configured for the specified AFU. AFU\nsource files are specified in a text file that is parsed by rtl_src_config,\nwhich is part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA platform name.\n -l LIB, --lib LIB FPGA platform release hw/lib directory. If not\n specified, the environment variables OPAE_FPGA_HW_LIB\n and then BBS_LIB_PATH are checked.\n -f, --force Overwrite target directory if it exists.\nThe $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using Avalon and AXI interfaces. However, this guide will use the AXI version of the host_chan_mmio AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the actual AFU hardware.
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\nExecute afu_synth_setup as follows to create the synthesis environment for a host_chan_mmio AFU that fits the PCIe Attach FIM previously constructed.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_dev\nafu_dev directory just created. From there, execute the afu_synth command. The successful completion of the command will produce the host_chan_mmio.gbs file under the synthesis environment directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev. $ cd afu_dev\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating host_chan_mmio.gbs\n==================================\n...\n...\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\nThe previous output indicates the successful compilation of the AFU and the compliance with the timing requirements. Analyze the reports generated in case the design does not meet timing. The timing reports are stored in the directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev/build/syn/board/n6001/syn_top/output_files/timing_report.
Once the compilation finishes successfully, load the new host_chan_mmio.gbs bitstream file into the partial reconfiguration region of the target Intel\u00ae FPGA SmartNIC N6001-PL board. Keep in mind, that the loaded image is dynamic - this image is not stored in flash and if the card is power cycled, then the PR region is re-loaded with the default AFU.
To test the AFU in actual hardware, load the host_chan_mmio.gbs to the Intel\u00ae FPGA SmartNIC N6001-PL card. For this step to be successful, the PCIe Attach FIM must have already been loaded to the Intel\u00ae FPGA SmartNIC N6001-PL card following the steps described in Section 2 of this document. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.
Verify Board and PCIe b.d.f. For the following example, the N6001 SKU2 PCIe b:d.f is B1:00.0, however this may be different in your system.
$ fpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\n...\nDownload AFU.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev\n$ sudo fpgasupdate host_chan_mmio.gbs B1:00.0\n[sudo] password for <<Your username>>: [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\nSet up host to interface with the newly loaded AFU.
List the PFs available, the default N6001 FIM has 5 PFs.
$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\nCreate the Virtual Functions (VFs) provided by the FIM, the default N6001 FIM has 3 VFs. If your FIM uses only PFs, skip this step.
$ sudo pci_device B1:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.5 Processing accelerators: Intel Corporation Device bccf\nB1:00.6 Processing accelerators: Intel Corporation Device bccf\nB1:00.7 Processing accelerators: Intel Corporation Device bccf\nBind PFs and VFs to VFIO driver (except PF0/B1:00.0, which is the FME PF).
# Enter your username.\n$ sudo opae.io init -d 0000:b1:00.1 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 183\nAssigning /dev/vfio/183 to ceg\nChanging permissions for /dev/vfio/183 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.2 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 184\nAssigning /dev/vfio/184 to ceg\nChanging permissions for /dev/vfio/184 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.3 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x1af4,0x1000) at 0000:b1:00.3 from virtio-pci\nBinding (0x1af4,0x1000) at 0000:b1:00.3 to vfio-pci\niommu group for (0x1af4,0x1000) at 0000:b1:00.3 is 185\nAssigning /dev/vfio/185 to ceg\nChanging permissions for /dev/vfio/185 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.4 <<Your username>>\n[sudo] password for <<Your username>>: Unbinding (0x8086,0xbcce) at 0000:b1:00.4 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.4 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.4 is 186\nAssigning /dev/vfio/186 to ceg\nChanging permissions for /dev/vfio/186 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.5 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.5 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.5 is 315\nAssigning /dev/vfio/315 to ceg\nChanging permissions for /dev/vfio/315 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.6 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.6 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.6 is 316\nAssigning /dev/vfio/316 to ceg\nChanging permissions for /dev/vfio/316 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.7 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 317\nAssigning /dev/vfio/317 to ceg\nChanging permissions for /dev/vfio/317 to rw-rw----\nVerify the new AFU is loaded. The host_chan_mmio AFU GUID is \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\".
$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xEC00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 76d7ae9c-f66b-461f-816a-5428bcebdbc5\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x6098000000000000\nPCIe s:b:d.f : 0000:B1:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 1aae155c-acc5-4210-b9ab-efbd90b970c4\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\nNow, navigate to the directory of the host_chan_mmio AFU containing the host application's source code, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw. Once there, compile the host_chan_mmio host application and execute it on the host server to excercise the functionality of the AFU.
Note: If OPAE SDK libraries were not installed in the default systems directories under /usr, you need to set the OPAE_LOC, LIBRARY_PATH, and LD_LIBRARY_PATH environment variables to the custom locations where the OPAE SDK libraries were installed.
# Move to the sw directory of the the host_chan_mmio AFU. This directory holds the source for the host application.\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw\n$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n$ make\n\n# Run the application\n$ ./host_chan_mmio\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\nThe platform-independent examples-afu repository also provides some interesting example AFUs. In this section, you will compile and execute the PIM based hello_world AFU. The RTL of the hello_world AFU receives from the host application an address via memory mapped I/O (MMIO) write and generates a DMA write to the memory line at that address. The content written to memory is the string \"Hello world!\". The host application spins, waiting for the memory line to be updated. Once available, the software prints out the string.
The hello_world example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using CCIP, Avalon, and AXI interfaces. However, this guide will use the AXI version of the AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the AFU hardware.
hello_world\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 ccip\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u2514\u2500\u2500 hello_world.json\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 hello_world\n \u251c\u2500\u2500 hello_world.c\n \u251c\u2500\u2500 Makefile\n \u2514\u2500\u2500 obj\n \u251c\u2500\u2500 afu_json_info.h\n \u2514\u2500\u2500 hello_world.o\nThe following instructions can be used to compile other AFU samples accompanying this repository.
If not done already, download and clone the examples-afu repository.
$ cd $OFS_BUILD_ROOT $ git clone https://github.com/OFS/examples-afu.git\n$ cd examples-afu\n$ git checkout tags/ofs-2024.2-1\nCompile the hello_word sample AFU.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_synth_setup --source hw/rtl/axi/sources.txt afu_dev\n$ cd afu_dev\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating hello_world.gbs\n==================================\n.\n.\n.\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'hello_world.gbs'\nDesign meets timing\n===========================================================================\nTo test the AFU in actual hardware, load the hello_world.gbs to the Intel\u00ae FPGA SmartNIC N6001-PL card. For this step to be successful, the PCIe Attach FIM must have already been loaded to the Intel\u00ae FPGA SmartNIC N6001-PL card following the steps described in Section 2 of this document. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.
Verify Board and PCIe b.d.f. For the following example, the N6001 SKU2 PCIe b:d.f is B1:00.0, however this may be different in your system.
$ fpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\n...\nDownload AFU.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_dev\n$ sudo fpgasupdate hello_world.gbs B1:00.0\n [sudo] password for <<Your username>>: [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\nSet up your Intel\u00ae FPGA SmartNIC N6001-PL board to work with the newly loaded hello_world.gbs file.
# List the PF's available, the default N6001 FIM has 5 PF's\n$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\nDownload AFU.
# Create the Virtual Functions (VFs) provided by the FIM, the default N6001 FIM has 3 VFs. \n# If your FIM uses only PFs, skip this step.\n$ sudo pci_device B1:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.5 Processing accelerators: Intel Corporation Device bccf\nB1:00.6 Processing accelerators: Intel Corporation Device bccf\nB1:00.7 Processing accelerators: Intel Corporation Device bccf\nBind PFs and VFs to VFIO driver (except PF0/B1:00.0, which is the FME PF).
#Enter your username\n$ sudo opae.io init -d 0000:b1:00.1 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 183\nAssigning /dev/vfio/183 to ceg\nChanging permissions for /dev/vfio/183 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.2 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 184\nAssigning /dev/vfio/184 to ceg\nChanging permissions for /dev/vfio/184 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.3 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.3 from virtio-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.3 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.3 is 185\nAssigning /dev/vfio/185 to ceg\nChanging permissions for /dev/vfio/185 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.4 <<Your username>>\n[sudo] password for <<Your username>>: Unbinding (0x8086,0xbcce) at 0000:v1:00.4 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.4 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.4 is 186\nAssigning /dev/vfio/186 to ceg\nChanging permissions for /dev/vfio/186 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.5 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.5 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.5 is 315\nAssigning /dev/vfio/315 to ceg\nChanging permissions for /dev/vfio/315 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.6 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.6 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.6 is 316\nAssigning /dev/vfio/316 to ceg\nChanging permissions for /dev/vfio/316 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.7 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 317\nAssigning /dev/vfio/317 to ceg\nChanging permissions for /dev/vfio/317 to rw-rw----\nVerify the new AFU is loaded. The hello_world AFU GUID is \"c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\".
$ fpgainfo port\n\n//****** PORT ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE098000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0xC098000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0xA098000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n//****** PORT ******//\nObject Id : 0x8098000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x6098000000000000\nPCIe s:b:d.f : 0000:B1:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 1aae155c-acc5-4210-b9ab-efbd90b970c4\n//****** PORT ******//\nObject Id : 0x4098000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x2098000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\nCompile and execute the host application of the hello_world AFU. You should see the application outputs the \"Hello world!\" message in the terminal.
# Move to the sw directory of the hello_world AFU\n$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n\n$ make\n\n# Launch the host application\n$ ./hello_world\n Hello world!\nAn OPAE compliant AFU specifies the frequency of the uclk_usr and uclk_usr_div2 clocks through the JSON file for AFU configuration located under the <afu_example>/hw/rtl directory of an AFU design. For instance, the AFU configuration file of the host_chan_mmio example is $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/host_chan_mmio.json.
The AFU specifies the frequency for uClk_usr in its platform configuration file using the following key:value pairs:
\"clock-frequency-high\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\n \"clock-frequency-low\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\nThese key:value tuples are used to configure the PLL of the target platform that provides the user clocks through the AFU clocks interface. In addition, the specified frequency affects the timing closure process on the user clocks during AFU compilation.
Setting the value field to a float number (e.g., 315.0 to specify 315 MHz) drives the AFU generation process to close timing within the bounds set by the low and high values and sets the AFU's JSON metadata to specify the user clock PLL frequency values.
The following example shows the JSON file of the host_chan_mmio to set the AFU uClk to 500 MHz and uClk_div2 to 250 MHz.
{\n \"version\": 1,\n \"afu-image\": {\n \"power\": 0,\n \"clock-frequency-high\": 500,\n \"clock-frequency-low\": 250,\n \"afu-top-interface\":\n {\n \"class\": \"ofs_plat_afu\"\n },\n \"accelerator-clusters\":\n [\n {\n \"name\": \"host_chan_mmio\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\"\n }\n ]\n }\n}\nSave the changes to host_chan_mmio.json file, then execute the afu_synth_setup script to create a new copy of the AFU files with the modified user clock settigns.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_clks\n\nCopying build from /home/<user_area>/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/build...\nConfiguring Quartus build directory: build_n6001_afu_clks/build\nLoading platform database: /home/<user_area>/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting platform/platform_afu_top_config.vh\nWriting platform/platform_if_addenda.qsf\nWriting ../hw/afu_json_info.vh\nhost_chan_mmio AFU with the new frequency values. $ cd afu_clks\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\nDuring the compilation phase, you will observe the Timing Analyzer uses the specified user clock frequency values as the target to close timing.
AFU developers must ensure the AFU hardware design meets timing. The compilation of an AFU that fails timing shows a message similar to the following.
.\n.\n.\n\nWrote host_chan_mmio.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\n*** Design does not meet timing\n *** See build/syn/board/n6001/syn_top/output_files/timing_report\n\n===========================================================================\nThe previous output indicates the location of the timing reports for the AFU designer to identify the failing paths and perform the necessary design changes. Next, is a listing of the timing report files from a host_chan_mmio AFU that fails to meet timing after modifying the user clock frequency values.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/n6001/afu_clks\n$ ls build/syn/board/n6001/syn_top/output_files/timing_report\n\nclocks.rpt clocks.sta.fail.summary clocks.sta.pass.summary\nWarning: AFU developers must inform software developers of the maximum operating frequency (Fmax) of the user clocks to avoid any unexpected behavior of the accelerator and potentially of the overall system.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#4-simulating-an-afu-using-ase","title":"4. Simulating an AFU using ASE","text":"The Application Simulation Environment (ASE) is a hardware/software co-simulation environment for your AFU. See diagram below illustrating ASE operation:
ASE uses the simulator Direct Programming Interface (DPI) to provide HW/SW connectivity. The PCIe connection to the AFU under testing is emulated with a transactional model.
The following list describes ASE operation:
The remainder of this section is a tutorial providing the steps on how to run ASE with either Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae using an example AFU and the AFU build tree previously created in this guide.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#41-set-up-steps-to-run-ase","title":"4.1. Set Up Steps to Run ASE","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#411-install-opae-sdk","title":"4.1.1. Install OPAE SDK","text":"The N6001 SKU2 card requires 2.13.0-3. Follow the instructions provided in the Follow the instructions in the Software Installation Guide: OFS for PCIe Attach FPGAs to build and install the required OPAE SDK for the Intel\u00ae FPGA SmartNIC N6001-PL. Make sure to check out the cloned repository to tag 2.13.0-3 and branch release/2.13.0.
$ git checkout tags/2.13.0-3 -b release/2.13.0\nASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK.
If not done already, set the environment variables as described in section, Set Up AFU Development Environment.
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim $ git checkout tags/2.13.0-2 -b release/2.13.0\nCreate a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\nOptionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> .. /usr. $ sudo make install The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ cd /usr/bin\n$ export PATH=$PWD:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ cd /usr/lib\n$ export LIBRARY_PATH=$PWD\n$ cd /usr/lib64\n$ export LD_LIBRARY_PATH=$PWD\n$ cd $OFS_BUILD_ROOT/ofs-platform-afu-bbb\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to Modelsim installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\nThe $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files:
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\nThis example AFU contains examples using both Avalon and AXI interface buses. This guide will use the AXI version of the host_chan_mmio AFU.
ASE uses client-server application architecture to deliver hardware/software co-simulation. You require one shell for the hardware based simulation and another shell where the software application is running. The hardware is started first with a simulation compilation and simulator startup script, once the simulator has loaded the design, it will wait until the software process starts. Once the software process starts, the simulator proceeds. Transaction logging and waveform capture is performed.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#421-set-up-and-run-the-hw-simulation-process","title":"4.2.1 Set Up and Run the HW Simulation Process","text":"You will run the afu_sim_setup script to create the scripts for running the ASE environment. The afu_sim_setup script has the following usage:
usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\nRun afu_sim_setup to create the ASE simulation environment for the host_chan_mmio example AFU. The '-t VCS' option indicates to prepare the ASE simulation environment for Synopsys\u00ae VCS\u00ae.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n\n$ afu_sim_setup -s ./hw/rtl/test_mmio_axi1.txt -t VCS afu_sim\n\nCopying ASE from /opae-sdk/install-opae-sdk/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /ofs-agx7-pcie-attach/work_pr/build_tree/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\nThe afu_sim_setup creates the ASE scripts in the directory host_chan_mmio_sim where the afu_sim_setup script was run. Start the simulator as shown below:
$ cd afu_sim\n$ make\n$ make sim\nThis process launches the AFU hardware simulator. Before moving to the next section, pay attention to the simulator output highlighted in the image below.
The simulation artifacts are stored in host_chan_mmio/work and consist of:
log_ase_events.tsv\nlog_ofs_plat_host_chan.tsv log_ofs_plat_local_mem.tsv log_pf_vf_mux_A.tsv log_pf_vf_mux_B.tsv Open an additional shell to build and run the host application that communicates with the actual AFU hardware. Set up the same environment variable you have set up in the shell you have been working on until this point.
Additionally, as indicated by the hardware simulator output that is currently executing in the \"simulator shell\", copy and paste the line \"export ASE_WORKDIR=...\", into the new \"software shell\". See the last image of the previous section.
$ export ASE_WORKDIR=$OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_sim/work\nhost_chan_mmio AFU example to compile the host application. $ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw $ make\n\nafu_json_mgr json-info --afu-json=../hw/rtl/host_chan_mmio.json --c-hdr=obj/afu_json_info.h\nWriting obj/afu_json_info.h\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c main.c -o obj/main.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c test_host_chan_mmio.c -o obj/test_host_chan_mmio.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/connect.c -o obj/connect.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/csr_mgr.c -o obj/csr_mgr.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/hash32.c -o obj/hash32.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/test_data.c -o obj/test_data.o\ncc -o host_chan_mmio obj/main.o obj/test_host_chan_mmio.o obj/connect.o obj/csr_mgr.o obj/hash32.o obj/test_data.o -z noexecstack -z relro -z now -pie -luuid -lopae-c-ase\nNow, launch the host application to exercise the AFU hardware running on the simulator shell. The next image shows the AFU hardware simulation process on the left side shell. The right hand shell shows the host application's output of a successful simulation.
$ with_ase ./host_chan_mmio\n [APP] Initializing simulation session ...\nRunning in ASE mode\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n [APP] Deinitializing simulation session\n [APP] Took 1,003,771,568 nsec\n [APP] Session ended\n
Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
$ make wave\nThis brings up the VCS\u00ae simulator GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | afu , as shown below.
Right click on the afu (afu) entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
In this section you will quickly simulate the PIM-based hello_world sample AFU accompanying the examples-afu repository.
Set the environment variables as described in section 4.1. Set Up Steps to Run ASE.
Prepare an RTL simulation environment for the AXI version of the hello_world AFU.
Simulation with ASE requires two software processes, one to simulate the AFU RTL and the other to run the host software that excercises the AFU. To construct an RTL simulation environment under the directory simulation, execute the following.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_sim_setup -s ./hw/rtl/axi/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/<user_area>/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\nThe afu_sim_setup script constructs an ASE environment in the hello_world_sim subdirectory. If the command fails, confirm that the path to the afu_sim_setup is on your PATH environment variable (in the OPAE SDK bin directory) and that your Python version is at least 2.7.
$ cd afu_sim\n$ make\n$ make sim The previous commands will build and run the Synopsys\u00ae VCS\u00ae RTL simulator, which prints a message saying it is ready for simulation. The simulation process also prints a message instructing you to set the ASE_WORKDIR environment variable in a second shell.
Open a second shell where you will build and execute the host software. In this new \"software shell\", set up the environment variables you have set up so far in the \"hardware simulation\" shell.
Also, set the ASE_WORKDIR environment variable following the instructions given in the \"hardware simulation\" shell.
$ export ASE_WORKDIR=$OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_sim/work\nhello_world AFU sample to build the host software. $ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n$ make hello_world host application to resume the work of the RTL simulation. The host software process and the RTL simulation execute in lockstep. If successful, you should see the Hello world! output.$ with_ase ./hello_world\n\n[APP] Initializing simulation session ...\nHello world!\n[APP] Deinitializing simulation session\n[APP] Took 43,978,424 nsec\n[APP] Session ended\nThe image below shows the simulation of the AFU hardware and the execution of the host application side-by-side.
make wave\nThis brings up the DVE GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | hello_afu, as shown below.
Right click on the hello_afu entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
The OPAE SDK provides a remote Signal Tap facility. It also supports the following in system debug tools included with the Quartus Prime Pro Edition:
This section is a short guide on adding remote Signal Tap instances to an AFU for in system debugging. You can follow the steps in the following sections, in order of execution to create an instrumented AFU. The host_chan_mmio AFU is used in this guide as the target AFU to be instrumented.
You need a basic understanding of Signal Tap. Please see the Signal Tap Logic Analyzer: Introduction & Getting Started Web Based Training for more information.
You will run with a Signal Tap GUI running locally on the server with the Intel\u00ae FPGA SmartNIC N6001-PL as shown below:
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#51-adding-rstp-to-the-host_chan_mmio-afu","title":"5.1. Adding RSTP to the host_chan_mmio AFU","text":"RSTP is added to an AFU by:
You can use these detailed steps to add Signal Tap to your AFU.
Set path to platform root directory and create the host_chan_mmio AFU Quartus project for adding Signal Tap.:
$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n\n# we will now build a new host_chahnel_mmio example based on Signal Tap\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_stp\nNavigate to host_chan_mmio AFU Quartus project and open the project using Quartus GUI.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top\n$ quartus ofs_top.qpf &\nOnce the project is loaded in Quartus, run Analysis & Synthesis Processing | Start | Start Analysis & Synthesis. When complete, review the project hierarchy as shown in the Project Navigator. This example will add Signal Tap probe points to the AFU region. Reviewing the code will give insight into the function of this block. You can bring up the code in the Project Navigator by expanding afu_top - port_gasket - pr_slot - afu_main - port_afu_instances - ofs_plat_afu, then select instance afu, right click, select Locacte Node - Locate in Design File as shown below.
Bring up Signal Tap to create the *.stp file. In the Quartus GUI, go to Tools - Signal Tap Logic Analyzer. In the New File from Template pop up, click Create to accept the default template. The Signal Tap Logic Analyzer window comes up.
Set up the clock for the Signal Tap logic instance by clicking ... button as shown below:
In the Hierarchy Level, navigate to top - afu_top - pg_afu.port_gasket - pr_slot - afu_main - port_afu_instances - ofs_plat_afu, then select instance afu.
Enter *clk* in the Named: box and click Search. This brings up matching terms. Click clk and >. Verify your Node Finder is as shown below and then click Ok:
Double click the Double-click to add nodes as shown below:
The Node Finder comes up. Once again navigate to top - afu_top - port_gasket - pr_slot - afu_main - port_afu_instances - ofs_plat_afu, then select instance afu and click Ok. Enter mmio64_reg* and click Search. Then click >> to add these signals to the STP instance as shown below:
Then click Insert and Close.
Save the newly created STP by clicking File - Save As and in the save as navigate to $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top and save the STP file as host_chan_mmio.stp as shown below:
Select Yes when asked to add host_chan_mmio.stp to current project. Close Signal Tap window.
Edit ofs_pr_afu.qsf to add host_chan_mmio.stp file and enable STP. Open $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top/ofs_pr_afu.qsf in an editor and add the lines shown below:
set_global_assignment -name VERILOG_MACRO \"INCLUDE_REMOTE_STP\"\nset_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE host_chan_mmio.stp\nset_global_assignment -name SIGNALTAP_FILE host_chan_mmio.stp\nThe host_chan_mmio AFU Quartus project is ready to be built. In your original build shell enter the following commands:
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\n\n...\n...\nWrote host_chan_mmio.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\nOnce compilation completes, the new host_chan_mmio.gbs file that contains the Signal Tap instance can be loaded.
# For the following example, the N6001 SKU2 PCIe b:d.f is assumed to be b1:00.0,\n# however this may be different in your system\n# Enusre FIM is loading prior to loading AFU\n# Load AFU\n$ sudo fpgasupdate host_chan_mmio.gbs b1:00.0\n[2021-12-04 07:16:59,101] [WARNING ] Update starting. Please do not interrupt.\n[2021-12-04 07:16:59,740] [INFO ] Partial Reconfiguration OK\nUse the OPAE SDK mmlink tool to create a TCP/IP connection to your Agilex\u00ae card under test. The mmlink command has the following format:
Usage:\nmmlink\n<Segment> --segment=<SEGMENT NUMBER>\n<Bus> --bus=<BUS NUMBER> OR -B <BUS NUMBER>\n<Device> --device=<DEVICE NUMBER> OR -D <DEVICE NUMBER>\n<Function> --function=<FUNCTION NUMBER> OR -F <FUNCTION NUMBER>\n<Socket-id> --socket-id=<SOCKET NUMBER> OR -S <SOCKET NUMBER>\n<TCP PORT> --port=<PORT> OR -P <PORT>\n<IP ADDRESS> --ip=<IP ADDRESS> OR -I <IP ADDRESS>\n<Version> -v,--version Print version and exit\nEnter the command below to create a connection using port 3333:
$ sudo mmlink -P 3333 -B 0xb1\n\n------- Command line Input START ----\n\nSocket-id : -1\n Port : 3333\nIP address : 0.0.0.0\n ------- Command line Input END ----\n\nPORT Resource found.\nServer socket is listening on port: 3333\nLeave this shell open with the mmlink connection.
In this step you will open a new shell and enable JTAG over protocol. You must have Quartus 24.1 Programmer loaded on the N6001 server for local debugging.
$ jtagconfig --add JTAG-over-protocol sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0\n\n# Verify connectivity with jtagconfig --debug\n$ jtagconfig --debug\n1) JTAG-over-protocol [sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0]\n(JTAG Server Version 21.4.0 Build 67 12/06/2021 SC Pro Edition)\n020D10DD VTAP10 (IR=10)\nDesign hash 86099113E08364C07CC4\n + Node 00406E00 Virtual JTAG #0\nCaptured DR after reset = (020D10DD) [32]\nCaptured IR after reset = (155) [10]\nCaptured Bypass after reset = (0) [1]\nCaptured Bypass chain = (0) [1]\nStart Quartus Signal Tap GUI, connect to target, load stp file by navigating to $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top. The Quartus Signal Tap must be the same version of Quartus used to compile the host_chan_mmio.gbs. Quartus Prime Pro Version 24.1 is used in the steps below:
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top\n$ quartus_stpw host_chan_mmio.stp &\nThis command brings up Signal Tap GUI. Connect to the Signal Tap over protocol by selecting the Hardware button on the right side of the GUI and click the \"Please Select\" pull down as shown below:
JTAG over protocol selected:
This connection process will take approximately 2-3 minutes for the Signal Tap instance to indicate \"Ready to acquire\".
Set the trigger condition for a rising edge on signal awvalid signal.
In the Signal Tap window, enable acquisition by pressing key F5, the Signal Tap GUI will indicate \"Acquisition in progress\". Create and bind the VFs, then run the host_chan_mmio application following 3.2. Loading and Running host_chan_mmio example AFU, and observe that the Signal Tap instance has triggered. You should see signals being captured in the Signaltap GUI.
See captured image below:
To end your Signal Tap session, close the Signal Tap GUI, then in the mmlink shell, enter ctrl c to kill the mmlink process. To remove the JTAG over protocol connection:
# This is assuming the JTAG over protocol is instance '1', as shown during jtagconfig --debug\n$ jtagconfig --remove 1\nThe vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\nEnable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\nIf you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\nSample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\nFor information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":"When using ASE in the application development cycle, consider the following limitations:
AFU development using the ASE includes the following four stages:
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\nThis directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\nBefore configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\nafu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.generate_ase_environment.py: This script instantiates your simulated platform configuration.The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
synopsys_sim.setup and vcs_run.tcl in the configuration directory.vsim_run.tcl in the configuration directory.The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\nThe AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code.cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\nASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\nCreate a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\nOptionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> .. /usr. $ sudo make install ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\nChange directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n... The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\nYou can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c. # Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\nNote: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\nUpon completion, the simulation generates the following files:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
ASE_LOG to 0.Two log levels are supported in ASE, controlled by env(ASE_LOG)
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\nThe following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If usingASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/images/readme/","title":"Readme","text":"images
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI XA Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\nThe OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support:<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.To interface the PIM with an AFU:
For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\nofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\nThe Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\nUsing the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\nThe following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\nLocal memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\nUsing the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\nThe following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\nThe High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\nIn digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\nInstantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\nThe AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\nThe AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\nThe Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\nThe MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\nThe AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.svThe HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\nThe host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\nThe ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\nThe Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\nThe source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\nBelow is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/images/readme/","title":"Readme","text":"images
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/","title":"oneAPI Accelerator Support Package (ASP): Getting Started User Guide","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#11-about-this-document","title":"1.1 About This Document","text":"This document serves as a quick start guide for setting up Intel\u00ae oneAPI Base Toolkit (Base Kit) on Open FPGA Stack (OFS) using oneapi-asp repository. Please see Table 1-1 for OFS reference platforms targeted in this guide.
Table 1-1 HLD Tools
Target Device for the oneAPI ASP Target Platform for the oneAPI ASP Agilex\u00ae 7 FPGA Intel\u00ae FPGA SmartNIC N6001-PL Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) Stratix\u00ae 10 FPGA Intel\u00ae FPGA PAC D5005Attention: Intel is discontinuing the Intel FPGA SDK for OpenCL software product. Refer to the Product Discontinuation Notice PDN2219. Alternatively, Intel recommends using the Intel\u00ae oneAPI Base Toolkit (Base Kit), which provides core tools and libraries for developing high-performance data-centric applications across diverse architectures. It features an industry-leading C++ compiler that implements SYCL*, an evolution of C++ for heterogeneous computing. For more information, refer to the Intel\u00ae oneAPI Base Toolkit (Base Kit) web page. To migrate your OpenCL FPGA designs to SYCL, review Migrating OpenCL\u2122 FPGA Designs to SYCL* that demonstrates important differences between OpenCL and SYCL for FPGA and provides steps to migrate your OpenCL designs.
After reviewing the document you will be able to:
This table defines some of the common terms used when discussing OFS.
Table 1-2: Terminology
Term Abbreviation Description Open FPGA Stack OFS A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. Accelerator Functional Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. FPGA Interface Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. High Level Design HLD For the purpose of this guide, this term refers to designing with Intel\u00ae oneAPI Base Toolkit (Base Kit). oneAPI Accelerator Support Package oneAPI ASP OR oneapi-asp A collection of hardware and software components that enable oneAPI kernels to communicate with oneAPI runtime as well as with OFS components. The hardware components of the oneAPI ASP along with the kernels lie in the AFU region. Open Programmable Acceleration Engine Software Development Kit OPAE SDK A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. Platform Interface Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Device Feature List DFL A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. Best Known Configuration BKC The exact hardware configuration Intel has optimized and validated the solution against. SYCL - SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL\u2122) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Installable Client Driver ICD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports the OpenCL ICD extension from the Khronos Group\u2122. The OpenCL ICD extension allows you to have multiple OpenCL implementations on your system. With the OpenCL ICD Loader Library, you may choose from a list of installed platforms and execute OpenCL API calls that are specific to your OpenCL implementation of choice. FPGA Client Driver FCD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports FPGA Client Driver(FCD) extension. FCD allows the runtime to automatically find and load the oneAPI ASP libraries at host run time"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#13-introduction-to-high-level-design-on-ofs","title":"1.3 Introduction to High Level Design on OFS","text":"Intel currently provides Intel\u00ae oneAPI Base Toolkit (Base Kit) for FPGA application development using high level languages like Data Parallel C++(DPC++).
Figure 1-1 shows how OFS components can be used with Intel HLD tool.
Figure 1-1 HLD Tool on OFS Platforms
For high level description and setup details for OFS components shown in figure above, please refer to the Getting Started guide for your target device.
For a more detailed diagram and more information about the FPGA Interface Manager(FIM) shown in figure above, please refer to the FIM developer guides for your target device.
The oneAPI ASP is required for compiling and running HLD application kernel on OFS platforms using Intel oneAPI. It is a collection of hardware and software components that enable oneAPI kernels to communicate with oneAPI runtime as well as with other OFS components. The hardware components of the oneAPI ASP along with the kernel lie in the AFU region shown in the figure above. For more details about the components of the oneAPI ASP, please refer to oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
Figure 1-2 shows the setup steps to use oneAPI base toolkit on OFS platforms.
Figure 1-2 Setup Steps for oneAPI base toolkit on OFS Platforms
The next section covers the setup steps in detail.
Note: Administrative privileges are needed for multiple setup steps, ensure you have administrative/sudo privileges before proceeding.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#20-setup-flow-for-using-hld-tool-on-ofs","title":"2.0 Setup Flow for Using HLD Tool on OFS","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#21-setup-server-for-ofs","title":"2.1 Setup Server for OFS","text":"As a first step, the server or host machine being used for developing HLD application needs to be setup for OFS. This involves setting up the FPGA card as well as installing OFS software stack including OPAE SDK and OFS DFL kernel driver.
Please follow steps in Software Installation Guide: OFS for PCIe Attach FPGAs to setup Linux DFL kernel driver and install OPAE SDK.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#22-clone-and-compile-fim","title":"2.2 Clone and Compile FIM","text":"As shown in Figure 1-1, OFS components in the FPGA include the FIM and Accelerator Functional Unit(AFU). The oneAPI ASP is in the Partial Reconfiguration(PR) region of the AFU and relies on the compiled database of the static region(FIM) to interface with the host and board peripherals(e.g. on-board memory).
Once the server is setup with OPAE SDK and DFL kernel driver, the next step is to clone and compile the static region of the design, i.e. FIM. You can use the default configuration of the FIM for all target platforms. Additionaly for Agilex\u00ae 7 FPGA you have the option to create 2 different types of FIM with different PCIe configurations, 1PF/1VF and 2PF (built using an ofss file provided, see table below for more information). Furthermore, for Intel\u00ae FPGA SmartNIC N6001-PL and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) platforms you can create a minimal FIM which removes the HSSI subsystem and host exercisers in the design and expand the AFU PR region with the area previously used by the removed components, follow the corresponding Shell Developer Guide for more information. The difference between this two PCIe configurations, are the amount of VFs that must be created, in case of 1PF/1VF , only 1 VF is needed. In case of 2PF FIM no VF creation is required. 2PF FIM could be used in FPGA development in virtual machines, see Section 3.0 OneAPI on OFS Running in a Virtual Machine. Please see Table 2-1 below for more information.
Table 2-1 Agilex\u00ae 7 FPGA FIM Configurations
Target Platform FIM 1PF/1VF configuration FIM 2PF configuration Intel\u00ae FPGA SmartNIC N6001-PL pcie_host_1pf_1vf.ofss pcie_host_2pf.ofss Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) pcie_host_1pf_1vf.ofss pcie_host_2pf.ofss Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) pcie_host_2link_1pf_1vf.ofss no ofss provided Note: For steps to build the 2PF FIM refer to Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAsPlease follow steps in the Shell Developer Guides for your target device to compile FIM supporting PR release.
A pr_build_template directory will be generated in the work directory specified as part of the FIM compile command (using OFS/ofs-common/scripts/common/syn/build_top.sh script with the '-p' option enable to create an out-of-tree PR release). The pr_build_template directory is required for successful setup of the oneAPI ASP.
Once the FIM compile is complete, please program FIM using fpgasupdate and Remote System Update(rsu) command for Intel\u00ae FPGA SmartNIC N6001-PL and Intel\u00ae FPGA PAC D5005. Use of these commands has been demonstrated in 4.6.3 Updating with fpgasupdate and 4.6.5 Loading Images with rsu sections in Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL), refer to sections 5.2.1 fpgasupdate and 5.2.3 rsu in Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs for Stratix\u00ae 10 FPGA. In case of Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) platforms, program the FIM with a SOF image via JTAG, refer to Chapter 5 in Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs and Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs guides for more information.
In addition to server setup and FIM compilation, a few linux packages are needed to setup the oneAPI ASP and develop HLD applications.
1) Install the following packages:
sudo dnf install numactl-devel ncurses-compat-libs\n2) Ensure that IOMMU is turned on as explained in section 3.2 Building and Installing the OFS DFL Kernel Drivers from Source in Software Installation Guide: OFS for PCIe Attach FPGAs.
You can verify this setting using cat /proc/cmdline command. The output must have intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200.
$ cat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-6.1.78-dfl root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\n3) Install HLD development software. Please see Table 2-2 below for download link. Install the latest version. Use sudo privileges to do the installation.
Table 2-2 Intel HLD Tool and Download Information
HLD Tool Target Platform for the oneAPI ASP Tool Download Information Intel\u00ae oneAPI Base Toolkit (Base Kit)Note: Install the oneAPI compiler patches required, according to the version you are using, you can download them in the patch history section from the FPGA Support Package for Intel\u00ae oneAPI DPC++/C++ Compiler page.
Tool installation guide for your reference:
4) Ensure you have all the Quartus patches installed, refer to Table 2-4 and 2-5 for required Quartus version.
Note: For Agilex\u00ae 7 FPGA ensure Quartus patches 0.18, 0.26 and 0.02iofs are installed. You can find patches 0.18 and 0.26 in a tar file under assets and Quartus Prime Pro License File 0.02iofs under the Summary section in the following link patch-agx7-2024-2-1.tar.gz. For Stratix\u00ae 10 FPGA ensure Quartus patch 0.01iofs is installed. You can find it in the following link 0.01iofs. For quartus patches installation to work properly, you must have Git Large File Storage (LFS) installed when cloning the ofs-fim repository.
Use following command to check Quartus version and installed patches.
quartus_sh -v\n5) After completing the tool installation, set the following environment variables required to execute build scripts successfully:
# Adding Quartus to PATH\nexport PATH=$PATH:path-to-quartus-installation-dir/bin\n export QUARTUS_ROOTDIR=path-to-quartus-installation-dir/quartus\n export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n# Other OFS environment variables\nexport OFS_ROOTDIR=path-to-directory-containing-cloned-ofs-fim-repo/#ofs-agx7-pcie-attach for Agilex\u00ae 7 FPGA or ofs-d5005 for Stratix\u00ae 10 FPGA\n export WORKDIR=$OFS_ROOTDIR\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n export OPAE_SDK_REPO_BRANCH=release/branch-tag # Refer to Table 2-4 and 2-5, for OPAE SDK branch tag\nexport OFS_PLATFORM_AFU_BBB=path-to-cloned-ofs-platform-afu-bbb-repo\n export OPAE_PLATFORM_ROOT=path-to-ofs-fim-pr_build_template-directory # (see section 2.2 for more details)\nexport OFS_ASP_ROOT=path-to-directory-containing-oneapi-asp/oneapi-asp/platform-name #platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005. \nexport LIBOPAE_C_ROOT=/usr # (OPAE libraries are installed in /usr/lib64 by default if you followed the OPAE SDK steps covered in section 2.1 as is and installed OPAE rpm packages. If you have a different OPAE installation path, please point LIBOPAE_C_ROOT to your OPAE installation location that you specified using -DCMAKE_INSTALL_PREFIX=installation-path in cmake command for building OPAE)\nNote: To re-use this environment setting, you can copy the above export statements to a shell script, update the paths to match your tool installations and source this script each time a new shell is started.
6) Source initialization script for oneAPI, path is shown in table below.
Table 2-3 Initialization Script for HLD tool
Tool Command to source initialization script Intel\u00ae oneAPI Base Toolkit (Base Kit) source path-to-intel-oneapi-toolkit-installation-directory/setvars.shOnce the environment variables are set, you can check the tool version using the following commands:
# Printing all (Quartus, Intel\u00ae oneAPI Base Toolkit, GCC) versions for user info\nquartus_sh -v\nicpx --version (for oneAPI Base Toolkit)\ngcc --version\nTable 2-4 and 2-5 summarize the tool version/Best Known Configurations(BKC).
Table 2-4 Best Known Configuration(BKC) for Agilex\u00ae 7 FPGA OFS
Component/Tool Version FPGA Platform Intel\u00ae FPGA SmartNIC N6001-PL Operating System RHEL 8.8 , Kernel : 4.18.0-dfl linux-dfl (DFL Kernel Driver) Tag: intel-1.11.0-2 opae-sdk Branch: release/2.13.0, Tag: 2.13.0-3 ofs-fim Tag: ofs-2024.2-1 oneapi-asp Tag: ofs-2024.2-2 > Note: Cloning and build of this repo is discussed in the section 2.4 Quartus Prime Pro Edition Version 24.1 Pro Edition with patches (0.18, 0.26 and 0.02iofs) under assets and summary sections in this link patch-agx7-2024-2-1.tar.gz Intel\u00ae oneAPI Base Toolkit (Base Kit) Latest version > Note: Ensure to install the oneAPI compiler patches required, according to the version you are using, you can download them in the patch history section from the FPGA Support Package for Intel\u00ae oneAPI DPC++/C++ Compiler page GCC 8.5.0 cmake 3.15Table 2-5 Best Known Configuration(BKC) for Stratix\u00ae 10 FPGA
Component/Tool Version FPGA Platform Intel\u00ae FPGA PAC D5005 Operating System RHEL 8.6 , Kernel : 6.1.78-dfl linux-dfl (DFL Kernel Driver) Tag: ofs-2024.1-6.1-2 opae-sdk Branch: release/2.12.0, Tag: 2.12.0-5 ofs-fim Tag: ofs-2024.1-1 oneapi-asp Tag: ofs-2024.2-2 > Note: Cloning and build of this repo is discussed in the section 2.4 Quartus Prime Pro Edition Version 23.4 Pro Edition with patch 0.01iofs Intel\u00ae oneAPI Base Toolkit (Base Kit) Latest version > Note: Ensure to install the oneAPI compiler patches required, according to the version you are using, you can download them in the patch history section from the FPGA Support Package for Intel\u00ae oneAPI DPC++/C++ Compiler page GCC 8.5.0 cmake 3.15"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#24-build-and-install-oneapi-asp","title":"2.4 Build and Installoneapi-asp","text":"Once all pre-requisites are installed and the environment variables are set, next step is to clone and build oneapi-asp.From ofs-2024.2-2 tag you have the option to use an editor tool for an easy parametrization of the oneAPI Accelerator Support Pacakges in OFS based platforms using the IP Parameter Editor GUI in Quartus Prime, you can generate the ASP and select only the board variants you want to include using the preset files provided, see appendix A for more information about the oneAPI ASP editor. This section covers the build of the oneAPI ASP with the default values, including all the board variants for the corresponding OFS platform.
1) Clone oneapi-asp repository and checkout tag matching the BKC for your target platform (see Tables 2-4 and 2-5 for the BKCs).
Note: You will need a personal access token (use the classic mode) to be used as the password to clone successfully. Please see more information about token authentication requirements for Git operations here.
git clone https://github.com/OFS/oneapi-asp.git\n cd oneapi-asp\n git checkout tags/ofs-2024.2-2\nEnsure the correct tag has ben checked out:
git describe --tags\n ofs-2024.2-2\n2) Ensure that OPAE_PLATFORM_ROOT and LIBOPAE_C_ROOT have been set as described in section 2.3. Generate the oneAPI ASP hardware and software using provided build-asp.sh script. This script clones required repositories and builds the oneAPI ASP libraries required by HLD host application to run successfully.
cd path-to-directory-containing-cloned-oneapi-asp-repo/oneapi-asp/platform-name\n ./scripts/build-asp.sh\nThe generated directory structure is shown below. For more details refer to the oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
\noneapi-asp/platform-name\n|--ase/\n| |--base/\n| |--hack_ip_files/\n| |--compile-kernel.sh\n| |--run-ase.sh\n| |--setup.sh\n| |--simulate-aocx.sh\n|--bringup\n| |--binaries/\n| |--source/simple-add-buffers/\n| | |--simple-add-buffers.cpp\n|--build/\n| |--bringup/\n| |--release/\n|--hardware/\n| |--common/build/\n| |--ofs_platform-name/\n| |--ofs_platform-name_iopipes/\n| |--ofs_platform-name_usm/\n| |--ofs_platform-name_usm_iopipes/\n|--linux64/\n| |--doc/\n| |--include/\n| |--lib/\n| | |--libintel_opae_mmd.so\n| | |--libMPF-cxx.so\n| | |--libMPF.so\n| |--libexec/\n| | |--diagnose\n| | |--flash\n| | |--initialize\n| | |--install\n| | |--program\n| | |--setup_permissions.sh\n| | |--uninstall\n|--scripts/\n| |--build-asp.sh\n| |--build-asp-sw.sh\n| |--build-default-binaries.sh\n| |--build-mmd.sh\n| |--build-opae.sh\n| |--create-tarball.sh \n| |--dedup-hardware.sh\n| |--README.txt\n| |--setup-asp.py\n|--board_env.xml\n|--README.md\n
3) Once the oneAPI ASP is generated, add the following to LD_LIBRARY_PATH. You can add it to your script for setting environment variables (if you created one as noted in step 5 in section 2.3)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:path-to-oneapi-asp/platform-name/linux64/lib\n4) Check if FPGA Client Driver(FCD) exists for any other version of oneAPI ASP or for any other board. You can check with aocl list-devices command. It is recommended to run aocl list-devices as root user (login with sudo su) to see all installed ASPs on the host.
A sample output when a oneAPI ASP FCD is installed is shown below:
--------------------------------------------------------------------\nDevice Name:\nacl0\n\nBSP Install Location:\n/home/ofsuser/oneapi-asp/platform-name\n\nVendor: Intel Corp\n\nPhysical Dev Name Status Information\n\nofs_ee00000 Uninitialized PR slot function not configured \n Need to follow instructions to bind vfio-pci driver to PR slot function\n\nASP DIAGNOSTIC_PASSED\n--------------------------------------------------------------------\nIf a oneAPI ASP/BSP is installed, uninstall using aocl uninstall path-to-oneapi-asp-install-location, where path-to-oneapi-asp-install-location is provided under BSP Install Location: in the output of aocl list-devices. If you are prompted with a question to unset the FCD, type Y. If you are prompted with a question to remove oneAPI-ASP configuration settings, type Y.
Sample output for aocl uninstall command:
$ aocl uninstall /home/ofsuser/oneapi-asp/platform-name\naocl uninstall: Removing the FPGA Client Driver (FCD) from the system\n[sudo] password for ofsuser:\naocl uninstall: Removing the board package /home/ofsuser/oneapi-asp/platform-name from the list of installed packages. This process may require admin privilege\naocl uninstall: Running uninstall from /home/ofsuser/oneapi-asp/platform-name/linux64/libexec\nDo you want to remove oneAPI-ASP configuration settings [Y/n] Y\nDeleting OPAE config files\nRemoving configuration files\nOFS oneAPI-ASP uninstall complete\n5) Install FPGA Client Driver(FCD) file for the oneAPI ASP using aocl install path-to-oneapi-asp/platform-name command as shown below. The host program uses FCD to find and link to the platform Memory Mapped Device (MMD) library. For more information about MMD library, refer to oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
aocl install path-to-directory-containing-oneapi-asp/oneapi-asp/platform-name\nNotes: 1. Type Y when prompted to setup FCD at /opt/Intel/OpenCLFPGA/oneAPI/Boards (default location for Intel oneAPI).
Sample output aocl install command in Intel\u00ae oneAPI Base Toolkit (Base Kit) environment is shown below.
aocl install: Setting up the FPGA Client Driver (FCD) to the system. This process may require admin privilege\nInstall the FCD file to /opt/Intel/OpenCLFPGA/oneAPI/Boards\n[sudo] password for ofsuser:\naocl install: Adding the board package path-to-oneapi-asp/platform-name to the list of installed packages\nInstalling the board package driver to the system.\naocl install: Running install from path-to-oneapi-asp/platform-name/linux64/libexec\nConfiguring locked memory setting\nConfiguring udev rules for DFL FPGA device permission\nConfiguring system with 1024 2M hugepages\nSetting access permisions of /dev/uio to 666\nFinished setup_permissions.sh script. All configuration settings are persistent.\nIntel OFS oneAPI ASP install complete.\nRun 'aocl diagnose' to list devices or 'aocl initialize <dev_name> <board_variant> to load default image\nOFS software stack expects boards to be initialized with a bitstream for the board variant intended to be used for development. An oneAPI sample application, named simple-add-buffers, has been provided in the oneapi-asp repository for generating initialization bitstreams for included board variants. The sample is located in oneapi-asp/platform-name/bringup/source.
oneapi-asp has four board variants for Intel\u00ae FPGA SmartNIC N6001-PL and two board variants for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and Intel\u00ae FPGA PAC D5005 (oneapi-asp/platform-name/hardware has the hardware design files for these). For more details on the architecture of the board variants, please refer to the oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
Table 2-6 oneAPI Sample Applications
Board Variants Sample ApplicationNote: platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005.
All samples are located in oneapi-asp/platform-name/bringup/source.
A script is provided in repo to compile simple-add-buffers oneAPI sample application. The script is oneapi-asp/platform-name/scripts/build-default-binaries.sh.
Script usage is as follows:
./build-default-binaries.sh -b name-of-board-variant\nNote: name-of-board-variant can be ofs_platform-name, ofs_platform-name_usm, ofs_n6001_iopipes or ofs_n6001_usm_iopipes where platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005. Refer to Table 2-6 for board variants available for each target platform. Compilation will take a few hours to complete.
The output directory of the sample application is written to oneapi-asp/platform-name/build/bringup. The generated bitstreams are also copied to oneapi-asp/platform-name/bringup/binaries/. These are used in the initialization of the platform.
Once the bitstreams are generated, create a VF and initialize the board as explained in following section. Ensure that the FIM has been programmed on the board as explained in section 2.2 Clone and Compile FIM
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#252-pfvf-mapping","title":"2.5.2 PF/VF mapping","text":"The oneAPI ASP is located in the PR region of the FIM and is accessed through PF/VF Mux. Refer to the FIM Reference Manual for your target platforms for more details about PF/VF mapping.
For Base_x16 FIM (default) and 1PF/1VF FIM (refer to table 2-1 for the ofss file needed to build this FIM configuration), VF0 is mapped to PR region and you can create 1 VF when using this FIM. Base_x16 FIM has 5 PF's.
For 2PF FIM (built using pcie_host_2pf.ofss) PF1 is mapped to PR region and no VF creation is required.
See Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for diagram showing PF/VF mapping.
Note: See Table 2-1 for available FIM configurations for Intel\u00ae FPGA SmartNIC N6001-PL, Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) platforms.
VF1 is mapped to PR region and you must create 2 VFs when using this FIM. This FIM has 1 PF. See Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs for diagram showing PF/VF mapping.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#2521-create-vf","title":"2.5.2.1 Create VF","text":"Note: This section only applies for Base_x16 FIM and 1PF/1VF FIM for Agilex\u00ae 7 FPGA and default FIM for Stratix\u00ae 10 FPGA.
fpgainfo fme (PCIe s\\:b\\:d.f output) sudo pci_device s:b:d.f vf num_vf #num_vf is 1 for Agilex\u00ae 7 FPGA and 2 for Stratix\u00ae 10 FPGA\nsudo opae.io ls command and note the PCIe ID for the VF(s) (the function number in s\\:b\\:d.f will be different for the VF). Sample output for Agilex\u00ae 7 FPGA 1PF/1VF FIM is shown below. Output for base_x16 FIM should display 5 PF's and the PCIe ID for VF0 will be s:b:d.5. $ sudo opae.io ls\n [0000:b1:00.0] (0x8086, 0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086, 0xbccf) Intel Acceleration Development Platform N6001 (Driver: None)\nSample output for Stratix\u00ae 10 FPGA is shown below.
$ sudo opae.io ls\n [0000:d8:00.0] (0x8086, 0xbcce) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\n [0000:d8:00.1] (0x8086, 0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\n [0000:d8:00.2] (0x8086, 0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\nNote:sudo opae.io ls will list the accelerators, respective PCIe ID as well as the driver it is currently bound to.
Note: For more information about pci_device and opae.io utilities, refer to the OPAE FPGA tools page here.
For Base_x16 FIM and 1PF/1VF FIM for Agilex\u00ae 7 FPGA and default FIM for Stratix\u00ae 10 FPGA :
s:b:d.vf will be 0000:b1:00.1 in command below. For base_x16 FIM should be s:b:d.5. sudo opae.io init -d s:b:d.vf $USER\n $ sudo opae.io init -d 0000:b1:00.1 $USER\n Unbinding (0x8086,0xbccf) at 0000:b1:00.1 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:b1:00.1 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:b1:00.1 is 319\n Assigning /dev/vfio/319 to ofsuser\n Changing permissions for /dev/vfio/319 to rw-rw----\n\n $ sudo opae.io ls\n [0000:b1:00.0] (0x8086, 0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086, 0xbccf) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n\n $ ls -lt /dev/vfio\n total 0\n crw-rw----. 1 ofsuser root 509, 0 Dec 3 20:41 319\n crw-rw-rw-. 1 root root 10, 196 Dec 3 20:41 vfio\n $sudo opae.io init -d 0000:12:00.1 $USER\n Unbinding (0x8086,0xbccf) at 0000:12:00.1 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:12:00.1 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:12:00.1 is 149\n Assigning /dev/vfio/149 to ofsuser\n Changing permissions for /dev/vfio/149 to rw-rw----\n\n $sudo opae.io init -d 0000:12:00.2 $USER\n Unbinding (0x8086,0xbccf) at 0000:12:00.2 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:12:00.2 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:12:00.2 is 152\n Assigning /dev/vfio/152 to ofsuser\n Changing permissions for /dev/vfio/152 to rw-rw----\n\n $ sudo opae.io ls\n [0000:12:00.0] (0x8086:0xbcce) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\n [0000:12:00.1] (0x8086:0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: vfio-pci)\n [0000:12:00.2] (0x8086:0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: vfio-pci)\n\n $ls -lt /dev/vfio\n total 0\n crw-rw----. 1 ofsuser root 235, 3 Dec 3 16:25 149\n crw-rw----. 1 ofsuser root 235, 0 Dec 3 16:22 152\n crw-rw-rw-. 1 root root 10, 196 Dec 1 07:28 vfio\nFor 2PF FIM for Agilex\u00ae 7 FPGA :
Bind the PF1 to vfio-pci driver, use sudo opae.io ls command and note the PCIe ID (s\\:b\\:d.f) for the PF(s). Verify you are using the PCIe ID of the PF1.
$ sudo opae.io ls\n [0000:b1:00.0] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\nOutput below shows the command to bind PF1, in this case s:b:d.f will be 0000:b1:00.1.
sudo opae.io init -d s:b:d.f $USER\n $ sudo opae.io init -d 0000:b1:00.1 $USER\n Unbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\n Binding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\n iommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 13\n Assigning /dev/vfio/13 to ofsuser\n Changing permissions for /dev/vfio/13 to rw-rw----\n\n $ sudo opae.io ls\n [0000:b1:00.0] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n\n $ ls -lt /dev/vfio\n total 0\n crw-rw----. 1 ofsuser root 511, 0 Feb 2 22:47 13\n crw-rw-rw-. 1 root root 10, 196 Feb 2 16:56 vfio\nIf the driver fails to bind due to an error related to iommu_group (e.g. `No such file or directory: '/sys/bus/pci/devices/0000:b1:00.5/iommu_group'), ensure IOMMU is turned on as explained in step 2 in Section 2.3 Prerequisites.
Note: For more information about opae.io utilities, refer to the OPAE FPGA tools page here.
Before initializing the board set the value of 2048 to OFS_ASP_ENV_NUM_HUGEPAGES env variable to configure 2048 2M hugepages:
export OFS_ASP_ENV_NUM_HUGEPAGES=2048\naocl initialize command aocl initialize device-name name-of-board-variant\nNote: device-name in aocl initialize command is from the ASP Diagnostics section in the output of aocl list-devices (sample output is shown in Section 2.4). device-name is acl0 if there is only 1 board connected to the server. name-of-the-board-variant can be one of the supported board variants listed in Table 2-6 provided the sample application bitstream using steps in section above.
Sample output for aocl initialize acl0 ofs_n6001 is shown below. Output for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and Intel\u00ae FPGA PAC D5005 platforms and other board variants should be similar.
$ aocl initialize acl0 ofs_n6001\naocl initialize: Running initialize from path-to-oneapi-asp/n6001/linux64/libexec\nInitializing with default ASP binary ofs_n6001.fpga\nSaving target image to \"ofs_n6001.aocx.0\"\nConfiguring locked memory setting\n[sudo] password for $USER:\nConfiguring udev rules for DFL FPGA device permission\nConfiguring system with 1024 2M hugepages\nSetting access permisions of /dev/uio to 666\nFinished setup_permissions.sh script. All configuration settings are persistent.\nProgram succeed.\nNotes: 1. aocl initialize command needs to be executed only one time unless you reboot the server.
Run aocl diagnose to check that the board Status under BSP Diagnostics is equal to Passed.
platform-name in command output below is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005.
$ aocl diagnose\n--------------------------------------------------------------------\nICD System Diagnostics\n--------------------------------------------------------------------\n\nUsing the following location for ICD installation:\n /etc/OpenCL/vendors\n\nFound 3 icd entry at that location:\n /etc/OpenCL/vendors/intel64.icd\n /etc/OpenCL/vendors/Altera.icd\n /etc/OpenCL/vendors/Intel_FPGA_SSG_Emulator.icd\n\n\nThe following OpenCL libraries are referenced in the icd files:\n /opt/intel/oneapi/compiler/latest/lib/libintelocl.so\n\nChecking LD_LIBRARY_PATH for registered libraries:\n /opt/intel/oneapi/compiler/latest/lib/libintelocl.so was registered on the system.\n /opt/intel/oneapi/compiler/latest/opt/oclfpga/host/linux64/lib/libalteracl.so\n\n\nChecking LD_LIBRARY_PATH for registered libraries:\n /opt/intel/oneapi/compiler/latest/linux/lib/oclfpga/host/linux64/lib/libalteracl.so was registered on the system.\n /opt/intel/oneapi/compiler/latest/linux/lib/x64/libintelocl_emu.so\n\nChecking LD_LIBRARY_PATH for registered libraries:\n /opt/intel/oneapi/compiler/latest/linux/lib/x64/libintelocl_emu.so was registered on the system.\n\nUsing OCL_ICD_FILENAMES to search for ICD clients, it is set to libintelocl_emu.so:libalteracl.so:/opt/intel/oneapi/compiler/2024.0/lib/libintelocl.so\n\nChecking LD_LIBRARY_PATH for registered libraries specified by OCL_ICD_FILENAMES\n libintelocl_emu.so was registered on the system at \n /opt/intel/oneapi/compiler/2024.0/lib\n libalteracl.so was registered on the system at \n /opt/intel/oneapi/compiler/2024.0/opt/oclfpga/host/linux64/lib\n /opt/intel/oneapi/compiler/2024.0/lib/libintelocl.so was registered on the system.\n\nUsing the following location for fcd installations:\n /opt/Intel/OpenCLFPGA/oneAPI/Boards\n\nFound 1 fcd entry at that location:\n /opt/Intel/OpenCLFPGA/oneAPI/Boards/platform-name.fcd\n\nThe following OpenCL libraries are referenced in the fcd files:\n libopae-c.so\n/home/ofsuser/oneapi-asp/platform-name/linux64/lib/libMPF.so\n/home/ofsuser/oneapi-asp/platform-name/linux64/lib/libintel_opae_mmd.so\n\nChecking LD_LIBRARY_PATH for registered libraries:\n libopae-c.so was registered on the system at /usr/lib64\n /home/ofsuser/oneapi-asp/platform-name/linux64/lib/libMPF.so was registered on the system.\n /home/ofsuser/oneapi-asp/platform-name/linux64/lib/libintel_opae_mmd.so was registered on the system.\n\nNumber of Platforms = 4\n 1. Intel(R) FPGA Emulation Platform for OpenCL(TM) | Intel(R) Corporation | OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3\n 2. Intel(R) FPGA SDK for OpenCL(TM) | Intel(R) Corporation | OpenCL 1.0 Intel(R) FPGA SDK for OpenCL(TM), Version 2024.0\n 3. Intel(R) OpenCL | Intel(R) Corporation | OpenCL 3.0 LINUX\n 4. Intel(R) FPGA SDK for OpenCL(TM) | Intel(R) Corporation | OpenCL 1.0 Intel(R) FPGA SDK for OpenCL(TM), Version 2024.0\n--------------------------------------------------------------------\nICD diagnostics PASSED\n--------------------------------------------------------------------\n--------------------------------------------------------------------\nBSP Diagnostics\n--------------------------------------------------------------------\n--------------------------------------------------------------------\nDevice Name:\nacl0\n\nBSP Install Location:\n/home/ofsuser/oneapi-asp/platform-name\n\nVendor: Intel Corp\n\nPhysical Dev Name Status Information\n\nofs_ee00001 Passed Intel OFS Platform (ofs_ee00001)\n PCIe b1:00.0\n FPGA temperature = 59 degrees C.\n\nASP DIAGNOSTIC_PASSED\n--------------------------------------------------------------------\n\nCall \"aocl diagnose <device-names>\" to run diagnose for specified devices\nCall \"aocl diagnose all\" to run diagnose for all devices\nRun complete diagnostic using aocl diagnose device-name command. device-name is acl0 if you have only 1 board in the server. The test reads and write data to the board to check the interfaces function correctly and report the measured bandwidth. The test must show ASP DIAGNOSTIC_PASSED message at the end.
aocl diagnose acl0\nNext section you will build and run oneAPI host applications.
Once you are done with your application testing, you can release the device from vfio-pci driver, the steps for this are provided in Section 2.7 Release VF.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#26-compile-and-run-oneapi-sample-applications","title":"2.6 Compile and Run oneAPI Sample Applications","text":"Different code samples are provided for testing different board interfaces. Please refer to table below for more information.
Table 2-7 DPC++ Sample Application
Board Variants Sample Application Description Link to Samples Repository Location of the Sample Samples Repository TagFirst clone the repository and then checkout to the corresponding tag.
git clone link-to-samples-repository\n cd oneAPI-samples # (oneapi-samples for examples-afu repository)\n git checkout tags/samples-repository-tag\nNote: For link-to-samples-repository and samples-repository-tag refer to the table 2-7. To check your oneAPI compiler version use the command:
\n icpx --version\n
Follow steps below to compile and run oneAPI board_test and io_streaming_one_pipe binaries. Use -DFPGA_DEVICE in cmake command to provide path to the oneAPI ASP and name of board variant being compiled. For board_test when targeting USM board variant add -DSUPPORTS_USM=1 flag in cmake command, for more information about this flag see the README file in the location of the sample, for this location refer to the table 2-7.
Ensure you have sourced setvars.sh script located in the root of your oneAPI installation as explained in Table 2-3 Initialization Script for HLD tool.
cd path-to-sample-location\n mkdir build\n cd build\nCmake for compiling board_test when targeting USM board variant.
cmake -DFPGA_DEVICE=full-path-to-oneapi-asp/platform-name:board_variant -DSUPPORTS_USM=1 ..\nCmake for compiling the rest of the board variants.
cmake -DFPGA_DEVICE=full-path-to-oneapi-asp/platform-name:board_variant ..\nCompile the design using the generated Makefile. The following build targets are provided:
make report\n make fpga\nHardware compilation takes several hours to complete. Once complete, you should see sample-name.fpga executable generated, where sample-name could be board_test or io_streaming_one_pipe.
For more information on additional environment settings required for running io_streaming_one_pipe sample see the README file in the location of the sample, for this location refer to the table 2-7 .
Run the generated hardware executable as follows:
./sample-name.fpga\nNote: If your FPGA compile fails to meet timing requirements, the Intel oneAPI compiler prints an error message, returns an error code and deletes the generated binary. In case of timing failure, *.failing_clocks.rpt and *.failing_paths.rpt files are generated in compiled output directory sample-name.fpga.prj, where sample-name could be board_test or io_streaming_one_pipe. You can recompile with a different seed using -Xsseed option. You can pass this option using USER_HARDWARE_FLAGS=-Xsseed=seed_value in the cmake command above and recompile hardware image.
To view test details and usage information using the binary, use the -help option.
./sample-name.fpga -help # sample-name could be board_test or io_streaming_one_pipe.\nOnce you are done with your application testing, you can release the device from vfio-pci driver using following command.
$ sudo opae.io release -d s:b:d.vf\nSample output for Agilex\u00ae 7 FPGA OFS target platform having programmed 1PF/1VF FIM is shown below. The output for 2PF FIM, base_x16 FIM for Agilex\u00ae 7 FPGA and base_x16 FIM for Stratix\u00ae 10 FPGA should be similar.
Note: For Stratix\u00ae 10 FPGA you will need to release an extra VF as for this target 2 Vfs were created.
$ sudo opae.io release -d 0000:b1:00.1\nReleasing (0x8086,0xbccf) at 0000:b1:00.1 from vfio-pci\nRebinding (0x8086,0xbccf) at 0000:b1:00.1 to dfl-pci\n\n$ sudo opae.io ls\n[0000:b1:00.0] (0x8086, 0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n[0000:b1:00.1] (0x8086, 0xbccf) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\nVirtual machines (VM's) can be used for FPGA development, 2PF FIM (built using pcie_host_2pf.ofss) is provided to use oneAPI on OFS Agilex\u00ae 7 FPGA device in a VM. For Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) refer to Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs to get the steps to build the 2PF FIM. For more information about 2PF FIM configuration refer to the FIM developer guides for your target device. - Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs. - Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs. - Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs.
The setup flow for your virtual machine could be found in the KVM User Guide: Open FPGA Stack, there are additional things for oneAPI flow listed below which you should ensure while configuring your VM:
5.1 Passing Devices to the VM when adding the PCIe Host Devices to the VM, ensure to have PF0 and PF1 BDF adjacent (s\\:b\\:d.f, s\\:b\\:d.f+1). The following example shows the address element of the PCIe Host Device XML file of PF0 and PF1, keeping the same value for domain, bus and slot attributes and only changing the function attribute (increasing its value by one):
$sudo yum install libnsl.so.1\nOnce this setup is done, follow Section 2.0 Setup Flow for Using HLD Tool on OFS to finish the configuration of the VM for oneAPI .
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#appendix","title":"Appendix","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#appendix-a-oneapi-accelerator-support-packageasp-editor","title":"Appendix A: oneAPI Accelerator Support Package(ASP) Editor","text":"The oneapi-asp tag ofs-2024.2-2 offers a new option for the users to generate and customize the ASP in OFS based platforms using an editor tool. The aim of this tool is to enable easy parametrization of the oneAPI Accelerator Support Packages in OFS based platforms using the IP Parameter Editor GUI in Quartus Prime. This section covers all the steps to use the editor tool to generate ASP for your board variant using the OFS reference platform presets files.
Tables A-1 to A-4 show the reference configurations of the editor tool for all OFS reference platforms. For more information about the functionality, parametization of the oneAPI ASP editor tool refer to the Appendix B of the oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
Table A-1 Reference configurations of the oneAPI ASP editor for Intel\u00ae FPGA SmartNIC N6001-PL
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 2 PF Minimal FIM (pcie_host_2pf.ofss) ofs_n6001 ofs_n6001.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 2 PF Minimal FIM (pcie_host_2pf.ofss) ofs_n6001_usm ofs_n6001_usm.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_n6001_iopipes ofs_n6001_iopipes.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 8 Data Width 64 Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_n6001_usm_iopipes ofs_n6001_usm_iopipes.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 8 Data Width 64Table A-2 Reference configurations of the oneAPI ASP editor for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_fseries-dk ofs_fseries-dk.qprs Number of Banks in each global memory 2 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_fseries-dk_usm ofs_fseries-dk_usm.qprs Number of Banks in each global memory 2 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/ATable A-3 Reference configurations of the oneAPI ASP editor for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 1PF/1VF minimal FIM (pcie_host_2link_1pf_1vf.ofss) ofs_iseries-dk ofs_iseries-dk.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 1PF/1VF minimal FIM (pcie_host_2link_1pf_1vf.ofss) ofs_iseries-dk_usm ofs_iseries-dk_usm.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/ATable A-4 Reference configurations of the oneAPI ASP editor for Intel\u00ae FPGA PAC D5005
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 default FIM ofs_d5005 ofs_d5005.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 default FIM ofs_d5005_usm ofs_d5005_usm.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#generate-the-oneapi-accelerator-support-package-asp-using-the-presets-files-with-the-editor","title":"Generate the oneAPI Accelerator Support Package (ASP) Using the Presets Files with the Editor","text":"Sections 2.1 to 2.3 (until step 4, the environment variables are quite different) must be covered before generating the ASP .
1) Clone oneapi-asp repository and checkout tag matching the BKC for your target platform (see Tables 2-4 and 2-5 for the BKCs).
Note: 1) You will need a personal access token (use the classic mode) to be used as the password to clone successfully. Please see more information about token authentication requirements for Git operations here. 2) Editor tool is an option enabled from ofs-2024.2-2 tag onwards.
git clone https://github.com/OFS/oneapi-asp.git\n cd oneapi-asp\n git checkout tags/ofs-2024.2-2\nEnsure the correct tag has ben checked out:
git describe --tags\n ofs-2024.2-2\n2) Set the following environment variables required to execute build scripts successfully:
# Adding Quartus to PATH\nexport PATH=$PATH:path-to-quartus-installation-dir/quartus/bin\n export PATH=$PATH:path-to-quartus-installation-dir/qsys/bin\n export QUARTUS_ROOTDIR=path-to-quartus-installation-dir/quartus\n export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n # Other OFS environment variables\nexport OPAE_PLATFORM_ROOT=path-to-ofs-fim-pr_build_template-directory # (see section 2.2 for more details)\nexport LIBOPAE_C_ROOT=/usr # (OPAE libraries are installed in /usr/lib64 by default if you followed the OPAE SDK steps covered in section 2.1 as is and installed OPAE rpm packages. If you have a different OPAE installation path, please point LIBOPAE_C_ROOT to your OPAE installation location that you specified using -DCMAKE_INSTALL_PREFIX=installation-path in cmake command for building OPAE)\nNote: To re-use this environment setting, you can copy the above export statements to a shell script, update the paths to match your tool installations and source this script each time a new shell is started.
3) Source initialization script for oneAPI as is demonstrated in section 2.3 Prerequisites.
source path-to-intel-oneapi-toolkit-installation-directory/setvars.sh\n4) Go to the oneapi_asp_editor folder
cd path-to-directory-containing-oneapi-asp/oneapi-asp/oneapi_asp_editor\n5) Run the Platform Designer editor from command-line and open the oneapi_asp_editor.ip file selecting the oneapi_asp_editor as project file. Figure A-1 shows the oneAPI ASP Editor GUI.
qsys-edit ip/oneapi_asp_editor.ip --qpf=oneapi_asp_editor\nFigure A-1 oneAPI ASP Editor GUI
6) At the right bottom of the page are displayed all the OFS reference platform presets provided in the oneapi-asp. Select the board variant you want to include in your ASP, click the Apply botton, you will see how the parameters will update according to the board variant settings. Then click the generate HLD botton, select the options suitable for you, click generate and save the changes, repeat the process for the rest of the board variants you want to add in your ASP.
Figure A-2 Editor GUI Presets
Figure A-3 Generate HLD window
7) Once you have added all the board variants of the OFS reference platform you are using, exit platform designer. A folder with the name of the platform you are using will be created inside the oneapi_asp_editor folder, the presets files you have generated will be already included in the hardware subfolder.
8) Go to the folder of the reference platform you are using, export two environment variables and run the build-asp.sh script.
cd platform-name/\n export AOCL_BOARD_PACKAGE_ROOT=$PWD\nexport OFS_ASP_ROOT=$PWD\n ./scripts/build-asp.sh\nNote: Where platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005
Once you have completed running the oneAPI sample application, you can start developing your own applications.
For more information about developing FPGA applications with Intel oneAPI, refer to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs.
If you want to customize the oneAPI ASP, you can refer to oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
The Open FPGA Stack (OFS) Docker image has two main personas:
A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"hw/common/user_guides/ug_docker/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\nAdd the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\nInstall the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\nStart the Docker daemon:
sudo systemctl start docker\nEnable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\nVerify that Docker is installed and running:
sudo systemctl status docker\nYou should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\nYou will need to log out and back in for the changes to take effect.
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\nThe Docker installation steps for Ubuntu are the following:
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\nInstall packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\nAdd Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\nThe following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\nUpdate the package manager index again:
sudo apt-get update\nInstall Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\nStart the Docker daemon:
sudo systemctl start docker\nEnable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\nVerify that Docker is installed and running:
sudo systemctl status docker\nYou should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\nYou will need to log out and back in for the changes to take effect.
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\nThe Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#build-the-image","title":"Build the image","text":"You can download the Dockefile from OFS GitHub Docker.
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\nSave the file
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\nNote: Never remove --no-cache this could cause issues with your environmental variables inside of the container
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\nDocker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\nFor more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\nInside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\nThe docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nTip: you can change myOFS with any other value. The value is the given name of the container.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\nNow the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\nYour Container ID is bdc1289fb081.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\nThe container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\nEverything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
The OFS Docker image already includes the evaluation script.
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\nThe Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\nIf you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\nall the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nExample, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\nTip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\nNow, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n\u200b Your Container ID is 25b41eb4d232.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\nThe container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\nEverything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
The OFS Docker image already includes the eval script.
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\nNow you can control and compile the design using the interactive or de-attach mode.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/","title":"Evaluation User Guide: OFS for Agilex\u00ae 7 PCIe Attach","text":""},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#1-overview","title":"1 Overview","text":""},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the checkout and evaluation of an Intel\u00ae FPGA SmartNIC N6001-PL, Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) development platform using Open FPGA Stack (OFS). After reviewing the document, you will be able to:
A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae FPGA SmartNIC N6001-PL can be found on the OFS ofs-2024.2-1 official release drop on GitHub.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#2-introduction-to-ofs-evaluation-script","title":"2 Introduction to OFS Evaluation Script","text":"Two scripts have been developed to allow the user to clone, build, compile and test the OFS platform hardware.
1) ofs-clone_repositories.sh. This script clones the repositories from GitHub needed to build and test any OFS platform. 2) ofs-agx7-pcie-attach_eval.sh. This script evaluates compiles, builds and tests hardware features that the OFS framework provides. This script can also be leveraged for your own development.
NOTE:
This guide uses the Intel\u00ae FPGA SmartNIC N6001-PL as the platform for all example steps. Additionally, this guide and the example steps can be used with other platforms, such as the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile).
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#21-pre-requisites","title":"2.1 Pre-Requisites","text":"This script uses the following set of software tools which should be installed using the directory structure below. Tool versions can vary.
Figure 2-1 Folder Hierarchy for Software Tools
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#22-agilex-7-pcie-attach-clone-and-evaluation-script-download-and-modification","title":"2.2 Agilex\u00ae 7 PCIe Attach Clone and Evaluation Script download and modification","text":"Untar to a user folder using the following command
tar -xvf scripts_ofs-2024.1-1_external.tar.gz\n````\n\n3. The folder structure containing the clone script (ofs-clone_repositories.sh) and evaluation script (ofs-agx7-pcie-attach_eval.sh) is as shown below in Figure 2-2\n\n**Figure 2-2 Directory Structure for the clone script**\n\n``` sh\n## ofs_scripts\n## -> host_chan_mmio.stp\n## -> ofs-agx7-pcie-attach_eval.sh\n## -> README_ofs-agx7-pcie-attach_eval.txt\n## ofs-clone_repositories.sh\nOpen the clone script ofs-clone_repositories.sh in any text editor
Note: Failing to add proxy server will prevent cloning of repositories and the user will be unable to build the OFS framework.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\nPlease enter the license file locations (lines 11-13) for the following tool variables
export LM_LICENSE_FILE=<user_license>\n export DW_LICENSE_FILE=<user_license>\n export SNPSLMD_LICENSE_FILE=<user_license>\nAdd the name of the directory where you want the platform repositories to be cloned (line 19)
export OFS_RELEASE=ofs-2024.2-1\nAfter setting the required variables, source the clone script with the following command
source ofs-clone_repositories.sh\nFigure 2-3 Directory Structure for OFS Project
Once the repositories are cloned, the user can navigate to the following directory
cd ofs-2024.2-1\nAfter cloning, the OFS repositories will be shown below in Figure 2-4.
Figure 2-4 Directory Structure for OFS Project
## ofs-2024.2-1\n## N6001(OFS platform)\n## -> examples-afu\n## -> ofs-agx7-pcie-attach\n## -> oneapi-asp\n## -> oneAPI-samples\n## -> opae-sim\n## -> opae-sdk\n## -> linux-dfl-backport\n## -> ofs-agx7-pcie-attach_eval.sh\n## -> README_ofs-agx7-pcie-attach_eval.txt\n## -> host_chan_mmio.stp\na) Tools Location (line 87, 88, 89, 90)
Set Location of Quartus, Synopsys, Questasim and oneAPI Tools
export QUARTUS_TOOLS_LOCATION=/home\nexport SYNOPSYS_TOOLS_LOCATION=/home\nexport QUESTASIM_TOOLS_LOCATION=/home\nexport ONEAPI_TOOLS_LOCATION=/opt\nIn the example above /home is used as the base location of Quartus, Synopsys and Questasim tools, /opt is used for the oneAPI tools
b) PCIe (Bus Number)
The Bus number must be entered by the user after installing the hardware in the chosen server, in the example below \"b1\" is the Bus Number for a single card as defined in the evaluation script.
export OFS_CARD0_BUS_NUMBER=b1\nThe evaluation script uses the bus number as an identifier to interrogate the card. The command below will identify the accelerator card plugged into a server.
lspci | grep acc\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\nb1:00.1 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.2 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\nb1:00.4 Processing accelerators: Intel Corporation Device bcce<br>\nThe result identifies the card as being assigned \"b1\" as the bus number so the entry in the script changes to
export OFS_CARD0_BUS_NUMBER=b1\nThe user can also run the following command on the ofs-agx7-pcie-attach_eval.sh script to automatically change the bus number to b1 in the ofs-agx7-pcie-attach_eval.sh script.
grep -rli 'b1' * | xargs -i@ sed -i 'b1' @
if the bus number is 85 for example
85:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\n85:00.1 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.2 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\n85:00.4 Processing accelerators: Intel Corporation Device bcce<br>\nthe command to change to 85 in the evaluation script would be
grep -rli 'b1' * | xargs -i@ sed -i '85' @
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#ofs-platform-example-n6000-n6001-fseries-dk-iseries-dk-custom_board-choice-line-173","title":"OFS Platform (Example:= n6000, n6001, fseries-dk, iseries-dk, custom_board) choice (line 173)","text":"The ofs-agx7-pcie-attach_eval.sh script has now been modified to the server set-up and the user can proceed to build, compile and simulate the OFS stack
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#3-ofs-agx7-pcie-attach-evaluation-script","title":"3 ofs-agx7-pcie-attach Evaluation Script","text":""},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#31-overview","title":"3.1 Overview","text":"The figure below shows a snapshot of the full evaluation script menu showing all 62 options and each one of 11 sub-menus which focus on different areas of evaluation. Each of these menu options are described in the next section.
Figure 3-1 ofs-agx7-pcie-attach_eval.sh Evaluation Menu
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#311-ofs-tools-menu","title":"3.1.1 OFS TOOLS MENU","text":"By selecting \"List of Documentation for OFS Project,\" a list of links to the latest OFS documentation appears. Note that these links will take you to documentation for the most recent release which may not correspond to the release version you are evaluating. To find the documentation specific to your release, ensure you clone the OFS tag that corresponds to your OFS version.
By selecting \"Check Versions of Operating System and Quartus Premier Design Suite\", the tool verifies correct Operating System, Quartus version, kernel parameters, license files and paths to installed software tools.
Menu Option Example Output 1 - List of Documentation for OFS PCI Attach Project Project Open FPGA Stack Overview Guides you through the setup and build steps to evaluate the OFS solution https://ofs.github.io 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS) Checking Linux release Linux version 5.15.52-dfl (guest@hw-rae-svr4-l) (gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4), GNU ld version 2.30-79.el8) #1 SMP Fri Sep 23 17:19:37 BST 2022 Checking RedHat release CentOS Linux release 8.3.2011 Checking Ubuntu release cat: /etc/lsb-release: No such file or directory Checking Kernel parameters BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.15.52-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 Checking Licenses LM_LICENSE_FILE is set to port@socket number:port@socket number DW_LICENSE_FILE is set to port@socket number:port@socket number SNPSLMD_LICENSE_FILE is set to port@socket number:port@socket number Checking Tool versions QUARTUS_HOME is set to /home/intelFPGA_pro/24.1/quartus QUARTUS_ROOTDIR is set to /home/intelFPGA_pro/24.1/quartus IMPORT_IP_ROOTDIR is set to /home/intelFPGA_pro/24.1/quartus/../ip QSYS_ROOTDIR is set to /home/intelFPGA_pro/24.1/quartus/../qsys/bin Checking QPDS Patches Quartus Prime Shell Version 24.1 Build XXX XX/XX/XXXX Patches X.XX SC Pro Edition Copyright (C) XXXX Intel Corporation. All rights reserved."},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#312-ofs-hardware-menu","title":"3.1.2 OFS HARDWARE MENU","text":"Identifies card by PCIe number, checks power, temperature and current firmware configuration.
Menu Option Example Output 3 - Identify Platform Hardware via PCIe PCIe card detected as b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.4 Processing accelerators: Intel Corporation Device bcce Host Server is connected to SINGLE card configuration 4 - Identify the Board Management Controller (BMC) Version and check BMC sensors Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** BMC SENSORS ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 5 - Identify the FPGA Management Engine (FME) Version Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Management Controller Build version: 3.2.0 //****** FME ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 Boot Page : user1 Factory Image Info : a7c6879683182ce61084c420e51f50b6 User1 Image Info : 8a7440ddff52e0e27dbb989d5eb954f4 User2 Image Info : a7c6879683182ce61084c420e51f50b6 6 - Check Board Power and Temperature Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** POWER ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 ( 1) VCCRT_GXER_0V9 Voltage : 0.91 Volts etc ...................... Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** TEMP ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 ( 1) FPGA E-Tile Temperature [Remote] : 33.50 Celsius etc ...................... Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 7 - Check Accelerator Port status //****** PORT ******// Object Id : 0xED00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 8 - Check MAC and PHY status Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** MAC ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 Number of MACs : 255 mac info is not supported Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** PHY ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 //****** HSSI information ******// HSSI version : 1.0 Number of ports : 8 Port0 :25GbE DOWN Port1 :25GbE DOWN Port2 :25GbE DOWN Port3 :25GbE DOWN Port4 :25GbE DOWN Port5 :25GbE DOWN Port6 :25GbE DOWN Port7 :25GbE DOWN Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#313-ofs-pfvf-mux-menu","title":"3.1.3 OFS PF/VF MUX MENU","text":"This menu reports the number of PF/VF functions in the reference example and also allows you to reduce the number to 1PF and 1VF to reduce resource utilisation and create a larger area for your workload development. This selection is optional and if the user wants to implement the default number of PF's and VF's then option 9, 10 and 11 should not be used. Additionally the code used to make the PF/VF modification can be leveraged to increase or modify the number of PF/VFs in the existing design within the limits that the PCIe Subsystem supports (8PF/2K VFs)
Menu Option Description 9 - Check PF/VF Mux Configuration This menu selection displays the current configuration of all n6001 ofss files located in the following directory $OFS_ROOTDIR/tools/ofss_config Check n6001 base config OFSS set up [ip] type = ofs [settings] platform = n6001 family = agilex fim = base_x16 part = AGFB014R24A2E2V device_id = 6001 Check n6001 hssi_2x100 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GCAUI-4 Check n6001 hssi_2x100_caui2 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GAUI-2 Check n6001 hssi_8x10 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 10GbE Check n6001 hssi_8x25 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 25GbE Check n6001 IOPLL OFSS set up [ip] type = iopll [settings] output_name = sys_pll instance_name = iopll_0 [p_clk] freq = 470 Check n6001 Memory OFSS set up [ip] type = memory [settings] output_name = mem_ss_fm preset = n6001 Check n6001 PCIe Host OFSS set up [ip] type = pcie [settings] output_name = pcie_ss [pf0] num_vfs = 3 bar0_address_width = 20 vf_bar0_address_width = 20 [pf1] [pf2] bar0_address_width = 18 [pf3] [pf4] Check n6001 PCIe 1pf_1vf OFSS set up [ip] type = pcie [settings] output_name = pcie_ss [pf0] num_vfs = 1 bar0_address_width = 20 vf_bar0_address_width = 20 10 - Modify PF/VF Mux Configuration As an example this menu selection modifies the pcie_host.ofss file to 1 PF located in the following directory $OFS_ROOTDIR/tools/ofss_config This option also displays the modified pcie_host.ofss file 11 - Build PF/VF Mux Configuration If option 10 is not used then then the default number of PF's and VF's is used to build the FIM, if option 10 is selected then only 1 VF is built to reduce logic utilisation"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#314-ofs-fimpr-build-menu","title":"3.1.4 OFS FIM/PR BUILD MENU","text":"Builds FIM, Partial Reconfiguration Region and Remote Signal Tap
Menu Option Description 12 - Check OFS software version for OFS Project OFS_ROOTDIR is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach OPAE_SDK_REPO_BRANCH is set to release/X.X.X OPAE_SDK_ROOT is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/../opae-sdk LD_LIBRARY_PATH is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/../opae-sdk/lib64: 13 - Build FIM for Hardware This option builds the FIM based on the setting for the $OFS_PLATFORM, $FIM_SHELL environment variable. Check this variable in the following file ofs-agx7-pcie-attach_eval.sh 14 - Check FIM Identification of FIM for Hardware The FIM is identified by the following file fme-ifc-id.txt located at $OFS_ROOTDIR/$FIM_WORKDIR/syn/board/$OFS_PLATFORM/syn_top/ 15 - Build Partial Reconfiguration Tree for Hardware This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the oneAPI build flow 16 - Build Base FIM Identification(ID) into PR Build Tree template This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and oneAPI workloads 17 - Build Partial Reconfiguration Tree for Hardware with Remote Signal Tap This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the oneAPI build flow for the Remote Signal Tap flow 18 - Build Base FIM Identification(ID) into PR Build Tree template with Remote Signal Tap This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree for Remote Signal Tap to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and oneAPI workloads"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#315-ofs-hardware-programmingdiagnostic-menu","title":"3.1.5 OFS HARDWARE PROGRAMMING/DIAGNOSTIC MENU","text":"The following submenu allows you to: * Program and check flash * Perform a remote system update (RSU) of the FPGA image into the FPGA * Bind virtual functions to VFIO PCIe driver * Run host exerciser (HE) commands such as loopback to test interfaces VFIO PCI driver binding * Read the control and status registers (CSRs) for bound modules that are part of the OFS reference design.
Menu Option Description 19 - Program BMC Image into Hardware The user must place a new BMC flash file in the following directory $OFS_ROOTDIR/bmc_flash_files. Once the user executes this option a new BMC image will be programmed. A remote system upgrade command is initiated to store the new BMC image 20 - Check Boot Area Flash Image from Hardware This option checks which location area in FLASH the image will boot from, the default is user1 Boot Page : user1 Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 21 - Program FIM Image into user1 area for Hardware This option programs the FIM image \"ofs_top_page1_unsigned_user1.bin\" into user1 area in flash Note: Please refer to the Getting Started Guide for details on flashing images for the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 22 - Initiate Remote System Upgrade (RSU) from user1 Flash Image into Hardware This option initiates a Remote System Upgrade and soft reboots the server and re-scans the PCIe bus for the new image to be loaded 2022-11-10 11:26:24,307 - [[pci_address(0000:b1:00.0), pci_id(0x8086, 0xbcce)]] performing RSU operation 2022-11-10 11:26:24,310 - [[pci_address(0000:b0:02.0), pci_id(0x8086, 0x347a)]] removing device from PCIe bus 2022-11-10 11:26:24,357 - waiting 10 seconds for boot 2022-11-10 11:26:34,368 - rescanning PCIe bus: /sys/devices/pci0000:b0/pci_bus/0000:b0 2022-11-10 11:26:35,965 - RSU operation complete Note: Please refer to the Getting Started Guide for details on flashing images for the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 23 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 24 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 25 - Create Virtual Functions (VF) and bind driver to vfio-pci Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 23 to check that the new drivers are bound 26 - Verify FME Interrupts with hello_events The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors 27 - Run HE-LB Test This option runs 5 tests 1) checks and generates traffic with the intention of exercising the path from the AFU to the Host at full bandwidth 2) run a loopback throughput test using one cacheline per request 3) run a loopback read test using four cachelines per request 4) run a loopback write test using four cachelines per request 5) run a loopback throughput test using four cachelines per request 28 - Run HE-MEM Test This option runs 2 tests 1) Checking and generating traffic with the intention of exercising the path from FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host 2) run a loopback throughput test using one cacheline per request 29 - Run HE-HSSI Test This option runs 1 test HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic 1) Send traffic through the 10G AFU 30 - Run Traffic Generator AFU Test This option runs 3 tests TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank 1) Run the preconfigured write/read traffic test on channel 0 2) Target channel 1 with a 1MB single-word write only test for 1000 iterations 3) Target channel 2 with 4MB write/read test of max burst length for 10 iterations 31 - Read from CSR (Command and Status Registers) for Hardware This option reads from the following CSR's HE-LB Command and Status Register Default Definitions HE-MEM Command and Status Register Default Definitions HE-HSSI Command and Status Register Default Definitions"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#316-ofs-hardware-afu-testing-menu","title":"3.1.6 OFS HARDWARE AFU TESTING MENU","text":"This submenu tests partial reconfiguration by building and loading an memory-mapped I/O example AFU/workload, executes software from host, and tests remote signal tap.
Menu Option Description 32 - Build and Compile host_chan_mmio example This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) ready for hardware programming 33 - Execute host_chan_mmio example This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test 34 - Modify host_chan_mmio example to insert Remote Signal Tap This option inserts a pre-defined host_chan_mmio.stp Signal Tap file into the OFS code to allow a user to debug the host_chan_mmio AFU example 35 - Build and Compile host_chan_mmio example with Remote Signal Tap This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS(Green Bit Stream) ready for hardware programming with Remote Signal tap enabled 36 - Execute host_chan_mmio example with Remote Signal Tap This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test. The user must open the Signal Tap window when running the host code to see the transactions in the Signal Tap window"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#317-ofs-hardware-afu-bbb-testing-menu","title":"3.1.7 OFS HARDWARE AFU BBB TESTING MENU","text":"This submenu tests partial reconfiguration using a hello_world example AFU/workload, executes sw from host
Menu Option Description 37 - Build and Compile hello_world example This option builds the hello_ world example from the following repo $FPGA_BBB_CCI_SRC/samples/tutorial/afu_types/01_pim_ifc/$AFU_BBB_TEST_NAME, where AFU_BBB_NAME=hello_world. This produces a GBS(Green Bit Stream) ready for hardware programming 38 - Execute hello_world example This option builds the host code for hello_world example and programs the GBS file and then executes the test"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#318-ofs-oneapi-project-menu","title":"3.1.8 OFS ONEAPI PROJECT MENU","text":"Builds oneAPI kernel, executes sw from host and runs diagnostic tests
Menu Option Result 39 - Check oneAPI software versions for Project This option checks the setup of the oneAPI software and adds the relevant oneAPI environment variables to the terminal. This option also informs the user to match the oneAPI software version to the oneAPI-samples version 40 - Build and clone shim libraries required by oneAPI host This option builds the oneAPI directory structure 41 - Install oneAPI Host Driver This option Installs the oneAPI Host driver at the following location /opt/Intel/OpenCLFPGA/oneAPI/Boards/, and requires sudo permission 42 - Uninstall oneAPI Host Driver This option Uninstall's the oneAPI Host driver, and requires sudo permissions 43 - Diagnose oneAPI Hardware This option Checks ICD (Intel Client Driver) and FCD (FPGA Client Driver), oneAPI library locations and detects whether oneAPI BSP is loaded into the FPGA 44 - Build oneAPI BSP Default Kernel (hello_world) This option Builds the oneAPI BSP using simple-add_buffers kernel 45 - Build oneAPI MakeFile Environment This option Builds the oneAPI environment using a Makefile for kernel insertion 46 - Compile oneAPI Sample Application (board_test) for Emulation This option compiles the board_test kernel for Emulation 47 - Run oneAPI Sample Application (board_test) for Emulation This option executes the board_test kernel for Emulation 48 - Generate oneAPI Optimization report for (board_test) This option generates an optimization report for the board_test kernel 49 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 50 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 51 - Create Virtual Function (VF) and bind driver to vfio-pci Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 45 to check that the new drivers are bound 52 - Program OpenCL BSP Default Kernel (hello_world) This option programs the FPGA with a aocf file based on the hello_world kernel 53 - Compile oneAPI Sample Application (board_test) for Hardware This option compiles the board_test kernel for Hardware 54 - Run oneAPI Sample Application (board_test) for Hardware This option builds the host code for board_test kernel and executes the program running through kernel and host bandwidth tests"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#319-ofs-unit-test-project-menu","title":"3.1.9 OFS UNIT TEST PROJECT MENU","text":"Builds, compiles and runs standalone simulation block tests. More unit test examples are found at the following location ofs_n6001/sim/unit_test
Menu Option Result 55 - Generate Simulation files for Unit Test This option builds the simulation file set for running a unit test simulation 56 - Simulate Unit Test dfh_walker and log waveform This option runs the dfh_walker based on the environment variable \"UNIT_TEST_NAME=dfh_walker\" in the evaluation script. A user can change the test being run by modifying this variable"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#3110-ofs-uvm-project-menu","title":"3.1.10 OFS UVM PROJECT MENU","text":"Builds, compiles and runs full chip simulation tests. The user should execute the options sequentially ie 68,69, 70 and 71
Menu Option Description 57 - Check UVM software versions for Project DESIGNWARE_HOME is set to /home/synopsys/vip_common/vip_Q-2020.03A UVM_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm VCS_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel VERDIR is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/verification VIPDIR is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/verification 58 - Compile UVM IP This option compiles the UVM IP 59 - Compile UVM RTL and Testbench This option compiles the UVM RTL and Testbench 60 - Simulate UVM dfh_walking_test and log waveform This option runs the dfh_walking test based on the environment variable \"UVM_TEST_NAME=dfh_walking_test\" in the evaluation script. A user can change the test being run by modifying this variable 61 - Simulate all UVM test cases (Regression Mode) This option runs the n6001 regression mode, cycling through all UVM tests defined in verification/tests/test_pkg.svh file"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#3111-ofs-build-all-project-menu","title":"3.1.11 OFS BUILD ALL PROJECT MENU","text":"Builds the complete OFS flow, good for regression testing and overnight builds
For this menu a user can run a sequence of tests (compilation, build and simulation) and executes them sequentially. After the script is successfully executed, a set of binary files is produced which a you can use to evaluate your hardware. Log files are also produced which checks whether the tests passed.
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 62 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 44, 45, 53, 55, 56, 57, 58, 59 and 60. These 24 menu options are chosen to build the complete OFS flow covering build, compile and simulation.
Menu Option Result 62 - Build and Simulate Complete Project Generating Log File with date and timestamp Log file written to /home/guest/ofs-2.3.1/log_files/ofs-agx7-pcie-attach_log_2022_11_10-093649/ofs-agx7-pcie-attach_eval.log"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#definition-of-multi-test-set-up","title":"Definition of Multi-Test Set-up","text":"Menu Option 62 above in the evaluation script can be refined to tailor it to the users need and is principally defined by the variable below
MULTI_TEST[A,B]=C
where
A= Total Number of menu options in script B= Can be changed to a number to select the test order C= Menu Option in Script
Example 1 MULTI_TEST[62,0]=2
A= 62 is the total number of options in the script B= 0 indicates that this is the first test to be run in the script C= Menu option in Script ie 2- List of Documentation for OFS n6001 Project
Example 2 MULTI_TEST[62,0]=2 MULTI_TEST[62,1]=9
In the example above two tests are run in order ie 0, and 1 and the following menu options are executed ie 2- List of Documentation for OFS n6001 Project and 9 - Check OFS software versions for OFS n6001 Project
The user can also modify the build time by de-selecting options they do not wish to use, see below for a couple of use-case scenarios.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#default-user-case","title":"Default User Case","text":"A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 62 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 44, 45, 53, 55, 56, 57, 58, 59 and 60. All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#user-case-for-ofs-fimpr-build-menu","title":"User Case for OFS FIM/PR BUILD MENU","text":"In the example below when the user selects option 62 from the main menu the script will only run options from the OFS FIM/PR BUILD MENU (7 options, main menu options 12, 13, 14, 15, 16, 17 and 18). All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#4-n6001-common-test-scenarios","title":"4 n6001 Common Test Scenarios","text":"This section will describe the most common compile build scenarios if a user wanted to evaluate an acceleration card on their server. The Pre-requisite column indcates the menu comamnds that must be run befere executing the test eg To run Test 5 then a user needs to have run option 13, 15 and 16 before running options 23, 24, 25, 32 and 33.
Test Test Scenario Pre-Requisite Menu Option Menu Option Test 1 FIM Build - 13 Test 2 Partial Reconfiguration Build 13 15, 16 Test 3 Program FIM and perform Remote System Upgrade 13 21, 22 Test 4 Bind PF and VF to vfio-pci drivers - 23, 24, 25 Test 5 Build, compile and test AFU on hardware 13, 15, 16 23, 24, 25, 32, 33 Test 6 Build, compile and test AFU Basic Building Blocks on hardware 13, 15, 16 23, 24, 25, 37, 38 Test 7 Build, compile and test oneAPI on hardware 13, 15, 16 39, 40, 41, 44, 45, 49, 50, 51, 52, 53, 54 Test 8 Build and Simulate Unit Tests - 55, 56 Test 9 Build and Simulate UVM Tests - 57, 58, 59, 60 Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/","title":"FPGA Developer Journey Guide: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#1-introduction","title":"1 Introduction","text":"This document is intended to help you understand the FPGA developer flow using OFS as well as considerations you should take when creating your custom platform.
After reviewing the document, you shall be able to:
The general development flow is depicted in the diagram below and discussed in more detail in each section of this document.
Figure 1-1: FPGA Developer Flow
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#2-evaluate-ofs","title":"2 Evaluate OFS","text":"The repositories in the OFS site are tagged based on year and release number in the format <yyyy>.<release>-<iteration>. For example, a tag of 2024.2-1 indicates the 2nd release of the year. If there are any updates to a particular release package, the last number will be incremented by 1, for example, as 2024.2-2.
By clicking on the release link to the right of the RTL repositories, you will find the latest release, the tag number and release notes.
Figure 2-1: OFS Repository Release Page Link
By scrolling to the end of the release page, you will find assets attached to the release that you can leverage for quick evaluation of OFS, such as FPGA binary files, POF and SOF as well as a sample AFU and a partial reconfiguration directory for building your own custom workload.
There are two ways to evaluate OFS depending on your needs:
Option 1: Setup your card and software in a server using the steps provided in one of the corresponding Getting Started Guides and leverage the appended binaries in the FIM RTL repository release page \"Assets\" tab to preview the software and design functionality the OFS framework provides you out of the box. This step will give you a good high-level overview of OFS. Getting Started Guides are available for the following FIM(shell) designs:
Option 2: After your card and software are installed using the steps provided in one of the corresponding Getting Started Guides listed above, use a corresponding Evaluation Guide and provided evaluation script to run through all the capabilities of the OFS framework by selecting one of the choices in the evaluation menu. The evaluation script gives you familiarity of the entire design, build, simulation, programming and test flow for OFS, including a OneAPI flow.
The scripts that go with each user guide are found in the assets tab of the corresponding FIM RTL repository's latest tag release page.
To use the full functionality of the script you will want to ensure you clone all of the appropriate repositories below at the same level just under an OFS directory you create, such as ofs-2024.2, similar to the figure below.
Figure 2-2: Directory Structure of Cloned Repositories
##|-- ofs-2024.2\n##| |--examples-afu\n##| |--linux-dfl-backport\n##| |--ofs-agx7-pcie-attach\n##| |--ofs-oneapi-asp\n##| |--opae-sdk\n##| |--opae-sim\n##| |--ofs-agx7-pcie-attach_eval.sh\nYou can access the repositories from ofs.github.io by clicking on the GitHub icon in the right corner of the site.
Figure 2-3: OFS Site Link from ofs.github.io
After making your top level directory and initializing the repository, clone one of the FIM RTL repositories you intend to use:
#Make top level directory\nmkdir OFS\ncd OFS\n\n#Initialize repository\ngit init\n\n#Select a FIM RTL repository to clone\n#To clone Intel Agilex 7 PCIe Attach FIM RTL repository\ngit clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\ncd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n\n#To clone Intel Agilex 7 SoC Attach FIM RTL Repository\ngit clone --recurse-submodules https://github.com/OFS/ofs-f2000x-pl.git\ncd ofs-f2000x-pl\ngit checkout --recurse-submodules tags/ofs-2024.1-1\n\n#To clone Intel Stratix 10 PCIe Attach FIM RTL Repository\ngit clone --recurse-submodules https://github.com/OFS/ofs-d5005.git\ncd ofs-d5005\ngit checkout --recurse-submodules tags/ofs-2024.1-1\nAfter cloning your FIM RTL repository, clone the other necessary repositories:
#All other repositories below should be cloned to use the evaluation script:\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n\n#Use this OPAE clone command for all PCIe Attach Cards\ngit clone https://github.com/OFS/opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\n\n#Use this OPAE clone command for the SoC Attach Card\ngit clone https://github.com/OFS/opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.12.0-5\n\ngit clone https://github.com/OFS/oneapi-asp.git cd /home/OFS/oneapi-asp\ngit checkout tags/ofs-2024.2-2\n\ngit clone https://github.com/OFS/ofs-platform-afu-bbb.git\ncd /home/OFS/ofs-platform-afu-bbb\ngit checkout tags/ofs-2024.1-1\n\ngit clone https://github.com/OFS/opae-sim.git\ncd /home/OFS/opae-sim\ngit checkout tags/2.13.0-2\n\ngit clone https://github.com/OFS/examples-afu.git\ncd /home/OFS/examples-afu\ngit checkout tags/ofs-2024.1-1\nYou will also want to ensure you install the correct version of Intel Quartus Prime Pro as directed in the release notes in addition to any Quartus patches. Note that Quartus Prime Pro software can be downloaded from the downloads tab on intel.com. Quartus Prime Pro patches required are attached to the assets tab at the bottom of the tagged RTL repository release page. Simulator tools as listed corresponding UVM Simulation User Guides:
To begin your development, start with an existing shell that most closely matches your device and end solution. The OFS site has both Intel Intel Agilex 7 and Stratix 10 FPGA reference designs you can use as a starting point for your own design. These designs can be ported to different device OPNs if a different resource utilization is required.
To begin you may want to learn more about Intel Stratix 10 and Intel Agilex 7 family offerings. Please refer to the following links:
Note that each reference design provides an integrated shell, called the FPGA Interface Manager (FIM), which is encompasses in blue in the diagrams below. This shell provides standard AXI interfaces to the Accelerator Functional Unit (AFU) region, shown in green, which depicts a region where a workload or partial reconfiguration slot resides. The regions are not drawn to scale. The figures and tables below give an overview of the available starting points for your design.
Figure 3-1: OFS FIM Targeting Agilex\u00ae7 PCIe Attach (P-tile, E-tile)
Key Feature Description Target OPN AGFB014R24A2E2V PCIe P-tile PCIe* Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 5 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each* Four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB Ethernet 2x4x25GbE, 2x4x10GbE or 2x100GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration)* Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Intel\u00ae FPGA SmartNIC N6001-PLClick here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
Figure 3-2: OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xF-tile)
Key Feature Description Target OPN AGFB027R24C2E2VR2 PCIe P-tile PCIe* Gen4x16 (currently downtrains to Gen4x8 in the ES version of the development kit) Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 3 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 2400 MHz, 1GB each* Two Fabric DDR4 banks, x64 (no ECC), 2400 MHz, 8GB Ethernet 2x4x25GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
Figure 3-3: OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xR-tile, F-tile)
Key Feature Description Target OPN AGIB027R29A1E2VR3 PCIe PR-tile PCIe 1xGen5x16R-tile PCIe 2xGen5x8R-tile PCIe 1xGen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 4 Fabric DDR4 channels, x64 (no ECC), 2666 MHz, 8GB Ethernet 2x4x25GbE, 2x200GbE, 2x400GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
Figure 3-4: OFS FIM Features Targeting Agilex\u00ae 7 SoC Attach
Key Feature Description Device OPN AGFC023R25A2E2VR0 PCIe P-tile PCIe* Gen4x16 to the HostP-tile PCIe* Gen4x16 to the SoC (IceLake-D) Virtualization Host: 2 physical functions SoC: 1 physical function and 3 Virtual functions Memory Four Fabric DDR4 banks, x40 (optional ECC, be configured as x32 and ECC x8 ), 1200 MHz, 4GB Ethernet 2x4x25GbE Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) * Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported Software Support * Linux DFL drivers targeting OFS FIMs * OPAE Software Development Kit * OPAE Tools Target Board Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PLNote: Source code for BMC RTL/Firmware that works with this reference FIM can be obtained by contacting your Intel Sales Representative.
Click here for the OFS Collateral for Agilex\u00ae SoC Attach Reference FIM documentation collection.
Figure 3-5: OFS FIM Targeting Stratix\u00ae 10 FPGA PCIe Attach
Key Feature Intel Stratix 10 Reference FIM Device OPN 1SX280HN2F43E2VG Ethernet Configuration 1x10GbE example with 2x100GbE capability PCIe Gen3x16 EMIF Up to four DDR channels PF/VF 1 PF/3 VFs Management FPGA Management Engine (FME) with FIM management registers Interface Arm\u00ae AMBA\u00ae4 AXI Interface HLD support oneAPI Software Kernel code upstreamed to Linux.org Target Board Intel\u00ae FPGA Programmable Acceleration Card D5005Click here for the OFS Collateral for Stratix\u00ae 10 FPGA PCIe Attach Reference FIM documentation.
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#4-review-release-notes","title":"4 Review Release Notes","text":"Before beginning your design, read the release notes for each repository you are using by selecting the appropriate release tag found by clicking on the \"Release\" link to the right of the corresponding repository. The release page may also have assets appended such as useful binaries, patches or scripts.
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#5-setup-your-environment-for-development","title":"5 Setup Your Environment For Development","text":"
When you are ready to begin development you will want to ensure you have any other setup requirements satisfied by reviewing instructions in the corresponding FIM Developer Guide and if you are implementing a OneAPI Board Support Package, the oneAPI ASP Getting Started User Guide as well.
For oneAPI setup:
For your own custom design, you may need to:
The FIM Developer Guides for each reference FIM show you how to make these changes and provide steps on how to run unit simulation or add SignalTap to your design for debugging.
If you are also interested in testing different examples for the Acceleration Functional Unit (AFU) or workload region of your design or creating your own AFU design, you can refer to the corresponding AFU Developer Guides:
Setup and test files to perform system-level Universal Verification Methodology (UVM) testing are provided in each FIM RTL repository. Please refer to the corresponding UVM Simulation User Guide for details on test bench architecture, setup and testing.
If you are considering providing oneAPI support for your custom board design, you must integrate the oneAPI ASP hardware and software components into your design. Reference the following documents to learn about the architecture and implementation flow for the oneAPI ASP with an OFS design.
As you add or remove interfaces to your custom design, you may need to modify or enhance existing drivers that accompany the OFS reference design. Additionally, you may decide you want to create additional utilities or plugins leveraging the OFS software infrastructure. In this case, refer to the Software Reference Manual: Open FPGA Stack to learn more about the underlying driver and software architecture and how to make modifications.
Additionally, for guidance on using a Kernel-based Virtual Machine with OFS, refer to our KVM User Guide.
KVM User Guide: Open FPGA Stack
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#10-test-and-deploy","title":"10 Test and Deploy","text":"When testing and deploying your platform you are encouraged to modify and tailor the evaluation scripts you used in Section 2 to assist in automating your continuous integration flow. You may also want to refer to our Docker User Guide to understand how to use Docker for Development and Deployment.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\nThis command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\nIn this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":"lsmod | grep kvm\nIf the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\nVirtual Machine Manager (also known as libvirt) can be installed by following the below steps:
Update your system package index by running the following command:
sudo dnf update\nsudo apt update\nInstall the libvirt package and any required dependencies by running the following command:
sudo dnf install @virtualization\nsudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\nStart the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\nsudo dnf install virt-manager\nsudo usermod -a -G libvirt USERNAME\nvirt-manager as the non-root user.Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\nYou will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\nreboot\nAfter completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
virt-manager GUI by running the following command:sudo virt-manager&\nCreate a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
In the virt-manager window, click the \"New virtual machine\" button.
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
Click on the \"+\" icon (Bottom left) to create the storage pool.
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n\u200b
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3 Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\nCheck that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\nThe following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbbEnsure you add the desired VF in your PCIE devices list.
For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nPass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\nEdit the XML file for your machine and include the following
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\nInside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\nEnsure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\nNote: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
Save the changes \"Apply\"
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\nEnsure your devices are enumerated properly.
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n2. Deployment Mode:
B1:00.5\nUnder the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n2. Deployment Mode:
177:00.0\nClick on \"Begin Installation.\" and follow the wizard installation of the OS.
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
Under your virtual machine, configure your VM proxy:
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\nUse the Virtual function (Not supported at management mode)
Ensure the DFL kernel drivers is install in your VM system
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\nVerify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\nTest the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\nAfter the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/","title":"UVM Simulation User Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel Max10 or Intel Cyclone10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implemented on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Management Component Transport Protocol MCTP A standardized model for communication with management controllers. Defines the transport protocol carrying PLDM messages through the BMC. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE-SDK The OPAE-SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Level Data Model PLDM A specification for reporting telemetry data to the host, such as board temperature, voltage, and current. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#1-overview","title":"1 Overview","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the UVM simulation tool using Open FPGA Stack (OFS) for Agilex\u00ae 7 FPGAs PCIe Attach and the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). After reviewing the document, you will be able to:
NOTE:
This guide uses the Intel\u00ae FPGA SmartNIC N6001-PL as the platform for all example steps. Additionally, this guide and the example steps can be used with other platforms, such as the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile).
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#2-introduction-to-uvm","title":"2 Introduction to UVM","text":"The Functional Verification Environment for OFS is UVM (Universal Verification Methodology) compliant and provides configurable setup for verifying various FIM features in simulation.
The purpose of this document is to demonstrate a full chip level and unit level UVM based verification environment for the current base shell FIM architecture as well as providing examples to extend the setup for different OFS variants by reusing the existing architecture and infrastructure for UVM based verification.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#3-universal-testbench-architecture","title":"3 Universal Testbench Architecture","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#31-overview","title":"3.1 Overview","text":"The main idea behind UVM is to develop modular, reusable, and a scalable testbench structure by providing an API framework that can be deployed across multiple projects.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#32-core-verification-concepts","title":"3.2 Core Verification Concepts","text":"The following is the list of verification components that will be used to design a UVM testbench architecture:
\u2022 Sequencer \u2022 Driver \u2022 Monitor \u2022 Scoreboard \u2022 Virtual Sequencer \u2022 Interface \u2022 Verification Environment \u2022 TOP Testbench
Figure 1 provides the general UVM Testbench and the verification components involved in the top-level architecture.
Figure 1 Typical UVM Testbench
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4-ofs-testbench-architecture","title":"4 OFS Testbench Architecture","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#41-overview","title":"4.1 Overview","text":"OFS (Open FPGA Stack) provides a UVM (Universal Verification Methodology) environment for the FIM with a modular, reusable, and scalable testbench structure via an API framework.
The framework consists of a FIM Testbench which is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this Testbench. UVM RAL(Register Abstraction Level) is used for CSR (Command and Status Registers) verification.
The qualified verification IPs will help to detect incorrect protocol behavior, help to focus on FIM features and accelerate the verification process.
Verification components include:
\u2022 FIM monitor to detect correct design behavior\n\u2022 FIM assertions for signal level integrity testing\n\u2022 Arm AMBA Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity\n\u2022 FIM coverage to collect functional data\nThe hardware architecture of an Agilex FIM is based on the OFS hardware architecture. The following is the list of features and subsystems supported in the base shell.
\u2022 PCIe Subsystem\n\u2022 HSSI Subsystem\n\u2022 Memory Subsystem\n\u2022 HPS Subsystem\n\u2022 FME\n\u2022 AFU with PR support\n\u2022 QSFP Controllers\n\u2022 PMCI Controller, MCTP\nFigure 2 DUT Base Shell Diagram
Figure 2 shows the high level architecture of an Agilex Base Shell. It has a Gen4x16, 100G Ethernet Datapath in a 2x4x25G configuration. The Agilex Base Shell is a shell that will enable a user to build other shell variants for a custom configuration. For the N6001 board there is one shell variant
base_x16
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43-full-chip-level-verification-archiecture-for-fim","title":"4.3 Full Chip Level Verification Archiecture for FIM","text":"Figure 3 shows a graphical representation a full chip testbench that includes major RTL blocks depicted in a OFS Agilex based UVM environment
Figure 3 OFS FIM Testbench
The major connection is the interface between the Xeon CPU and the FPGA where the PCIe Verification IP is connected to PCIe Subsystem. Therefore, as a full chip simulation environment, PCIe host VIP is the sole VIP/BFM used. PCIe host VIP connects to PCIe device which resides in FPGA in serial mode.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#431-testbench-components","title":"4.3.1 Testbench components","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4311-tb-top","title":"4.3.1.1 TB TOP","text":"TOP is the top level testbench and consists of a FIM DUT instance and top-level UVM Verification Environment instance. It binds RTL inputs with the verification environmnet interfaces to drive stimulus. It also has clock generation and reset driving logic.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4312-fim-verification-environment","title":"4.3.1.2 FIM Verification Environment","text":"This is the top most verification environment class and consists of the protocol specific PCI Express and AXI UVM environment VIP instances, Monitors, Scoreboards, Assertions, Functional coverage Modules and other verification components. It instantiates Virtual sequencers to control stimuli for FIM traffic from different sequencers of the VIPs.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4313-synopsys-vips","title":"4.3.1.3 Synopsys VIPs","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43131-pci-vip-as-host","title":"4.3.1.3.1 PCI VIP as Host","text":"This is Synopsys Licensed PCI Express Gen4 VIP and acts as Root Port. The PCI Express link is connected to the DUT using TX-RX lanes. Agent is an active component and includes a sequencer to generate TLPs, Driver to drive it on a PCI Express link and Monitor to check the protocol correctness.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43132-axi-streaming-vip-monitors","title":"4.3.1.3.2 AXI-Streaming VIP Monitors","text":"This is Synopsys Licensed AXI streaming interface Verification IP used as a Passive Agent to monitor AXI-ST links at various points. Please refer to Figure 3 to see all the AXI-ST monitors connected to different modules.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43133-axi4-memory-mapped-vip-monitors","title":"4.3.1.3.3 AXI4-Memory Mapped VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 memory mapped interface Verification IP used in passive mode to observe memory requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and performing data integrity checks.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43134-axi4-lite-vip-monitors","title":"4.3.1.3.4 AXI4-Lite VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used in passive mode to observe MMIO requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and perfoming data integrity checks. Please refer to Figure 3 to see all the Arm\u00ae AMBA\u00ae 4 AXI4-Lite monitors connected to different modules.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43135-axi4-lite-vip-as-pmci-master","title":"4.3.1.3.5 AXI4-Lite VIP as PMCI Master","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used to drive and observe MMIO requests as PMCI master. For each master-slave pair, the verification environment has a VIP instantiated in active mode and includes a sequencer to generate MMIO requests, driver to drive these requests on AXI-4 Lite interface to BPF and a monitor for observing protocol violations and data integrity checks.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4314-axi4-s-scoreboard","title":"4.3.1.4 AXI4-S Scoreboard","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-S scoreboard checks data integrity of source and sink components. It has input transactions from VIP monitors and a TLP to AXI converter for PCIe TLP packets. It makes sure the valid TLPs or AXI transactions are not missed while traversing from Host to AFU and reverse. The scoreboard will be disabled for error testing especially for invalid TLP requests and UR responses.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4315-virtual-sequencer","title":"4.3.1.5 Virtual Sequencer","text":"The virtual sequencer is the top-level sequencer which controls Enumeration, MMIO Requests, downstream and Upstream traffic as well as HSSI and Memory transactions. It makes sure that the transactions are ordered correctly as per the design specification while running a UVM test simulation.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4316-fim-monitor","title":"4.3.1.6 FIM Monitor","text":"The FIM monitor is used for checking the correctness of a specific FIM feature. As an example, a user can add interrupt related checks, error generation and clearing checks, Protocol checker impacts etc. in this component.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4317-fim-assertions","title":"4.3.1.7 FIM Assertions","text":"The assertion component is used for checking the signal level integrity of a specific FIM feature or behavior. As an example, we can add interrupt signal triggering and clear assertions, Error detection to error register bit assertions, protocol checker and clearing logic, FLR behavior assertion etc. in this top-level module. There are alos assertions to make sure the inputs are driven correctly to catch errors in a users simulation.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4318-fim-functional-coverage","title":"4.3.1.8 FIM Functional Coverage","text":"The FIM functional coverage component will have the functional coverage of each feature like interrupts, CSR's, MMIO requests, Byte align transactions, error reporting etc. to demonstrate a variety of FIM features. This will help us to find holes in the design as well as wide verification coverage to make sure thorough testing of a design. It will also provide a user what the FIM is capable of and how much code coverage they are testing.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#5-uvm-verification-set-up","title":"5 UVM Verification set-up","text":"To run the tutorial steps in this guide requires the following development environment:
Item Version Intel Quartus Prime Pro Intel Quartus Prime Pro 24.1 Simulator Synopsys VCS S-2023.03-SP2-1 or newer for UVM simulation of top level FIM"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#51-uvm-prerequisite","title":"5.1 UVM Prerequisite","text":"Retrieve OFS repositories.
The OFS FIM source code is included in the OTCShare GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories. Cloning the repo using the HTTPS method requires a personal access token. Please see this blog post for information about obtaining a personal access token Token authentication requirements for Git operations.
Navigate to location for storage of OFS source, create the top-level source directory and clone OFS repositories.
$ mkdir ofs-2024.2-1\n$ cd ofs-2024.2-1\n$ export OFS_BUILD_ROOT=$PWD\n$ git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n\nCloning into 'ofs-agx7-pcie-attach'...'\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\n\n$ cd ofs-agx7-pcie-attach\n$ git checkout tags/ofs-2024.2-1\nVerify that the correct tag/branch have been checked out
$ git describe --tags\n\n$ ofs-2024.2-1\nThe FIM Testbench is UVM compliant and integrates third party VIP's from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this TB. UVM RAL (Register Abstraction Layer) is used for CSR Verification.
The Qualified Verification IPs will help to detect incorrect protocol behavior easily, help to focus on FIM features and accelerate the verification process.
\u2022 VCS & DVE\n\u2022 SNPS-Assertions\n\u2022 Verdi\n\u2022 VerdiCoverage\n\u2022 VerdiSimDB\n\u2022 VerdiTransactionDebugUltra\n\u2022 VIP-AMBA-AXI-SVT\n\u2022 VIP-AMBA-STREAM-SVT\n\u2022 VIP-PCIE-SVT\n\u2022 VIP-PCIE-TS-SVT\n\u2022 VIP-PCIE-G3-OPT-SVT\n\u2022 VIP-Ethernet-SVT\nThe following tools are required for successful UVM set-up
The UVM tool set-up is best done by creating a simple set-up script so all applicable tools are sourced before running the tests.
The following environment variables can be pasted into a script and used prior to running the UVM verification environment
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#license-files","title":"License Files","text":"export LM_LICENSE_FILE=\nexport SNPSLMD_LICENSE_FILE=\nThe license environment variables LM_LICENSE_FILE and SNPSLMD_LICENSE_FILE can point to a server license on your system.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#general-environment-variables","title":"General Environment Variables","text":"export IOFS_BUILD_ROOT=$PWD\nexport OFS_ROOTDIR=<user_path>/ofs-agx7-pcie-attach\nexport WORKDIR=$OFS_ROOTDIR\nexport QUARTUS_HOME=<user_path>/intelFPGA_pro/24.1/quartus\nexport QUARTUS_ROOTDIR=$QUARTUS_HOME\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\nexport DESIGNWARE_HOME=<user_path>/synopsys/vip_common/vip_Q-2020.03A\nexport PATH=$DESIGNWARE_HOME/bin:$PATH\nexport UVM_HOME=<user_path>/synopsys/vcsmx/S-2023.03-SP2-1/linux64/rhel/etc/uvm\nexport VCS_HOME=<user_path>/synopsys/vcsmx/S-2023.03-SP2-1/linux64/rhel\nexport PATH=$VCS_HOME/bin:$PATH\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\nThe default simulator used in the simulation script is Synopsys VCS-MX. Users can refer to the options and adopt the options for other simulators. The script is a makefile that calls vlogan, vcs and simv for compilation, elaboration and simulation, respectively.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#62-file-structure","title":"6.2 File Structure","text":"After cloning the repo, the verification and ofs-common directories contain all UVM verification related files. The directory structure is shown in Figure 4 below.
Figure 4 UVM Verification Directory File Structure
ofs-agx7-pcie-attach/verification/testbench has a testbench, uvm env, virtual sequencer, RAL etc.
ofs-agx7-pcie-attach/tests contains all uvm tests and sequences.
Users can run the simulation under \"ofs-agx7-pcie-attach/verification/scripts\" directory and the simulation result is outputted to a \"sim\" directory for Synopsys VCS.
The simulation result folder is named after the test name with increasing suffix number. If user runs the same test multiple times, the suffix is incremented by 1 each time.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#63-uvm-test-suite","title":"6.3 UVM Test Suite","text":"The UVM environment contains a variety of tests that have been developed to test out the FIM portion of OFS.
The table below has four columns which describe the \"Test Name\", \"DUT Scope\", \"Test Scenario\" and the \"Checking Criteria\".
Tests are located at ofs-agx7-pcie-attach/verification/tests
Test Name DUT Scope Test Scenario Checking Criteria afu_mmio_flr_pf0_test PF0 FLR Reset Apply FLR Reset for PF0 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF0 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf0_test PF0_VF0_FLR Reset Apply FLR Reset for PF0_VF0 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf1_test PF0_VF1_FLR Reset Apply FLR Reset for PF0_VF1 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf2_test PF0_VF2_FLR Reset Apply FLR Reset for PF0_VF2 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf2_test PF2 FLR Reset Apply FLR Reset for PF2 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF2 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf3_test PF3 FLR Reset Apply FLR Reset for PF3 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF3 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf4_test PF4 FLR Reset Apply FLR Reset for PF4 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF4 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_stress_5bit_tag_test AFU-Stress To check the AFU Stress by sending traffic from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_8bit_tag_test AFU-Stress To check the AFU Stress by sending traffic from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_test AFU-Stress 1. Initiate transactions to all the supported PF/VF from PCIE VIP and ensure that traffic is sent to all blocks of the AFU. 2. Ensure that CE/HE-LB/HE-MEM/HSSI/BPF/FME are seeing traffic. 3. Ensure that HE-LB/HE-MEM/CE sends DMWR/DMRD requests to PCIE VIP. 4. Ensure the Mux/DeMux blocks is able to handle the traffic based on the PF's/VF's and proper muxing/demuxing happens. Data checking bar_32b_test PCIe MMIO Path Set the BAR values to 32bit and test mmio access BAR address, Register Base Offset bar_64b_test PCIe MMIO Path Set the BAR values to 64bit and test mmio access BAR address, Register Base Offset dfh_walking_test DFH DFH walking offset checking, eol checking -> tb emif_csr_test EMIF CSR access data checking fme_csr_test FME CSR CSR accesses data checking fme_err_intr_test Interrupt FME Interrupt request using FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_intr_test Interrupt FME interrupt request using RAS ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_ras_cat_fat_err_test FME Error FME This test verifies the RAS fatal error register in FME. RAS error is generated by forcing/writing into the error register. MSI Generation is verified via PBA mechanism by masking and unmasking interrupt vector Error Checking fme_ras_no_fat_err_test FME Error This test verifies the RAS no-fatal error register in FME. RAS error is generated by forcing/writing into the error register. MSI Generation is verified via PBA mechanism by masking and unmasking interrupt vector Error Checking fme_multi_err_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_hssi_csr_test HE-HSSI CSR accesses data checking he_hssi_rx_lpbk_25G_10G_test HE-HSSI Sending back to back ethernet data traffic with 25G speed on HSSI RX Port0-7 lanes using Ethernet VIPs Enable the loopback mode in HE-HSSI and compare the pkts recived on HSSI TX Port(DATA CHECKING) he_hssi_tx_lpbk_P0_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port0 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P1_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port1 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P2_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port2 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P3_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port3 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P4_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port4 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P5_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port5 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P6_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port6 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P7_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port7 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_lpbk_cont_test HE-LPBK Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_lpbk_csr_test HE-LPBK CSR accesses data checking he_lpbk_flr_rst_test HE-LPBK Set HE_LPK in all the modes randomly and iterate the transactions in loop. At the end of every loop assert the Soft reset in the middle of the transactions data checking, counter checking he_lpbk_long_rst_test HE-LPBK Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_lpbk_long_test HE-LPBK Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_lpbk_multi_user_intr_test HE-LPBK generate user HE_LB interrupt interrupt checking he_lpbk_rd_cont_test HE-LPBK Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_rd_test HE-LPBK Read only mode. Randomize num_lines, addresses, req_len counter checking he_lpbk_reqlen16_test HE-LPBK To check the behavior of HE_LPK block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_lpbk_reqlen1_test HE-LPBK Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_lpbk_reqlen2_test HE-LPBK Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_lpbk_reqlen4_test HE-LPBK Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_lpbk_reqlen8_test HE-LPBK Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_lpbk_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_thruput_contmode_test HE-LPBK Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking he_lpbk_thruput_test HE-LPBK generate user HE_LB interrupt counter checking he_lpbk_user_intr_test HE-LPBK Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_lpbk_wr_cont_test HE-LPBK Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_wr_test HE-LPBK Write only mode. Randomize num_lines, addresses, req_len counter checking he_mem_cont_test HE-MEM Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking, counter checking he_mem_csr_test HE-MEM CSR accesses data checking he_mem_lpbk_long_rst_test HE-MEM Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_mem_lpbk_long_test HE-MEM Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_mem_lpbk_reqlen16_test HE-MEM To check the behavior of HE_MEM block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_mem_lpbk_reqlen1_test HE-MEM Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_mem_lpbk_reqlen2_test HE-MEM Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_mem_lpbk_reqlen4_test HE-MEM Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_mem_lpbk_reqlen8_test HE-MEM Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_mem_lpbk_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_multi_user_intr_test Interrupt Back to back multiple User interrupt request from HE MEM Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read for multiple back to back request he_mem_rd_cont_test HE-MEM Read only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_test HE-MEM Read only mode. Randomize num_lines, addresses, req_len counter checking he_mem_thruput_contmode_directed_test HE-MEM Set HE_LPK in thruput mode and send traffic with req len 1 and num_lines set to 40 data checking, counter checking he_mem_thruput_contmode_test HE-MEM Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_thruput_test HE-MEM Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_mem_user_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_mem_wr_cont_test HE-MEM Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_wr_test HE-MEM Write only mode. Randomize num_lines, addresses, req_len counter checkingt he_mem_flr_rst_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len Read the Control and Status registers before FLR Reset. Apply PF0VF0 FLR reset and check again the control and status registers. Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from HE-MEM he_random_test All HEs Enable all HEs and randomize modes data checking if in lpbk mode, counter checking helb_rd_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Read only mode Measure the performance helb_rd_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Read only mode Measure the performance helb_rd_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Read only mode Measure the performance helb_thruput_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_4cl_5bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_8bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Thruput mode Measure the performance helb_wr_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Write only mode Measure the performance helb_wr_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Write only mode Measure the performance helb_wr_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Write only mode Measure the performance hssi_ss_test HSSI CSR accesses data checking malformedtlp_pcie_rst_test Protocol Checker generate malformed TLP protocol error and initiate pcie reset to clear the error Check for error malformedtlp_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MaxTagError_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transactions are completing. mem_tg_csr_test MEM-TG CSR access data checking mem_tg_traffic_gen_test MEM-TG This test checks the MEM_TG traffic generator flow for 1 bank data checking mini_smoke_test All HEs shorter simpler version of random test for turn-in sanity check data checking if in lpbk mode, counter checking mix_intr_test Interrupt Mix interrupt testcase to send multiple FME and User interrupt request simultaneously Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests through different sources - FME and HE-MEM modules mmio_pcie_mrrs_128B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_128B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Read Checking valid possible combination of MPS & MRRS mmio_stress_nonblocking_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux with non-blocking MMIO reads data checking mmio_stress_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux data checking mmio_test PCIe MMIO Path MMIO targeting PF0(ST2MM, FME, PMCI, QSFP, HSSI SS), PF1, PF2,PF3, PF4, PF1.VF1, PF1.VF2 data checking mmio_unimp_test PCIe MMIO Path MMIO acccess to unimplemented addresses data checking MMIODataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOTimedout_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. pcie_csr_test PCIESS Earlier msix registers were in fme block but now it has moved from fme to pciess. Hence coded a seperate test for msix data checking pcie_pmci_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pcie_pmci_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM (Vendor Defined Message) single packet received from PCIe HIA subsystem to PMCI controller over APF and BPF via AXI4-lite memory write request pmci_csr_test PMCI CSR CSR accesses data checking pmci_fme_csr_test PMCI FME CSR CSR accesses data checking pmci_pcie_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pcie_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM single packet received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pciess_csr_test PMCI PCIESS CSR CSR accesses data checking pmci_qsfp_csr_test PMCI QSFP CSR CSR accesses data checking port_gasket_csr_test PORT GASKET CSR accesses data checking qsfp_csr_test QSFP CSR accesses data checking TxMWrDataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. TxMWrInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. uart_intr_test UART Checking Generates UART interrupt Check interrupt UnexpMMIORspErr_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. protocol_checker_csr_test Protocol Checker CSR access to Protocol Checker data checking vdm_err_vid_test Vendor ID check generate a packet with undefined Vendor-ID from Host to PMCI_SS ID checkThe next section describes how to compile and build the UVM environment prior to running each UVM test and analyinsg the results in the log files
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#64-ip-compile","title":"6.4 IP Compile","text":"To compile all IPs for the Synopsys VCS simulater targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk cmplib_adp\nTo compile all IPs for the Synopsys VCS simulater targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk cmplib_adp FTILE_SIM=1\nThe RTL file list for compilation is located here: verification/scripts/rtl_comb.f
The TB file list for compilation is located here: verification/scripts/ver_list.f
To compile RTL and Testbench for the Synopsys VCS simulater targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_adp DUMP=1\nTo compile RTL and Testbench for the Synopsys VCS simulater targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_adp FTILE_SIM=1\nIf the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS targetting the Intel\u00ae FPGA SmartNIC N6001-PL then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_all\nIf the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_all FTILE_SIM=1\nTo run a simulation for Synopsys VCS targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk run TESTNAME=ofs_mmio_test\nTo run a simulation for Synopsys VCS targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk run TESTNAME=ofs_mmio_test FTILE_SIM=1\nTo dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
gmake -f Makefile_VCS.mk build_adp DUMP=1\n\n gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> DUMP=1\nOr
gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> DUMP=1\nTo dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
gmake -f Makefile_VCS.mk build_adp FTILE_SIM=1 DUMP=1\n\n gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> FTILE_SIM=1 DUMP=1\nOr
gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> FTILE_SIM=1 DUMP=1\nThere are some optimizations in the Table below for convenience if you want to bypass some commands for both Synopsys VCS:
Command (Synopsys VCS) for n6001 * Command (Synopsys VCS) for fseries-dk Details gmake -f Makefile_VCS.mk build_all DUMP=1 gmake -f Makefile_VCS.mk build_all FTILE_SIM=1 DUMP=1 compile IP + compile RTL gmake -f Makefile_VCS.mk build_run TESTNAME= DUMP=1 gmake -f Makefile_VCS.mk build_run TESTNAME= FTILE_SIM=1 DUMP=1 compile RTL + run test gmake -f Makefile_VCS.mk do_it_all TESTNAME= DUMP=1 gmake -f Makefile_VCS.mk do_it_all TESTNAME= FTILE_SIM=1 DUMP=1 compile IP, RTL and run test gmake -f Makefile_VCS.mk rundb TESTNAME= DUMP=1 gmake -f Makefile_VCS.mk rundb TESTNAME= FTILE_SIM=1 DUMP=1 run test in sim dir + over-writes contentNote: * For N6000, use flag DISABLE_EMIF=1 for the tests to run successfully
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#67-uvm-regression-test","title":"6.7 UVM Regression Test","text":"If the user wants to run the complete set of UVM tests in one command for VCS targetting the Intel\u00ae FPGA SmartNIC N6001-PL then follow the procedure below
cd $VERDIR/scripts\n\npython uvm_regress.py -l -n 8 -p adp -k <pkg_name> -s vcs -c none\nIf the user wants to run the complete set of UVM tests in one command for VCS targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) then follow the procedure below
python uvm_regress.py -l -n 8 -p adp -k <pkg_name> -s vcs -c none -e -t ftile\nTest Packages for Intel\u00ae FPGA SmartNIC N6001-PL and Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) are listed below:
<pkg_name> :top_pkg (it contains two packages test_pkg + test_long_pkg)\n :test_pkg\n :test_long_pkg\n :tx_pkg\n :tx_pkg_100G_200G\n :tx_pkg_400G\n :rx_pkg\n :rx_pkg_100G\n :rx_pkg_400G\nTest Package files are located in tests directory ($VERDIR/tests) please go through the list of testcases that package file contains before running the regression test
Results are created in a sim directory ($VERDIR/sim) with individual testcase log dir
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#68-uvm-waveform-and-transcript-analysis","title":"6.8 UVM Waveform and Transcript Analysis","text":"Running Synopsys VCS UVM tests will generate a ofs-agx7-pcie-attach/verification/sim directory
\u2022 All build time logs are located at ofs-agx7-pcie-attach/verification/sim\n\n\u2022 Each testcase will have separate directory inside sim ofs-agx7-pcie-attach/verification/sim/<test_case_name>\nThere are two tracker or log files that are available: runsim.log and trans.log.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#runsimlog","title":"runsim.log","text":"runsim.log is the simulation log file generated from Synopsys VCS. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 5
Figure 5 runsim.log
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#translog","title":"trans.log","text":"trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 6
Figure 6 trans.log
The waveform generated is named as \"inter.vpd\". To open the waveform, go to simulation result directory and run
dve -full64 -vpd inter.vpd &\nFor Intel\u00ae FPGA SmartNIC N6001-PL
The following command allows to run a single testcase with coverage enabled
gmake -f Makefile_VCS.mk cmplib_adp && gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1 COV_FUNCTIONAL=1&& gmake -f Makefile_VCS.mk run TESTNAME=<TESTCASE-NAME> DUMP=1 DEBUG=1 COV_FUNCTIONAL=1 &\nThe following command shows how to merge and generate the coverage report
urg -dir <$VERDIR/sim/simv.vdb> <$VERDIR/sim/regression.vdb> -format both -dbname <regression_database_name>\nThis will generate both urgreport directory and .vdb file Multiple regression.vdb from different regressions can be merged with the same command.
e.g \"urg -dir <path1_till_simv.vdb> <path1_till_regression.vdb> <path2_till_regression.vdb> -report <dir> -format both -dbname <dirname>\"\nThe following commands shows how to launch DVE and check the coverage reports
To open DVE of a single regression or testcase, execute:\n\n dve -full64 -cov -covdir simv.vdb regression.vdb &\n\nTo open DVE of a merged regression test, execute:\n\n dve -full64 -cov -covdir <dirname.vdb> &\nAgilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
The following command allows to run a single testcase with coverage enabled
gmake -f Makefile_VCS.mk cmplib_adp FTILE_SIM=1 && gmake -f Makefile_VCS.mk build_adp FTILE_SIM=1 DUMP=1 DEBUG=1 COV_FUNCTIONAL=1&& gmake -f Makefile_VCS.mk run TESTNAME=<TESTCASE-NAME> FTILE_SIM=1 DUMP=1 DEBUG=1 COV_FUNCTIONAL=1 &\nThe following command shows how to merge and generate the coverage report
urg -dir <$VERDIR/sim/simv.vdb> <$VERDIR/sim/regression.vdb> -format both -dbname <regression_database_name>\nThis will generate both urgreport directory and .vdb file Multiple regression.vdb from different regressions can be merged with the same command.
e.g \"urg -dir <path1_till_simv.vdb> <path1_till_regression.vdb> <path2_till_regression.vdb> -report <dir> -format both -dbname <dirname>\"\nThe following commands shows how to launch DVE and check the coverage reports
To open DVE of a single regression or testcase, execute:\n\n dve -full64 -cov -covdir simv.vdb regression.vdb &\n\nTo open DVE of a merged regression test, execute:\n\n dve -full64 -cov -covdir <dirname.vdb> &\nThe UVM Register Layer provides a standard base class library that enable users to implement the object-oriented model to access the DUT registers and memories. The UVM Register Layer is also referred to as UVM Register Abstraction Layer (UVM RAL). Design registers can be accessed independently of the physical bus interface. i.e. by calling read/write methods. This is shown in Figure 9 below.
Figure 9 RAL UVM Testbench
The RAL register models for different CSR's mimics the design registers. All RAL files are located here.
ofs-agx7-pcie-attach/verification/testbench/ral\nThe RAL model is generated through the Synopsys RALGEN tool and is used for CSR verification.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#72-ral-integration","title":"7.2 RAL Integration","text":"For UVM RAL model integration to the environment, adapters for each CSR is implemented and integrated into the Testbench Environment. It is used to convert the PCIe bus sequence items into uvm_reg_bus_op and vice versa. The CSR test cases pick up all the registers from the respective CSR blocks and perform a default value, wr/rd check on them.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#73-ral-model-generation","title":"7.3 RAL Model Generation","text":"Steps for RAL model generation
Excel(xls) file containing the registers is required. Make sure there are separate xls files for each CSR and the worksheet name must contain the name \"reg_fields\".
Excel sheet snapshot example below for EMIF_CSR.xls located at /ipss/mem/rtl/adp
\u2022 Navigate to ofs-agx7-pcie-attach/ofs-common/verification/common/scripts/ral\n\u2022 Copy the excel file (xls) to the above area\n\u2022 In the bash terminal run mk_ral.sh <Excel sheet name without extension > <output *.sv file name without ral_ prepended >\n\u2022 The above steps generate two ral *.sv files. File with _cov suffix is a coverage enabled ral model. \n\u2022 Copy *.sv files to ofs-agx7-pcie-attach/verification/testbench/ral\n\u2022 As an example to generate ral_ac_ce.sv from AC_CE_CSR.xls file the command is\n\n mk_ral.sh AC_CE_CSR ac_ce\nThis generates two ral models (ral_ac_ce.sv and ral_ac_ce_cov.sv)
To add new registers
\u2022 To create new registers, copy existing ones and modify the cells in the xls. Make sure the last line is also a copied blank line\n\u2022 Follow all the steps of RAL model generation\nTo Generate a RAL model when a new xls sheet is created for a new component
\u2022 Copy the relevant xls sheet to ofs-agx7-pcie-attach/ofs-common/verification/common/scripts/ral\n\u2022 Follow all the steps of RAL model generation\nThe testbench components for RAL are defined below
\u2022 ral_reg_iofs_* (uvm_reg) generated by the steps as mentioned in section 5.3\nThe uvm register class is written by extending the uvm_reg. A register represents a set of fields that are accessible as a single entity Each register contains any number of fields, which mirror the values of the corresponding elements in hardware
\u2022 ral_block_iofs_* (uvm_block) generated in the same register file\nA register model is an instance of a register block, which may contain any number of registers, register files, memories, and other blocks
\u2022 ral_block_ofs (uvm_block) \u2013 Contains all the CSR block instantiations\n\u2022 Reg2vip_*_adapter (uvm_reg_adapter) \u2013 This class defines an interface for converting between uvm_reg_bus_op and a specific bus transaction. For each CSR a respective adapter is present\nAll the components are defined in ofs-agx7-pcie-attach/ofs-common/verification/testbench
Integration of components in testbench
\u2022 The RAL blocks and adapters for each CSR is instantiated in tb_env\n\u2022 The bar range for each CSR is also set in the tb_env\nSample Environment Integration snippets
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#8-modifying-uvm-testbench","title":"8 Modifying UVM Testbench","text":"The next sections describe what needs to be considered when modifying the UVM, adding a new interface to the testbench and creating a new UVM test for a customised OFS Accelerator platform.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#81-modifying-uvm-environment-for-new-shell-variant","title":"8.1 Modifying UVM environment for new Shell Variant","text":"OFS n6001 comprises a shell based on PCIe Gen4x16 and is named base_x16
This base_x16 shell is described by an RTL file list, IP File lists and setup scripts to complete the build flow
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#82-modifying-uvm-environment-and-setting-up-compile-and-run-flow","title":"8.2 Modifying UVM environment and setting up Compile and Run flow","text":"All the variants can mostly reuse the existing UVM infrastructure to setup the build and run flow
\u2022 Create directory under $OFS_BUILD_ROOT new variant e.g ofs-n9000\n\u2022 Change directory to $OFS_BUILD_ROOT/ofs-n9000/verification/scripts\n\u2022 modify Makefile it to point to the new RTL, IP and script files.\nFollowing these three steps above will enable the build and sim flow to run the existing UVM TB and tests with new IOFS n9000 variant.
Adding a new interface requires signal connections in the testbench. An additional BFM or verification IP is needed to drive the new interface. The main testbench file tb_top.sv is found at the following location
ofs-agx7-pcie-attach/verification/testbench\nIn case the design has many register changes and the user decides to generate all the new RAL models instead of reusing from existing base RAL models, the following steps will help to create and integrate a new RALDIR in the UVM environment.
\u2022 Generate the new RAL files in desired directory. Preferably under the \"ofs-agx7-pcie-attach/verification/common/testbench\" \n\u2022 By default, the Makefile points to base FIM RAL so set the RALDIR path in the Makefile to the new generated RAL file directory\nCreate a define for the new variant. e.g 'define FIM_NEW. If you are modifying common files then add new logic under this define so other projects will not get affected with variant specific change.
If there are more changes, please create separate \"testbench\" and \"test\" folders under this new variant.
\u2022 Extend variant specific env file from base env file to add variant specific changes.\n\u2022 Create new test/seq lib files in \"tests\" folder.\n\u2022 Create new variant package, add new TB/Tests/Sequence lib files and also import the base package files.\nIf you are adding new files then make sure it's included in Makefile for the build+run flow.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#85-uvm-pcie-drivers","title":"8.5 UVM PCIe Drivers","text":"The \"svt_pcie_driver_app_transaction_base_sequence\" is part of Synopsys PCIe VIP library. You can find the sequence definition in the following two directories
\u2022 Navigate to \"$DESIGNWARE_HOME/vip/svt/pcie_svt/Q-2020.03/sverilog/src/vcs/svt_pcie_driver_app_transaction_sequence_collection.svp\" file. All the base and PCIe sequences are defined in this file.\n\n\u2022 When the OFS UVM build command is executed, it creates \"vip\" directory under \"$OFS_BUILD_ROOT/ofs-agx7-pcie-attach/verification\". You can also find the same sequence file at \"$IOFS_BUILD_ROOT/ofs-agx7-pcie-attach/verification/vip/pcie_vip/src/sverilog/vcs/svt_pcie_driver_app_transaction_sequence_collection.sv\"\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/","title":"Software Developer Journey Guide: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#1-introduction","title":"1 Introduction","text":"This document is intended to help you understand the FPGA Software developer flow using OFS as well as considerations you should take when creating custom software.
After reviewing the document, you shall be able to:
The general development flow is depicted in the diagram below and discussed in more detail in each section of this document.
Figure 1-1: SW Developer Flow
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#2-evaluate-ofs","title":"2 Evaluate OFS","text":"The repositories in the OFS site are tagged based on year and release number. For example, a tag of 2024.2-1 indicates the first release package of the quarter. If there are updates to this release package, the last number will be incremented by 1, such as 2024.2-2. Not all repositories follow the exact same naming scheme; both the OPAE SDK user space software and Linux DFL Backport kernel driver repository tags differ from their hardware counterparts. See the below table for a summary of the most recent release versions for each software component:
Table 1: Combined OPAE and Platform Version Summary
The OPAE SDK repository is located at https://github.com/OFS/opae-sdk.
Component Platform Most Recent Release OPAE SDK Intel\u00ae FPGA SmartNIC N6000-PL 2.13.0-3 OPAE SDK Intel\u00ae FPGA SmartNIC N6001-PL 2.13.0-3 OPAE SDK Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 2.13.0-3 OPAE SDK Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) 2.13.0-3 OPAE SDK Intel\u00ae FPGA PAC D5005 2.12.0-5 OPAE SDK IPU Platform F2000X-PL 2.12.0-5Table 2: Combined DFL and Platform Version Summary
The Linux DFL repository is located at https://github.com/OFS/linux-dfl, the backport repository is located at https://github.com/OFS/linux-dfl-backport.
Component Platform Most Recent Release Linux DFL Intel\u00ae FPGA SmartNIC N6000-PL intel-1.11.0-2 Linux DFL Intel\u00ae FPGA SmartNIC N6001-PL intel-1.11.0-2 Linux DFL Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) intel-1.11.0-2 Linux DFL Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) intel-1.11.0-2 Linux DFL Intel\u00ae FPGA PAC D5005 ofs-2024.1-6.1-2 Linux DFL IPU Platform F2000X-PL ofs-2024.1-6.1-2Not all OFS releases will support all OFS enabled hardware platforms. Review the official OFS release pages at https://github.com/OFS/ofs-agx7-pcie-attach/releases for PCIe Attach and https://github.com/OFS/ofs-f2000x-pl/releases for SoC Attach.
By clicking on the release link to the right side of the GitHub web page in either of the software repositories, you will find the latest release, the tag number and release notes.
Figure 2-1: OFS Repository Release Page Link
By scrolling to the end of the release page, you will find assets attached to the release that you can leverage for quick evaluation of OFS, such as FPGA binary files, POFs and SOFs, sample AFUs, and pre-compiled binaries for the OFS software stack.
Before evaluating an OFS product, you will need to choose a platform and associated software release that supports it. Each OFS enabled platform has an associated Getting Started Guide, alongside the Software Installation Guidelines and Board Installation Guidelines. The Getting Started Guide will provide a high level overview of the entire setup process, and guide you to the other documents as needed.
Set up your card and software in a server using the steps provided in one of the corresponding Getting Started Guides and leverage the appended binaries in the official OFS release repository release page under the \"Assets\" tab to preview the software and design functionality the OFS framework out of the box. Getting Started Guides are available for the following FIM(shell) designs:
Every OFS release has been validated against a specific OS release. This information, alongside any recommended BIOS settings changes, can be found in the Getting Started Guides.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#3-select-a-starting-shellfim","title":"3 Select a Starting Shell/FIM","text":"To begin your development, start with an existing shell that most closely matches your device and end solution. The OFS site has both Intel Intel Agilex 7 and Stratix 10 FPGA reference designs you can use as a starting point for your own design. These designs can be ported to different device OPNs if a different resource utilization is required, although that will not be covered here.
To begin you may want to learn more about Intel Stratix 10 and Intel Agilex 7 family offerings. Please refer to the following links:
In this document, we assume you have downloaded and configured your device with the appropriate pre-compiled binary image for your chosen OFS release and platform without any further customization. For more information on your OFS Shell design and its available key features go to https://ofs.github.io/latest, choose your shell design on the top navigation bar, and select the Shell Technical Reference Manual and Shell Developer Guide. You can also review the FPGA Developer Journey Guide: Open FPGA Stack document.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#4-software-documentation","title":"4 Software Documentation","text":"The Software Reference Manual: Open FPGA Stack provides a complete picture of the OPAE SDK code API alongside concepts you will need to know when developing user space application and driver code. OPAE provides API bindings in C, C++, and Python, although not all functions are supported across languages. Section 13.0 Building a Sample Application provides example application code and build steps that can be used as a quick reference.
You also have the OPAE FPGA Tools available to use for deployment and debug, provided you have set up your environment to work with OFS platforms. These tools can monitor device status, program new PR or statically compiled images, probe CSR space and run basic peripheral exercisers to validate functionality. They can also be used to quickly bind / unbind your device from specific drivers. A full description of the capabilities of these commands can be found on the OFS Site, with shorter descriptions provided in the Getting Started User Guides.
The OPAE SDK repository contains all of the source code for plugins needed for each supported OFS platform. Review section 2.3 Plugin Manager of the Software Reference Manual for further reading on the topic of plugins and where they fit into the OFS stack architecture.
An older documentation repository that stopped being the primary API resource after OPAE 2.3.0 can be found at https://opae.github.io/latest/index.html. The information in this repo should be considered out of date unless using a legacy version of the OPAE SDK software.
The Linux DFL Wiki page provides and introduction to the topics and design philosophy that goes into DFL enabled driver creation. DFL driver documentation has also been upstreamed into the Linux kernel and can be found under Documentation/fpga/dfl.rst.
The below picture demonstrates the structure of the OFS stack stack at a high level:
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#5-setup-your-environment-for-development","title":"5 Setup Your Environment For Development","text":"Before beginning development you will need to ensure you have OFS setup requirements satisfied by reviewing instructions in the corresponding Software Installation Guidelines.
For any additional software you plan on using in conjunction with OPAE and DFL - for example, DPDK and IPDK - refer to that software solution's installation guidelines, and make sure your environment has been properly initialized.
All OFS repositories are open source and do not require permissions to clone.
The primary component of the OFS software stack that must be installed before developing any application is the OPAE SDK. Additionally, if you plan on testing your applications on hardware then the DFL driver set must also be installed. All Getting Started User Guides will point you to the correct location for install steps relevant to a given platform.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#6-pick-a-reference","title":"6 Pick a Reference","text":"Before beginning development of a new driver or user space software application, it can help to find an already existing piece of code that can help guide your development.
If looking to write user space application code that uses the OPAE SDK API, we provide a series of sample applications under the /opae/samples directory. These cover a wide range of topics, including taking advantage of CXL, built-in host exercisers, HSSI, and AFU and FIM event checking. If you have never written an application before you should review hello_afu and its included README.
Driver code samples vary widely depending on their implementation and the needs of the associated IP, sometimes being split into multiple source files. The 8250 DFL UART IP Driver represents a minimal driver implementation, and clearly demonstrates where IP specific implementation ends and DFL framework integration begins. This driver calls into the existing UART driver framework, while providing the bus \"glue\" logic for core DFL functionality (specifically, the line dfluart->line = serial8250_register_8250_port(&uart); is where the driver calls into the existing Linux kernel 8250 driver framework).
A second driver example you can review is the DFL specific implementation of a generic SPI driver. Where again the line err = devm_spi_register_controller(dev, host); is used to register this device with the SPI framework provided by the kernel, while also calling upon DFL framework specific constructs such as the FEATURE_ID and FME_ID to properly identify the device based on its DFH.
Building an OPAE application involves linking your source code with the OPAE libraries. A minimal build command using gcc is provided below as a reference for an application using the C API. The API uses some features from the C99 language standard. The -std=c99 switch is required if the compiler does not support C99 by default.
This library internally uses libuuid and libjson-c. But they are not distributed by it. Make sure you have these libraries properly installed. The -c flag may not be necessary depending on the platform being used.
gcc -std=c99 hello_fpga.c -I/usr/local/include -L/usr/local/lib -lopae-c -luuid -ljson-c -lpthread -o hello_fpga\nThe OPAE SDK has a built-in debug logging facility. To enable it, set the cmake flag -DCMAKE_BUILD_TYPE=Debug and then use the following environment variables:
To enable gdb-based debugging, the cmake configuration step must specify a value for -DCMAKE_BUILD_TYPE of either Debug or RelWithDebInfo so that debug symbols are included in the output binaries. The OPAE SDK makes use of dynamically-loaded library modules. When debugging with gdb, the best practice is to remove all OPAE SDK libraries from the system installation paths to ensure that library modules are only loaded from the local build tree:
$ cd opae-sdk/build\n$ LD_LIBRARY_PATH=$PWD/lib gdb --args <some_opae_executable> <args>\nDeployment of an AFU or statically compiled FIM+AFU image is detailed in the associated Getting Started User Guide for a given platform. If you wish to deploy your application into a VM or docker image you can review the Virtual machine User Guide: Open FPGA Stack + KVM and the Docker user guide, located under the Virtualization header of your chosen platform.
When testing a custom written user space application, you can use the following procedure:
fpgasupdate, or JTAG flowfpgainfoopae.io ls 4.a. If not, you can swap between dfl-pci and vfio-pci with opae.io bind/unbindIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/","title":"AFU Developer Guide: OFS for Stratix\u00ae 10 FPGA PCIe Attach FPGAs","text":""},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#1-introduction","title":"1. Introduction","text":"This document is a design guide for creating an Accelerator Functional Unit (AFU) using Open FPGA Stack (OFS) for Stratix\u00ae 10 FPGA. The AFU concept consists of separating the FPGA design development process into two parts, the FIM and AFU, as shown in the diagram below:
This diagram shows the FPGA board interface development separation from the internal FPGA workload creation. This separation starts with the FPGA Interface Manager (FIM), which consists of the external interfaces and board management functions. The FIM is the base system layer typically provided by board vendors. The FIM interface is specific to a particular physical platform. The AFU uses the external interfaces with user-defined logic to perform a specific application. Separating the lengthy and complicated process of developing and integrating external interfaces for an FPGA into a board allows the AFU developer to focus on their workload needs. OFS for Stratix\u00ae 10 FPGA provides the following tools for rapid AFU development:
Please notice that the AFU region consists of both static and PR logic in the above block diagram. Creating AFU logic for the static region is described in Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs. This guide covers logic in the AFU Main (PR) region.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#11-document-organization","title":"1.1 Document Organization","text":"This document is organized as follows:
This guide provides theory followed by tutorial steps to solidify your AFU development knowledge.
This guide uses the Intel\u00ae FPGA PAC D5005 as the platform for all tutorial steps. Additionally, this guide and the tutorial steps can be used with other platforms; However, please consult the board and FIM supplier of other platforms for specific instructions on the use of custom FIM to develop AFU design.
If you have worked with previous Programmable Acceleration products, you will find OFS for Stratix\u00ae 10 FPGA is similar; however, there are differences, and you are advised to carefully read and follow the tutorial steps to understand the design tools and flow fully.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#12-prerequisite","title":"1.2 Prerequisite","text":"This guide assumes you understand the following FPGA logic design-related knowledge and skills:
To run the tutorial steps in this guide requires this development environment:
Item Version Operating System RHEL 8.6 Python 3.6.8 cmake 3.15 GCC 8.5.0 git 1.8.3.1 perl 5.8.8Verify your development has the above tools installed.
The following server and PAC card are required to run the examples in this guide:
OFS Stack provides a rapid design methodology for creating complex FPGA applications. In addition, you are provided with the following:
For any non-Altera\u00ae platform, contact your board vendor for the above components specific to the platform.
To start with AFU development, the first step should be to understand your platform capabilities. For example, what interface is the FPGA connected to the Host machine over PCI-E, if it is AXI like the Stratix\u00ae 10 FPGA Platform, or CCIP or CXL. Does the platform provide an External Memory Interface or the HSSI interface? Once you know what the platform offers, you can develop your AFU requirements and architecture as the next step.
This document will cover example AFU architecture and things that will help build AFU for Stratix\u00ae 10 FPGA reference platform and others coming in the future. In addition, this knowledge can be relatively applied for AFU development on other vendor-provided platforms.
The figure below shows a typical AFU development process independent of the platform used.
flowchart TB;\n A[Understand platform capabilities with OFS]-->B[Review AFU requirements and code samples provided];\n B[Review AFU requirements and code samples provided]-->C[Define AFU architecture];\n C[Define AFU architecture]-->D[Design AFU hardware];\n D[Design AFU hardware]-->E[Develop AFU software to control hardware];\n E[Develop AFU software to control hardware]-->F{\"Simulate in AFU Simulation Enviroment (ASE)\"};\n F:::if -- Pass --> H[\"Compile AFU for synthesis, place & route and timing (uses Quartus)\"];\n H[\"Compile AFU for synthesis, place & route and timing (uses Quartus)\"] --> I[\"Analyze Quartus Compile reports\"];\n I --> J{\"Quartus reports clean? (e.g. timing closed)\"};\n J:::if -- No --> P2;\n J -- Yes --> K[\"Run/Validate design on OFS Platform\"];\n K --> L{\"Hardware validation pass?\"};\n L == Yes ==> M[\"AFU ready to deploy\"];\n L -- No --> N[\"Debug on hardware using traditional FPGA tools (e.g. SignalTab\"];\n N --> P2[\"Fix AFU design (e.g Design changes, timing closure constraints)\"];\n P2 --> O{\"Need functional validation?\"};\n O:::if -- Yes -->P[\"Fix AFU design (e.g Functional design changes, bug fixes)\"];\n O -- No -->H; \n F -- Fail --> P;\n P -->D; \n\n classDef default color:#fff,fill:#0071c5,stroke:#71c5,stroke-width:1px\n classDef if color:#0071c5,fill:#fff,stroke:#0071c5,stroke-width:2pxThe OFShigh-level data flow is shown below: The control and data are composed of the following:
Peripherals are connected using AXI or Avalon:
Peripherals are presented to software as:
The peripherals connected to the peripheral fabric are primarily OPAE managed resources, whereas the peripherals connected to the AFU are \"primarily\" driven by native OS drivers. The word \"primarily\" is used since the AFU is not mandated to expose all its peripherals to OPAE. Instead, it can be connected to the peripheral fabric but can choose to expose only a subset of its capability to OPAE.
OFS uses a defined set of CSRs to expose the functionality of the FPGA to the host software. These registers are described in Open FPGA Stack Reference Manual - MMIO Regions section.
If you make changes to the FIM that affect the software operation, OFS provides a mechanism to communicate that information to the proper software driver. The Device Feature Header (DFH) structure provides a mechanism to maintain compatibility with OPAE software. Please see FPGA Device Feature List (DFL) Framework Overview for an excellent description of DFL operation from the driver perspective.
When planning your address space for your FIM updates, please be aware OFS FIM targeting Intel\u00ae FPGA PAC D5005, 256KB of MMIO region is allocated for external FME features, and 128kB of MMIO region is given for external port features. Each external feature must implement a feature DFH, and the DFH needs to be placed at the 4KB boundary. The last feature in the external feature list must have the EOL bit in its DFH set to 1 to mark the end of the external feature list. Since the FPGA address space is limited, consider using an indirect addressing scheme to conserve address space.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#132-considerations-for-pim-usage","title":"1.3.2. Considerations for PIM Usage","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
The following resources are available to assist in creating an AFU:
PIM Core Concepts provides details on using the PIM and its capabilities.
PIM Based AFU Developer Guide provides details on interfacing your AFU to the FIM using the PIM.
The examples-afu repo provides several AFU examples:
Example Description PIM-based Hybrid Native clocks Example AFU using user configurable clocks. X copy_engine Example AFU moving data between host channel and a data engine. X dma Example AFU moving data between host channel and local memory with a DMA. X hello_world Example AFU sending \"Hello World!\" to host channel. X X X local_memory Example AFU reading and writing local memory. X XThese examples can be run with the current OFS FIM package. There are three AFU types of examples provided (PIM based, hybrid and native). Each example provides the following:
The figure below shows the interfaces available to the AFU in this architecture. It also shows the design hierarchy with module names from the FIM (top.sv) to the PR region AFU (afu_main.sv). One of the main differences from the previous Stratix\u00ae 10 FPGA OFS architecture is a static port gasket region (port_gasket.sv) that has components to facilitate the AFU and also consists of the GBS region (afu_main.sv) via the PR slot. The Port Gasket contains all the PR -specific modules and logic, e.g., PR slot reset/freeze control, user clock, remote STP etc. Architecturally, a Port Gasket can have multiple PR slots to which user workload can be programmed. However, only one PR slot is supported for OFS Release for Stratix\u00ae 10 FPGA. Therefore, everything in the Port Gasket until the PR slot should be provided by the FIM developer. The task of the AFU developer is to add their desired application in the afu_main.sv module by stripping out unwanted logic and instantiating the target accelerator. As shown in the figure below, here are the interfaces connected to the AFU (highlighted in green) via Intel\u00ae FPGA PAC D5005 FIM:
The FIM targets operation in the Intel\u00ae FPGA PAC D5005 card. The block diagram of the Intel\u00ae FPGA PAC D5005 is shown below:
The key Intel\u00ae FPGA PAC D5005 FPGA interfaces are:
The internal FPGA architecture is shown below:
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2-set-up-afu-development-environment","title":"2. Set Up AFU Development Environment","text":"This section covers:
Additionally, this section includes steps to demonstrate loading and running the host_chan_mmio example AFU in an Intel\u00ae FPGA PAC D5005 equipped Linux server.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#21-prepare-afu-development-environment","title":"2.1. Prepare AFU development environment","text":"Typical development and hardware test environments consist of a development server or workstation with installed FPGA development tools and a separate server installed with the target OFS-compatible FPGA PCIe card. The typical usage and flow of data between these two servers are shown below:
Please refer to Unit Level Simulation if you would like to make any simulation Unit Level Simulation.
Note that both development and hardware testing can be performed on the same server if desired.
This guide uses Intel\u00ae FPGA PAC D5005 as the target OFS-compatible FPGA PCIe card platform for demonstration steps. The Intel\u00ae FPGA PAC D5005 must be fully installed following Board Installation Guide: OFS for Acceleration Development Platforms. If using a different OFS FPGA PCIe card, contact your supplier for instructions on how to install and operate a user-developed AFU.
NOTE:
The following chapters assume you use the same server for development and Deployment (Run the FIM/AFU/SW over the Intel\u00ae FPGA PAC D5005):
Development: Modify the FIM/AFU/SW run simulation and compile the design (Generate the binaries). Deployment: Program the binaries under the Intel\u00ae FPGA PAC D5005 and exercise the Hardware and Sw with real hardware
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#211-installation-of-quartus-and-ofs","title":"2.1.1. Installation of Quartus and OFS","text":"Building AFU with OFS forStratix\u00ae 10 FPGA requires the build machine to have at least 64 GB of RAM.
The following is a summary of the steps to set up for AFU development:
Quartus\u00ae Prime Pro Edition version 23.4 is the currently verified version of Quartus\u00ae Prime Pro Edition 23.4 used for building the AFU images. The recommended Best Known Configuration (BKC) OFS Version 2024.1-1:
Item Version Quartus\u00ae Prime Pro Edition 23.4 Operating System RHEL 8.6 OPAE SDK 2.12.0-5 OFS Release ofs-2024.1-1 Python 3.6.8 cmake 3.15 GCC 8.5.0 git 1.8.3.1 perl 5.8.8"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2111-installation-of-quartus","title":"2.1.1.1 Installation of Quartus","text":"Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.6 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\nCreate the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
Download your required Quartus Prime Pro Linux version here.
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are No patches for this release.
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\nFor example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\nVerify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\nRetrieve OFS repositories:
The OFS FIM source code is included in the OFS GitHub repository. First, create a new directory to store the retrieved files as a clean starting point. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories.
Navigate to the location for storage of OFS source, create the top-level source directory, and clone OFS repositories.
mkdir ofs_fim_build_root\ncd ofs_fim_build_root\nexport OFS_BUILD_ROOT=$PWD\ngit clone --recurse-submodules https://github.com/OFS/ofs-d5005.git\nConsole Output:
Cloning into 'ofs-d5005' ...\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\nEdit your bashrc file ~/.bashrc to add the following lines:
export OFS_ROOTDIR=$OFS_BUILD_ROOT/ofs-d5005\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\ncd ofs-d5005\nSelect the latest OFS Release
git checkout tags/ofs-2024.1-1\nConsole Output: ```sh You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:
git checkout -b HEAD is now at 7e4dc70 ofs-2024.1-1"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2113-directory-structure-of-ofs","title":"2.1.1.3. Directory Structure of OFS","text":"
Verify the following directories in the $OFS_BUILD_ROOT directory with the following command.
cd $OFS_ROOTDIR\nls\nConsole Output:
eval_scripts ipss ofs-common license LICENSE.md README.md sim src syn verification\nThe directories are arranged as shown below:
\u251c\u2500\u2500 eval_scripts\n\u2502 \u251c\u2500\u2500 ofs_d5005_eval.sh\n\u2502 \u251c\u2500\u2500 README_ofs_d5005_eval.txt\n|\n\u251c\u2500\u2500 ofs-common\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 src\n\u2502 \u251c\u2500\u2500 verification\n| \u251c\u2500\u2500 LICENSE.txt \u2502 \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 ipss **Directory ipss consists of Platform Designer subsystems used in FIM**\n\u2502 \u251c\u2500\u2500 hssi\n\u2502 \u251c\u2500\u2500 mem\n\u2502 \u251c\u2500\u2500 pcie\n| \u251c\u2500\u2500 pmic | \u251c\u2500\u2500 spi \u2502 \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 license\n\u2502 \u2514\u2500\u2500 quartus-0.0-0.01iofs-linux.run ** Quartus Patch with IP licenses. \u2502 ** Note, these licenses are not used for Intel\u00ae FPGA PAC D5005** \u251c\u2500\u2500 sim **Unit level simulation files**\n\u2502 \u251c\u2500\u2500 unit_test\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 bfm\n\u2502 \u251c\u2500\u2500 rp_bfm \u2502 \u2514\u2500\u2500 readme.txt \u2502 \u251c\u2500\u2500 LICENSE.txt\n\u251c\u2500\u2500 README.md\n|\n\u251c\u2500\u2500 src **Source RTL files**\n\u2502 \u251c\u2500\u2500 afu_top\n\u2502 \u251c\u2500\u2500 includes\n\u2502 \u251c\u2500\u2500 pd_qsys\n\u2502 \u251c\u2500\u2500 top\n| \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 external\n\u2502 \u2514\u2500\u2500 ofs-platform-afu-bbb\n|\n\u251c\u2500\u2500 syn **Quartus compilation settings**\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 setup\n\u2502 \u251c\u2500\u2500 syn_top\n\u2502 \u251c\u2500\u2500 readme.txt\n\u2502 \u2514\u2500\u2500 README\nThe required setup OFS License quartus-0.0-0.01iofs-linux.run, follow the following steps :
cd $OFS_ROOTDIR/license\nchmod +x quartus-0.0-0.01iofs-linux.run\nsudo ./quartus-0.0-0.01iofs-linux.run\n# Confirm the license instaltion using below command.\nquartus_syn --version\nThe ofs-platform-afu-bbb repository contains the PIM files and example AFU that can be used for testing and demonstration purposes. This guide will use the host_chan_mmio example in the remaining sections to demonstrate OFS capabilities.
cd $OFS_BUILD_ROOT\ngit clone https://github.com/OFS/ofs-platform-afu-bbb.git\nexport OFS_PLATFORM_AFU_BBB=$OFS_BUILD_ROOT/ofs-platform-afu-bbb\nVerify the following directories are present in $OFS_BUILD_ROOT directory.
cd $OFS_PLATFORM_AFU_BBB\nls\nConsole Output:
COPYING plat_if_develop plat_if_release plat_if_tests README.md\nOFS provides a build script with the following FPGA image creation options:
The build scripts included with OFS are verified to run in a bash shell. Other shells have not been tested. Each build script step will take several hours to complete. Building in Quartus GUI is not supported - you must build with the provided scripts.
The following sections describe how to set up the environment and build the provided FIM with a relocatable tree supporting PR . You will use this relocatable PR tree for all example AFU simulation and compilation steps in this guide.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2121-setting-up-required-environment-variables","title":"2.1.2.1. Setting Up Required Environment Variables","text":"Set required environment variables as shown below. These environment variables must be set before simulation or compilation tasks, so creating a simple script to set these variables saves time.
Edit your bashrc file ~/.bashrc to add the following lines:
export OPAE_SDK_REPO_BRANCH=release/2.12.0\nCheck point : Ensure you file ~/.bashrc have all the following lines:
export QUARTUS_MAINPATH=<Quartus install directory>\nexport QUARTUS_ROOTDIR=$QUARTUS_MAINPATH/quartus\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ipexport\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_MAINPATH/qsys/bin\nexport PATH=$PATH:$QUARTUS_ROOTDIR/bin\nexport OFS_BUILD_ROOT=<root location> ** Here should be located your ofs-d5005 and ofs-platform-afu-bbb\nexport OFS_ROOTDIR=$OFS_BUILD_ROOT/ofs-d5005\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\nexport OFS_PLATFORM_AFU_BBB=$OFS_BUILD_ROOT/ofs-platform-afu-bbb\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\nThe usage of the compile build script is shown below:
ofs-common/scripts/common/syn/build_top.sh [-p] target_configuration work_dir \n\n * target_configuration - Specifies the project \n For example: d5005\n\n * work_dir - Work Directory for this build in the form a directory name. It is created in the <local repo directory>/ofs-d5005/<work_dir> \n - NOTE: The directory name must start with \"work\". If the working directory exists, the script stops and asks if you want to overwrite the directory.\n - e.g.\n - ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005\n\n work directory as a name will be created in <local repo directory>/ofs-d5005/work_d5005\n\n The obmission of <work_dir> results in a default work directory (<local repo directory>/ofs-d5005/work)\n\n - compile reports and artifacts (.rpt, .sof, etc) are stored in <work_dir>/syn/<OFS_PROJECT>/<OFS_FIM>/<OFS_BOARD>/syn_top/output_files\n\n - There is a log file created in ofs-d5005 directory. \n - [-p] Optional switch for creating a relocatable PR build tree supporting the creation of a PR -able AFU workload. \n The \"-p\" switch invokes generate_pr_release.sh at the end of the FIM build and writes the PR build tree to the top of the working directory. More information on this option is provided below. \nBuild the provided base example design:
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh d5005 work_d5005\nConsole Output:
... build takes ~5 hours to complete\nCompile work directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top\nCompile artifact directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top/output_files\n\n***********************************\n***\n*** OFS_PROJECT: d5005\n*** Q_PROJECT: d5005\n*** Q_REVISION: d5005\n*** SEED: 03\n*** Build Complete\n*** Timing Passed!\n***\nPro Tip: if the timing report fails, try to go into the iofs_pr_afu.qsf and modify the seed number from 3 to 4, it will create multiple seed/starting points of your design to find the best timing/fit. /home/<myuser>/<mainfolderforOFS>/ofs-d5005/work_d5005/syn/syn_top/iofs_pr_afu.qsf
set_global_assignment -name SEED 0 #0-4\nThe build script copies the ipss, sim, src, and syn directories to the specified work directory, and then these copied files are used in the Quartus compilation process. Therefore, do not edit the files in the work directory; these files are copies of source files.
Some of the critical output files are described below:
$OFS_ROOTDIR//syn/syn_top
\u251c\u2500\u2500 syn_top //Intel\u00ae FPGA PAC D5005 Quartus build area with Quartus files used this build\n\u2502 \u251c\u2500\u2500 d5005.ipregen.rpt // IP regeneration report states the output of IP upgrade\n\u2502 \u251c\u2500\u2500 d5005.qpf // Quartus Project File (qpf) mentions about Quartus version and project revision\n\u2502 \u251c\u2500\u2500 d5005.qsf // Quartus Settings File (qsf) lists current project settings and entity level assignments\n\u2502 \u251c\u2500\u2500 d5005.stp // Signal Tap file included in the d5005.qsf. This file can be modified as required\n\u2502 \u251c\u2500\u2500 fme_id.mif // the fme id hex value is stored in a mif file format\n\u2502 \u251c\u2500\u2500 iofs_pr_afu.json // PR JSON file\n\u2502 \u251c\u2500\u2500 iofs_pr_afu.qsf // PR AFU qsf file\n\u2502 \u251c\u2500\u2500 iofs_pr_afu_sources.tcl // AFU source file list\n$OFS_ROOTDIR//syn/syn_top/output_files == Directory with build reports and FPGA programming files.
The programming files consist of the Quartus generated d5005.sof and d5005.pof. The Intel\u00ae FPGA PAC D5005 board hardware provides a 2 Gb flash device to store the FPGA programming files and a BMC CARD that reads this flash and programs the Intel\u00ae FPGA PAC D5005 Stratix\u00ae 10 FPGA. The ./ofs-common/scripts/common/syn/build_top.sh script runs script file ./ofs-common/scripts/common/syn/build_top.sh which takes the Quartus generated d5005.sof and creates binary files in the proper format to be loaded into the 2 Gb flash device. You can also run build_flash.sh by itself if needed.
The build script will run PACSign and create an unsigned FPGA programming file for both user1 and user2 locations of the Intel\u00ae FPGA PAC D5005 flash. Please note, if the Intel\u00ae FPGA PAC D5005 has the root entry hash key loaded, then PACsign must be run to add the proper key to the FPGA binary file. Please see Security User Guide: Open FPGA Stack for details on the security aspects of Open FPGA Stack and refer to Board Management User Guide for Flash partition.
The following table provides further detail on the generated bin files.
File Description d5005.sof This is the Quartus generated programming file created by Quartus synthesis and place and route. This file can be used to program the FPGA using a JTAG programmer. This file is the source file for the binary files used to program the FPGA flash. d5005.bin This is an intermediate raw binary image of the FPGA d5005_page1.bin This is the binary file created from the input file, d5005.sof. This file is used as the input file to the PACSign utility to generate d5005_page1_unsigned.bin binary image file. d5005_page1_unsigned.bin This is the unsigned PACSign output which can be programmed into the FPGA flash of an unsigned Intel\u00ae FPGA PAC D5005 using the OPAE SDK utility fpgasupdate mfg_d5005_reversed.bin A particular programming file for a third-party device used in board manufacturing. This file is typically not used.build/output_files/timing_report == Directory containing clocks report, failing paths and passing margin reports
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#213-relocatable-pr-directory-tree","title":"2.1.3. Relocatable PR Directory Tree","text":"If you are developing FIM to be used by another team developing the AFU workload, scripts are provided that create a relocatable PR directory tree. ODM and board developers will use this capability to enable a broad set of AFU to be loaded on a board using PR . The relocatable PR directory contains the Quartus *.qdb file that goes the FIM.
Creating the relocatable PR directory tree requires a clone of the Basic Building Blocks (BBB) repository. The OFS_PLATFORM_AFU_BBB environment variable must point to the repository. If not done previously, clone the Basic Building Blocks repository and create OFS_PLATFORM_AFU_BBB environment variable.
cd $OFS_BUILD_ROOT\ngit clone https://github.com/OFS/ofs-platform-afu-bbb.git\nexport OFS_PLATFORM_AFU_BBB=$OFS_BUILD_ROOT/ofs-platform-afu-bbb\ncd $OFS_ROOTDIR\nYou can create this relocatable PR directory tree by either:
The generate_pr_release.sh has the following command structure:
ofs-common/scripts/common/syn/generate_pr_release.sh -t <path to generated release tree> *Board Build Target* <work dir from build_top.sh>\nWhere:\n-t <path to generated release tree> = location for your relocatable PR directory tree\n*Board Build Target* is the name of the board target/FIM e.g. d5005\n<work dir from build_top.sh> ofs-common/scripts/common/syn/generate_pr_release.sh -t work_d5005/build_tree d5005 work_d5005\nConsole Output:
**********************************\n********* ENV SETUP **************\nFIM Project:\n OFS_PROJECT = d5005\n OFS_FIM = .\n OFS_BOARD = .\n Q_PROJECT = d5005\n Q_REVISION = d5005\n Fitter SEED = 03\nFME id\n BITSTREAM_ID = 040100022c164db1\n BITSTREAM_MD = 0000000002212053\n...\n...\nThe resulting relocatable build tree has the following structure:
.\n\u251c\u2500\u2500 bin\n\u2502 \u251c\u2500\u2500 afu_synth\n\u2502 \u251c\u2500\u2500 build_env_config\n\u2502 \u251c\u2500\u2500 run.sh -> afu_synth\n\u2502 \u2514\u2500\u2500 update_pim\n\u251c\u2500\u2500 hw\n\u2502 \u251c\u2500\u2500 blue_bits\n\u2502 \u2502 \u251c\u2500\u2500 d5005_page1_unsigned.bin\n\u2502 \u2502 \u2514\u2500\u2500 d5005.sof -> ../lib/build/syn/syn_top/ output_files/d5005.sof\n\u2502 \u2514\u2500\u2500 lib\n\u2502 \u251c\u2500\u2500 build\n\u2502 \u251c\u2500\u2500 fme-ifc-id.txt\n\u2502 \u251c\u2500\u2500 fme-platform-class.txt\n\u2502 \u2514\u2500\u2500 platform\n\u251c\u2500\u2500 README\nEdit your bashrc file ~/.bashrc to add the following line:
export OPAE_PLATFORM_ROOT=$OFS_ROOTDIR/work_d5005/build_tree\nsudo fpgainfo fme\nConsole Output:
Board Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nDevice Id : 0xBCCE\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\nThe base FIM used in AFU compilation must be loaded on the board. In this step, you will load the generated FIM binary into the Intel\u00ae FPGA PAC D5005 FPGA flash. By performing this step, subsequent AFU developed in this guide will use this base FIM and allow your newly created AFU to match the base FIM loaded on the board.
More information related to fpgaupdate is located Software Installation Guide: OFS for PCIe Attach FPGAs.
Run fpgasupdate to load the image into the user location of the Intel\u00ae FPGA PAC D5005 FPGA flash and the RSU command to reboot the PCIE Card:
sudo fpgasupdate $OFS_ROOTDIR/work_d5005/syn/syn_top/output_files/d5005_page1_unsigned.bin 3b:00.0\nsudo rsu bmcimg 3b:00.0\nsudo fpgainfo fme\nConsole Output: ```sh
Board Management Controller, MAX10 NIOS FW version: 2.0.8 Board Management Controller, MAX10 Build version: 2.0.8 //****** FME ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:3b:00.0 Device Id : 0xBCCE Socket Id : 0x00 Ports Num : 01 Bitstream Id : 288511860124977321 Bitstream Version : 4.0.1 Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e Boot Page : user
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#30-opae-software-development-kit","title":"3.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines the integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and re-configure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, please visit the OPAE.io page.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE GitHub. This repository is open source and should not require any permissions to access.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#31-opae-sdk-build-environment-setup","title":"3.1 OPAE SDK Build Environment Setup","text":"This installation process assumes the user has access to an internet connection to pull specific GitHub repositories and satisfy package dependencies. If an offline install process is required, please reach out to your Altera\u00ae representative.
1. Before OPAE SDK installation, the user must remove any prior OPAE frameworks. To remove these packages:
sudo dnf remove opae*\n2. The user must enable the following repository changes in order to install all dependencies on CentOS 8.3:
sudo dnf config-manager --set-enabled powertools\nsudo dnf install epel-release\n3. The following package dependencies must be satisfied by the user. Double check that all packages have been found and installed:
sudo dnf install autoconf automake bison boost boost-devel cmake doxygen dwarves elfutils-libelf-devel \\\nflex gcc gcc-c++ git hwloc-devel json-c-devel libarchive libedit libedit-devel libpcap libpng12 libuuid libuuid-devel libxml2 libxml2-devel make ncurses \\\nncurses-devel ncurses-libs openssl-devel python2-pip python3-devel python3-jsonschema rsync tbb-devel libudev-devel\nAll steps in this installation will use a generic top-level directory at $OFS_BUILD_ROOT. If the user has created a different top-level directory, replace this path with the user's custom path.
4. Initialize an empty git repository and clone the tagged OPAE SDK source code:
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#32-install-opae-sdk","title":"3.2 Install OPAE SDK","text":"Perform the following steps to install OPAE SDK:
cd $OFS_BUILD_ROOT\ngit clone https://github.com/OFS/opae-sdk\ncd opae-sdk\ngit checkout tags/2.12.0-5 -b release/2.12.0\ngit describe\n 2.12.0-5\n\ngit branch\n master\n * release/2.12.0\n1. Build the OPAE SDK source code, and pack it into several local RPM packages. Building the code into packages allows for easier installation and removal. This build script can use multiple processors to parallelize the build process. Display how many processors are available with the nproc command, and then specify how many make threads to utilize with the -j option. Note that the number of threads can exceed the number of processors. In this case, the number of threads is set to the number of processors in the system.
cd $OFS_BUILD_ROOT/opae-sdk\nmkdir install-opae-sdk\ncd install-opae-sdk\ncmake .. -DCPACK_GENERATOR=RPM -DOPAE_BUILD_FPGABIST=ON -DOPAE_BUILD_PYTHON_DIST=ON -DCMAKE_BUILD_PREFIX=/install-opae-sdk make -j `nproc`\nmake -j `nproc` package_rpm\nThe install-opae-sdk directory location was selected for ease of use. If the user wishes to build the OPAE SDK in a different location, they will need to replace the '..' in the above command with the direct or relative path to the opae-sdk repository.
2. After a successful compile, there should be eight packages present:
cd $OFS_BUILD_ROOT/opae-sdk/install-opae-sdk\nls | grep rpm\nopae-2.12.0-5.x86_64.rpm opae-PACSign-2.12.0-5.x86_64.rpm opae-devel-2.12.0-5.x86_64.rpm opae-libs-2.12.0-5.x86_64.rpm opae-opae.admin-2.12.0-5.x86_64.rpm opae-packager-2.12.0-5.x86_64.rpm opae-tests-2.12.0-5.x86_64.rpm opae-tools-2.12.0-5.x86_64.rpm opae-tools-extra-2.12.0-5.x86_64.rpm\n3. Install the OPAE SDK packages:
cd $OFS_BUILD_ROOT/opae-sdk/install-opae-sdk\nsudo dnf localinstall -y opae*.rpm\n4. check that all packages have been installed:
rpm -qa | grep opae\nopae-devel-2.12.0-5.x86_64 opae-packager-2.12.0-5.x86_64 opae-2.12.0-5.x86_64 opae-tools-2.12.0-5.x86_64 opae-PACSign-2.12.0-5.x86_64 opae-tools-extra-2.12.0-5.x86_64 opae-opae.admin-2.12.0-5.x86_64 opae-tests-2.12.0-5.x86_64 opae-libs-2.12.0-5.x86_64\n5. Setup required environment variables
export PATH=$PATH:$OFS_BUILD_ROOT/opae-sdk/install-opae-sdk/bin\nexport LIBRARY_PATH=$OFS_BUILD_ROOT/opae-sdk/install-opae-sdk/lib\nexport LD_LIBRARY_PATH=$OFS_BUILD_ROOT/opae-sdk/install-opae-sdk/lib64\ncd ../lib/python*/site-packages\nexport PYTHONPATH=$PWD\nThis section will use the FIM build tree created in the previous steps to compile an example AFU. This section will continue the work with the host_chan_mmio AFU.. You can perform the build steps listed below to demonstrate the ease in building and running a real example on the Intel\u00ae FPGA PAC D5005.
To run the steps in this section, you must complete all steps in section 2. Set Up AFU Development Environment, and ensure the OPAE_PLATFORM_ROOT \"environment variable that points to the directory of the PR build tree generated previously.
Ensure your bashrc file ~/.bashrc have the following line:
export OPAE_PLATFORM_ROOT=$OFS_ROOTDIR/work_d5005/build_tree\nHere, you will create the synthesis environment to build the host_chan_mmio example. The PIM flow includes the synthesis environment creation script afu_synth_setup for this task. The usage of afu_synth_setup is shown below:
usage: afu_synth_setup [-h] -s SOURCES [-p PLATFORM] [-l LIB] [-f] dst\nGenerate a Quartus build environment for an AFU. A build environment is\ninstantiated from a release and configured for the specified AFU. AFU\nsource files are specified in a text file parsed by rtl_src_config,\nwhich is part of the OPAE base environment.\npositional arguments:\n dst Target directory path (directory must not exist).\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA platform name.\n -l LIB, --lib LIB FPGA platform release hw/lib directory. If not\n specified, the environment variables OPAE_FPGA_HW_LIB\n and then BBS_LIB_PATH are checked.\n -f, --force Overwrite target directory if it exists.\nExecute afu_synth_setup \"as follows to create the synthesis environment for a host_chan_mmio \"AFU that fits the Intel\u00ae FPGA PAC D5005 FIM previously constructed.
cd $OFS_ROOTDIR/work_d5005/\nafu_synth_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt build_d5005_x16\nNow, execute the afu_synth command that resides inside the $OFS_ROOTDIR/work_d5005/build_tree/bin directory, to actually build the host_chan_mmio AFU.
cd $OFS_ROOTDIR/work_d5005/build_d5005_x16\n$OPAE_PLATFORM_ROOT/bin/afu_synth\n...\n...\nWrote host_chan_mmio.gbs\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\nOnce the compilation completes successfully, load the new bitstream file, host_chan_mmio.gbs, into the partial reconfiguration region of the target Intel\u00ae FPGA PAC D5005. Keep in mind, that the loaded image is dynamic - this image is not stored in flash, and if the card is power cycled, then the PR region is re-loaded with the default AFU.
To load the image, perform the following steps:
cd $OFS_ROOTDIR/work_d5005/build_d5005_x16\nsudo fpgasupdate host_chan_mmio.gbs 3b:00.0\n[sudo] password for <<Your username>>: [WARNING ] Update starting. Please do not interrupt.\n[INFO ] Partial Reconfiguration OK\n[INFO ] Total time: 0:00:01.88\nDetermine the BDF of the Intel\u00ae FPGA PAC D5005.
The PCIe BDF address is initially determined when the server powers on. The user can determine the addresses of all Intel\u00ae FPGA PAC D5005 using lspci:
lspci -d :bcce\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nSet up your board to work with the newly loaded host_chan_mmio.gbs
Create the Virtual Functions (VFs):
sudo pci_device 3b:00.0 vf 3\nVerify that all three VFs have been created.
$ lspci -s 3b:00\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nsudo opae.io init -d , e.g.
$ sudo opae.io init -d 0000:3b:00.1 user:user\nBinding (0x8086,0xbccf) at 0000:3b:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.1 is 142\nAssigning /dev/vfio/142 to <local user>\nChanging permissions for /dev/vfio/142 to rw-rw----\n\n$ sudo opae.io init -d 0000:3b:00.2 user:user\nBinding (0x8086,0xbccf) at 0000:3b:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.2 is 143\nAssigning /dev/vfio/143 to <local user>\nChanging permissions for /dev/vfio/143 to rw-rw-----\n\n$ sudo opae.io init -d 0000:3b:00.3 user:user\nBinding (0x8086,0xbccf) at 0000:3b:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.3 is 144\nAssigning /dev/vfio/144 to <local user>\nChanging permissions for /dev/vfio/144 to rw-rw----\n$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x603B000000000000\nPCIe s:b:d.f : 0000:3B:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0x403B000000000000\nPCIe s:b:d.f : 0000:3B:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 76d7ae9c-f66b-461f-816a-5428bcebdbc5\n//****** PORT ******//\nObject Id : 0x203B000000000000\nPCIe s:b:d.f : 0000:3B:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\nRun the host_chan_mmio software application to demonstrate the newly loaded AFU image. You navigate to $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw and compile the software application and then run.
If OPAE SDK libraries were not installed in the default systems directory /usr/lib64/ \", define the OPAE_LOC environment variable to point to the directory where the OPAE SDK libraries were installed.
$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib64:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\ncd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw\nmake ./host_chan_mmio\nConsole Output:
AFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 250 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\nThe platform-independent examples-afu repository provides some interesting examples AFU's. In this section, you will compile and execute the PIM-based hello_world AFU. The RTL of the hello_world AFU receives from the host application an address via memory-mapped I/O (MMIO) write and generates a DMA write to the memory line at that address. The content written to memory is the string \"Hello world!\". The host application spins, waiting for the memory line to be updated. Once available, the software prints out the string.
The hello_world example AFU consists of the following files.
hello_world\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 ccip\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u2514\u2500\u2500 hello_world.json\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 hello_world.c\n \u2514\u2500\u2500 Makefile\nThe following instructions can be used to compile other AFU samples accompanying this repository.
cd $OFS_BUILD_ROOT git clone https://github.com/OFS/examples-afu.git\n # OPAE and MPF libraries must either be on the default linker search paths or on both LIBRARY_PATH and LD_LIBRARY_PATH. \n$ export OPAE_LOC=/usr\n $ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n# OPAE_PLATFORM_ROOT points to a release tree that has been configured with the Platform Interface Manager (PIM). \n$ export OPAE_PLATFORM_ROOT=$OFS_ROOTDIR/work_d5005/build_tree\nCompile the hello_word sample AFU.
$ cd $OFS_ROOTDIR/work_d5005\n $ afu_synth_setup -s $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt hello_world_synth\n $ cd hello_world_synth\n $ ${OPAE_PLATFORM_ROOT}/bin/afu_synth\n\n.\n.\n.\nInfo (19538): Reading SDC files took 00:00:06 cumulatively in this process.\nWrote hello_world.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'hello_world.gbs'\nDesign meets timing\n===========================================================================\nTo test the AFU in actual hardware, load the hello_world.gbs to the Intel\u00ae FPGA PAC D5005 card. For this step to be successful, the Intel\u00ae FPGA PAC D5005 FIM must have already been loaded to the Intel\u00ae FPGA PAC D5005 card following the steps described in Section 2 of this document.
$ cd $OFS_ROOTDIR/work_d5005/hello_world_synth\n $ sudo fpgasupdate hello_world.gbs 3b:00.0\n [sudo] password for <<Your username>>: [2022-12-06 13:25:10.22] [WARNING ] Update starting. Please do not interrupt.\n[2022-12-06 13:25:12.06] [INFO ] Partial Reconfiguration OK\n[2022-12-06 13:25:12.06] [INFO ] Total time: 0:00:01.83\nSet up your Intel\u00ae FPGA PAC D5005 board to work with the newly loaded hello_world.gbs file.
# Create the Virtual Functions (VFs):\n$ sudo pci_device 3b:00.0 vf 3\n# Verify:\n$ lspci -s 3b:00\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.1 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.2 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.3 Processing accelerators: Intel Corporation Device bccf (rev 01)\n# Bond VFs to VFIO driver. Enter <<Your username>>\nsudo opae.io init -d 0000:3b:00.1 <Your username>\n Unbinding (0x8086,0xbcce) at 0000:3b:00.1 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:3b:00.1 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:3b:00.1 is 142\nAssigning /dev/vfio/142 to <Your username>\n Changing permissions for /dev/vfio/142 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.2 <Your username>\n Unbinding (0x8086,0xbccf) at 0000:3b:00.2 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:3b:00.2 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:3b:00.2 is 143\nAssigning /dev/vfio/143 to <Your username>\n Changing permissions for /dev/vfio/143 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.3 <Your username>\n Unbinding (0x8086,0xbccf) at 0000:3b:00.3 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:3b:00.3 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:3b:00.3 is 144\nAssigning /dev/vfio/144 to <Your username>\n Changing permissions for /dev/vfio/144 to rw-rw----\n\n# < Verify the new AFU is loaded. The hello_world AFU GUID is \"c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\".\n$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x603B000000000000\nPCIe s:b:d.f : 0000:3B:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n//****** PORT ******//\nObject Id : 0x403B000000000000\nPCIe s:b:d.f : 0000:3B:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n//****** PORT ******//\nObject Id : 0x203B000000000000\nPCIe s:b:d.f : 0000:3B:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\nhello_world AFU. You should see the application outputs the \"Hello world!\" message in the terminal. # Move to the sw directory of the hello_world AFU and run the following commands in user mode\ncd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw/\n\nmake\n\n# Launch the host application\n./hello_world\n Hello world TLP!\nAn OPAE compliant AFU specifies the frequency of the uclk_usr and uclk_usr_div2 clocks through the JSON file for AFU configuration located under the <afu_example>/hw/rtl directory of an AFU design. For instance, the AFU configuration file of the host_chan_mmio example is $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/host_chan_mmio.json.
The AFU specifies the frequency for uClk_usr in its platform configuration file using the following key:value pairs:
\"clock-frequency-high\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\n \"clock-frequency-low\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\nThese key:value tuples are used to configure the PLL of the target platform that provides the user clocks through the AFU clocks interface. In addition, the specified frequency affects the timing closure process on the user clocks during AFU compilation.
Setting the value field to a float number (e.g., 315.0 to specify 315 MHz) drives the AFU generation process to close timing within the bounds set by the low and high values and sets the AFU's JSON metadata to specify the user clock PLL frequency values.
The following example shows the JSON file of the host_chan_mmio to set the AFU uClk to 300 MHz and uClk_div2 to 150 MHz.
{\n \"version\": 1,\n \"afu-image\": {\n \"power\": 0,\n \"clock-frequency-high\": 300,\n \"clock-frequency-low\": 150,\n \"afu-top-interface\":\n {\n \"class\": \"ofs_plat_afu\"\n },\n \"accelerator-clusters\":\n [\n {\n \"name\": \"host_chan_mmio\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\"\n }\n ]\n }\n}\nSave the changes to host_chan_mmio.json file, then execute the afu_synth_setup script to create a new copy of the AFU files with the modified user clock settigns.
$ cd $OFS_ROOTDIR/work_d5005\n $ afu_synth_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt build_d5005_afu_clks\n\nLoading platform database: /home/<Your username>/<Your localpath>/ofs-d5005/work_d5005/build_tree/hw/lib/platform/platform_db/ofs_d5005.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting platform/platform_afu_top_config.vh\nWriting platform/platform_if_addenda.qsf\nWriting ../hw/afu_json_info.vh\nhost_chan_mmio AFU with the new frequency values. cd $OFS_ROOTDIR/work_d5005/build_d5005_afu_clks\n $OFS_ROOTDIR/work_d5005/build_tree/bin/afu_synth\nDuring the compilation phase, you will observe the Timing Analyzer uses the specified user clock frequency values as the target to close timing.
AFU developers must ensure the AFU hardware design meets timing. The compilation of an AFU that fails timing shows a message similar to the following.
.\n.\n.\n\nWrote host_chan_mmio.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\n*** Design does not meet timing\n *** See build/syn/syn_top/output_files/timing_report\n\n===========================================================================\nThe previous output indicates the location of the timing reports for the AFU designer to identify the failing paths and perform the necessary design changes. Next, is a listing of the timing report files from a host_chan_mmio AFU that fails to meet timing after modifying the user clock frequency values.
$ cd $OFS_ROOTDIR/work_d5005/build_d5005_afu_clks\n $ ls build/syn/syn_top/output_files/timing_report\n\nclocks.rpt clocks.sta.fail.summary clocks.sta.pass.summary iofs_pr_afu_2_slow_900mv_0c_recovery.rpt\niofs_pr_afu_2_slow_900mv_0c_setup.rpt\niofs_pr_afu_2_slow_900mv_100c_recovery.rpt\niofs_pr_afu_2_slow_900mv_100c_setup.rpt\niofs_pr_afu_2_slow_vid2_0c_recovery.rpt\niofs_pr_afu_2_slow_vid2_0c_setup.rpt\niofs_pr_afu_2_slow_vid2_100c_recovery.rpt\niofs_pr_afu_2_slow_vid2_100c_setup.rpt\niofs_pr_afu_MIN_fast_900mv_0c_recovery.rpt\niofs_pr_afu_MIN_fast_900mv_0c_setup.rpt\niofs_pr_afu_MIN_fast_900mv_100c_recovery.rpt\niofs_pr_afu_MIN_fast_900mv_100c_setup.rpt\nWarning: AFU developers must inform software developers of the maximum operating frequency (Fmax) of the user clocks to avoid any unexpected behavior of the accelerator and potentially of the overall system.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#5-simulating-an-afu-using-ase","title":"5. Simulating an AFU using ASE","text":"The AFU Simulation Environment (ASE) is a hardware/software co-simulation environment for your AFU. See diagram below illustrating ASE operation:
ASE uses the simulator Direct Programming Interface (DPI) to provide HW/SW connectivity. The PCIe connection to the AFU under testing is emulated with a transactional model.
The following list describes ASE operation:
The remainder of this section is a tutorial providing the steps on how to run ASE with either VCS or QuestaSim using an example AFU and the AFU build tree previously created in this guide.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#51-set-up-steps-to-run-ase","title":"5.1. Set Up Steps to Run ASE","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"The Intel\u00ae FPGA PAC D5005 PAC card requires opae-2.12.0-5. Follow the instructions documented in the Software Installation Guide: OFS for PCIe Attach FPGAs to build and install the required OPAE SDK for the Intel\u00ae FPGA PAC D5005 PAC card. Make sure to check out the cloned repository to tag 2.12.0-5 and branch release/2.12.0.
git checkout tags/2.12.0-5 -b release/2.12.0\nASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE-SDK. However, the recommendation is to install it in the same target directory as OPAE-SDK.
If not done already, set the environment variables as described in section, Set Up AFU Development Environment.
Clone the ase-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n $ cd opae-sim mock/opae_std.h. If the OPAE-SDK was installed under the default system directories, the C_INCLUDE_PATH variable must be set as follows. export C_INCLUDE_PATH=\"/usr/src/debug/opae-2.12.0-5.el8.x86_64/tests/framework\"\n mkdir build\n cd build\n cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n make\nOptionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> .. /usr. sudo make install The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
cd /usr/bin\n export PATH=$PWD:$PATH\ncd ../lib/python*/site-packages\n export PYTHONPATH=$PWD\ncd /usr/lib\n export LIBRARY_PATH=$PWD\ncd /usr/lib64\n export LD_LIBRARY_PATH=$PWD\ncd $OFS_BUILD_ROOT/ofs-platform-afu-bbb\n export OFS_PLATFORM_AFU_BBB=$PWD\ncd $OFS_ROOTDIR/work_d5005/build_tree\n export OPAE_PLATFORM_ROOT=$PWD\n## For VCS, set the following:\nexport VCS_HOME=<Set the path to VCS installation directory>\n export PATH=$VCS_HOME/bin:$PATH\n## For QuestaSIM, set the following:\nexport MTI_HOME=<path to Modelsim installation directory>\n export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\nThe $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files:
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_axi.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\nThis example AFU contains examples using both Avalon and AXI interface buses. This guide will use the AXI version of the host_chan_mmio AFU.
ASE uses client-server application architecture to deliver hardware/software co-simulation. You require one shell for the hardware based simulation and another shell where the software application is running. The hardware is started first with a simulation compilation and simulator startup script, once the simulator has loaded the design, it will wait until the software process starts. Once the software process starts, the simulator proceeds. Transaction logging and waveform capture is performed.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#521-set-up-and-run-the-hw-simulation-process","title":"5.2.1 Set Up and Run the HW Simulation Process","text":"You will run the afu_sim_setup script to create the scripts for running the ASE environment. The afu_sim_setup script has the following usage:
usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n [-f] [--ase-mode ASE_MODE] [--ase-verbose]\n dst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\n Default simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\nRun afu_sim_setup to create the ASE simulation environment for the host_chan_mmio example AFU. The '-t VCS' option indicates to prepare the ASE simulation environment for VCS.
cd $OFS_ROOTDIR/work_d5005/\n\nafu_sim_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt -t VCS host_chan_mmio_sim\n\nCopying ASE from /opae-sdk/install-opae-sdk/share/opae/ase...\nCopying ASE from /usr/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\n\nTool Brand: VCS\nLoading platform database: /home/<Your username>/<Your localpath>/ofs-d5005/work_d5005/build_tree/hw/lib/platform/platform_db/ofs_d5005.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\nThe afu_sim_setup creates the ASE scripts in the directory host_chan_mmio_sim where the afu_sim_setup script was run. Start the simulator as shown below in user mode:
cd host_chan_mmio_sim\n make\n make sim\nThis process launches the AFU hardware simulator. Before moving to the next section, pay attention to the simulator output highlighted in the image below.
The simulation artifacts are stored in host_chan_mmio/work and consist of:
log_ase_events.tsv\nlog_ofs_plat_host_chan.tsv \nlog_ofs_plat_local_mem.tsv \nlog_pf_vf_mux_A.tsv \nlog_pf_vf_mux_B.tsv \nOpen an additional shell to build and run the host application that communicates with the actual AFU hardware. Set up the same environment variable you have set up in the shell you have been working on until this point.
Additionally, as indicated by the hardware simulator output that is currently executing in the \"simulator shell\", copy and paste the line \"export ASE_WORKDIR=...\", into the new \"software shell\". See the last image of the previous section.
export ASE_WORKDIR= <<as directed in HW simulation shell>>\nhost_chan_mmio AFU example to compile the host application. cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw make\n\nafu_json_mgr json-info --afu-json=../hw/rtl/host_chan_mmio.json --c-hdr=obj/afu_json_info.h\nWriting obj/afu_json_info.h\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c main.c -o obj/main.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c test_host_chan_mmio.c -o obj/test_host_chan_mmio.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/connect.c -o obj/connect.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/csr_mgr.c -o obj/csr_mgr.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/hash32.c -o obj/hash32.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/test_data.c -o obj/test_data.o\ncc -o host_chan_mmio obj/main.o obj/test_host_chan_mmio.o obj/connect.o obj/csr_mgr.o obj/hash32.o obj/test_data.o -z noexecstack -z relro -z now -pie -luuid -lopae-c\nNow, launch the host application to exercise the AFU hardware running on the simulator shell. The next image shows the AFU hardware simulation process on the left side shell. The right hand shell shows the host application's output of a successful simulation.
Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
make wave\nThis brings up the VCS simulator GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ofs_plat_afu | afu , as shown below.
Right click on the afu (afu) entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#53-simulating-the-hello_world-afu","title":"5.3 Simulating the hello_world AFU","text":"
In this section, you will quickly simulate the PIM-based hello_world sample AFU accompanying the examples-afu repository.
Set the environment variables as described in section 5.1. Set Up Steps to Run ASE.
Prepare an RTL simulation environment for the AXI version of the hello_world AFU.
Simulation with ASE requires two software processes, one to simulate the AFU RTL and the other to run the host software that exercises the AFU. To construct an RTL simulation environment under the directory $OFS_ROOTDIR/work_d5005, execute the following.
$ cd $OFS_ROOTDIR/work_d5005\n $ afu_sim_setup -s $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt -t VCS hello_world_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/<Your username>/<Your localpath>/ofs-d5005/work_d5005/build_tree/hw/lib/platform/platform_db/ofs_d5005.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\nThe afu_sim_setup script constructs an ASE environment in the hello_world_sim subdirectory. If the command fails, confirm that the path to the afu_sim_setup is on your PATH environment variable (in the OPAE SDK bin directory) and that your Python version is at least 3.7.
Build and execute the AFU RTL simulator in user mode.
cd $OFS_ROOTDIR/work_d5005/hello_world_sim\n make\n make sim The previous commands will build and run the VCS RTL simulator, which prints a message saying it is ready for simulation. The simulation process also prints a message instructing you to set the ASE_WORKDIR environment variable in a second shell.
Open a second shell where you will build and execute the host software. In this new \"software shell\", set up the environment variables you have set up so far in the \"hardware simulation\" shell.
Also, set the ASE_WORKDIR environment variable following the instructions given in the \"hardware simulation\" shell.
export ASE_WORKDIR=$OFS_ROOTDIR/work_d5005/hello_world_sim/work\nhello_world AFU sample to build the host software. cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n make hello_world host application to resume the work of the RTL simulation. The host software process and the RTL simulation execute in lockstep. If successful, you should see the Hello world! output. $ with_ase ./hello_world\n\n[APP] Initializing simulation session ...\nHello world!\n [APP] Deinitializing simulation session\n [APP] Took 43,978,424 nsec\n [APP] Session ended\nThe image below shows the simulation of the AFU hardware and the execution of the host application side-by-side.
make wave\nThis brings up the DVE GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the AFU instance located under, ase_top | ase_top_plat | ofs_plat_afu, as shown below.
Right click on the ofs_plat_afu entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
The OPAE SDK provides a remote Signal Tap facility. It also supports the following in system debug tools included with the Quartus\u00ae Prime Pro Edition:
This section is a short guide on adding remote Signal Tap instances to an AFU for in-system debugging. In order of execution, you can follow the steps in the following sections to create an instrumented AFU. The host_chan_mmio AFU is used in this guide as the target AFU to be instrumented.
You need a basic understanding of Signal Tap. Please see the Signal Tap Logic Analyzer: Introduction & Getting Started Web-Based Training for more information.
You will run with a Signal Tap GUI running locally on the server with the Intel\u00ae FPGA PAC D5005 as shown below:
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#61-adding-rstp-to-the-host_chan_mmio-afu","title":"6.1. Adding RSTP to the host_chan_mmio AFU","text":"RSTP is added to an AFU by:
The following steps use the previously built host_chan_mmio AFU example. You can use these detailed steps to instrument your AFU.
Navigate to host_chan_mmio AFU Quartus project and open the project using Quartus GUI.
cd $OFS_ROOTDIR/work_d5005/build_d5005_x16/build/syn/syn_top\n $ quartus d5005.qpf &\nOnce the project is loaded in Quartus, review the project hierarchy as shown in the Project Navigator. This example will add Signal Tap probe points to the AFU region. Reviewing the code will give insight into the function of this block. You can up the code in the Project Navigator by expanding afu_top - port_gasket - pr_slot - afu_main - ofs_plat_afu, then select instance afu, right-click, select Locate Node - Locate in Design File as shown below.
Bring up Signal Tap to create the *.stp file. In the Quartus GUI, go to Tools - Signal Tap Logic Analyzer. Click Create to accept the default template in the New File from Template pop-up. The Signal Tap Logic Analyzer window comes up.
Set up the clock for the Signal Tap logic instance by clicking ... button as shown below:
5. The Node Finder comes up, and you will click ... as shown below to bring up the hierarchy navigator or copy-paste the following location at Look in:
iofs_top|afu_top|port_gasket|pr_slot|afu_main|ofs_plat_afu|afu\nIn the Select Hierarchy Level, navigate to top - afu_top - port_gasket - pr_slot - afu_main - ofs_plat_afu, then select instance afu and click Ok.
Enter *clk* in the Named: box and click Search. This brings up matching terms. Click mmio64_if.clk and >. Verify your Node Finder is as shown below and then click Ok:
Double click the Double-click to add nodes and once again, click ... and navigate to top - afu_top - port_gasket - pr_slot - afu_main - ofs_plat_afu, then select instance afu and click Ok. Enter mmio64 then click >> to add these signals to the STP instance as shown below:
Then click Insert and Close.
Save the newly created STP by clicking File - Save As in the save as navigate to $OFS_ROOTDIR/work_d5005/build_d5005_x16/build/syn/syn_top and save the STP file as host_chan_mmio.stp as shown below:
ofs_top.qsf to add host_chan_mmio.stp file and enable STP. Open $OFS_ROOTDIR/work_d5005/build_d5005_x16/build/syn/syn_top/d5005.qpf in an editor and modify lines as shown below:set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE host_chan_mmio.stp\nset_global_assignment -name SIGNALTAP_FILE host_chan_mmio.stp\nSave the d5005.qpf and close Quartus.
iofs_pr_afu.qsf to add host_chan_mmio.stp file and enable STP. Open $OPAE_PLATFORM_ROOT/hw/lib/build/syn/syn_top/iofs_pr_afu.qsf in an editor and ensure the lines are included as below (note: the verilog macro INCLUDE_REMOTE_STP will already be present), also copy and paste the file host_chan_mmio.stp in this location:The updated lines are:
set_global_assignment -name VERILOG_MACRO \"INCLUDE_REMOTE_STP\"\nset_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE host_chan_mmio.stp\nset_global_assignment -name SIGNALTAP_FILE host_chan_mmio.stp\n $ cd $OFS_ROOTDIR/build_d5005_x16\n $ afu_synth_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt build_d5005_x16_stp\n\n Notice that your previous build_d5005_x16_stp directory is preserved, and a new build_d5005_x16_stp directory is created. You will use build_d5005_x16_stp to build the STP-enabled image.\n\n $ cd build_d5005_x16_stp\n $ $OPAE_PLATFORM_ROOT/bin/afu_synth\n\n...\n...\nWrote host_chan_mmio.gbs\n\n===========================================================================\n PR AFU compilation complete\n AFU gbs file is 'host_chan_mmio.gbs'\n Design meets timing\n===========================================================================\n$ sudo fpgasupdate host_chan_mmio.gbs 3b:00.0\n[sudo] password for <myuser>: \n[WARNING ] Update starting. Please do not interrupt.\n [INFO ] \nPartial Reconfiguration OK\n[INFO ] Total time: 0:00:01.87\nUsage:\nmmlink\n<Segment> --segment=<SEGMENT NUMBER>\n<Bus> --bus=<BUS NUMBER> OR -B <BUS NUMBER>\n<Device> --device=<DEVICE NUMBER> OR -D <DEVICE NUMBER>\n<Function> --function=<FUNCTION NUMBER> OR -F <FUNCTION NUMBER>\n<Socket-id> --socket-id=<SOCKET NUMBER> OR -S <SOCKET NUMBER>\n<TCP PORT> --port=<PORT> OR -P <PORT>\n<IP ADDRESS> --ip=<IP ADDRESS> OR -I <IP ADDRESS>\n<Version> -v,--version Print version and exit\nProTip:
Open a new shell session for mmlink; this console needs to remain open to allow mmlink connection.
Enter the command below to create a connection using port 3333:
$ sudo mmlink -P 3333 -B 0x3b\n\n------- Command line Input START ----\n\nSocket-id : -1\n Port : 3333\nIP address : 0.0.0.0\n ------- Command line Input END ----\n\nPORT Resource found.\nServer socket is listening on port: 3333\nLeave this shell open with the mmlink connection.
$ jtagconfig --add JTAG-over-protocol sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0\n\nVerify connectivity with jtagconfig --debug\n\n$ jtagconfig --debug\n1) JTAG-over-protocol [sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0]\n(JTAG Server Version ${{ env.D5005_QUARTUS_PRIME_PRO_VER_S }}.0 Build 104 09/14/2022 SC Pro Edition)\n020D10DD VTAP10 (IR=10)\nDesign hash D41D8CD98F00B204E980\n + Node 00406E00 Virtual JTAG #0\nCaptured DR after reset = (020D10DD) [32]\nCaptured IR after reset = (155) [10]\nCaptured Bypass after reset = (0) [1]\nCaptured Bypass chain = (0) [1]\ncd $OPAE_PLATFORM_ROOT/hw/lib/build/syn/syn_top/\nquartus_stpw host_chan_mmio.stp\nThis command brings up Signal Tap GUI. Connect to the Signal Tap over protocol by selecting the Hardware button on the right side of the GUI and clicking the \"Please Select\" pull-down as shown below:
JTAG over protocol selected:
This connection process will take approximately 2-3 minutes for the Signal Tap instance to indicate \"Ready to acquire\".
8) Set the trigger condition for a rising edge on signal valid signal. 9) In the Signal Tap window, enable acquisition by pressing key F5. The Signal Tap GUI will indicate \"Acquisition in progress\". Run the hello_world application and observe that the Signal Tap instance has triggered. You should see signals being captured in the Signaltap GUI.
See captured image below:
To end your Signal Tap session, close the Signal Tap GUI, then in the mmlink shell, enter ctrl c to kill the mmlink process.
The vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\nEnable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\nIf you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\nSample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\nIntel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/","title":"Shell Developer Guide: Open FPGA Stack for Stratix 10\u00ae FPGA","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a design guide for FPGA developers, system architects and hardware developers using OFS as a starting point for the creating the FPGA Interface Manager (FIM) for a custom FPGA acceleration board or Platform with Altera FPGAs.
This document uses the Intel\u00ae FPGA PAC D5005 as an example platform to illustrate key points and demonstrate how to extend the capabilities provided in OFS (Open FPGA Stack) to custom platforms. The demonstration steps serves as a tutorial for the development of your OFS knowledge.
This document covers OFS architecture lightly. For more details on the OFS architecture, please see Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
You are encouraged to read Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs to fully understand how AFU Developers will use your newly developed FIM.
Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#12-introduction","title":"1.2. Introduction","text":"Open FPGA Stack (OFS) addresses the scalability for FPGA acceleration boards and workloads by providing a powerful and systematic methodology for the rapid development of FPGA-based Acceleration systems. This methodology addresses the key challenges of hardware, software and workload developers by providing a complete FPGA project consisting of RTL and simulation code, build scripts and software. The FPGA project released in OFS can be rapidly customized to meet new market requirements by adding new features, custom IPs and interface subsystems IPs.
A high-level overview of the OFS Stratix\u00ae 10 FPGA hardware architecture on the Stratix\u00ae 10 FPGA reference platform, Intel\u00ae FPGA PAC D5005 is shown in the below figure. The provided FPGA architecture is divided into two main components
- The outer area in white, the FPGA Interface manager (or FIM) - The inner area in green, the Acceleration Function Unit or AFU Region.
The outer area, the FIM, provides the core infrastructure and interfaces within the FPGA. The AFU region is where a user\u2019s custom logic would reside for their specific workload.
* FPGA external interfaces and IP cores (e.g. Ethernet, DDR-4, PCIe, etc) * PLLs/resets * FPGA - Board management infrastructure * Interface to Acceleration Function Unit (AFU)
The AFU region has both static and dynamic partial reconfiguration regions enabling a lot of customization.
* Uses the FIM interfaces to perform useful work inside the FPGA * Contains logic supporting partial reconfiguration * Remote Signal Tap core for remote debugging of workload
Outside of the FPGA is the Board Management Controller which provides board management, root of trust, board monitoring, and remote system updates.
The overall architecture is built to be very composable and modular in blocks that can be modified while leaving the rest of the infrastructure intact so you may only need to modify a few of these blocks.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#12-release-capabilities","title":"1.2. Release Capabilities","text":"This release of OFS FIM supports the following key features:
OFS is extensible to meet the needs of a broad set of customer applications, however not all use cases are easily served. The general uses cases listed below are examples where the OFS base design can be easily re-used to build a custom FIM: 1. Use OFS reference design as-is - Porting the code to another platform that is identical to the OFS reference platform only changing target FPGA device and pinout - Change I/O assignments without changing design 2. Update the configuration of peripheral IP in OFS reference design, not affecting FIM architecture - External memory settings - HSSI analog settings 3. Remove/update peripheral feature in OFS reference design, not affecting FIM architecture - External memory speed/width change - Change 10G Ethernet to 25 or 100G Ethernet IP - Change number of VFs supported 4. Add new features as an extension to OFS reference design, not affecting FIM architecture - Add/remove external memory interface to the design - Add/remove user clocks for AFU - Add/remove IP to the design with connection to AFU
More advanced use cases requiring changes or additions to the host PCIe channel are not easily supported with this release of the OFS FIM.
Reuse of the provided host management FPGA logic and software is the fastest and most simple approach to FIM customization.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#13-prerequisites","title":"1.3. Prerequisites","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#131-base-knowledge-and-skills-prerequisites","title":"1.3.1. Base Knowledge and Skills Prerequisites","text":"OFS is an advanced application of FPGA technology. This guide assumes you have the following FPGA logic design-related knowledge and skills:
To run the tutorial steps in this guide requires this development environment:
Item Version Quartus Prime Pro Quartus Prime Pro 23.4 (with license patch) Target D5005 Sever Operating System RHEL 8.6 OPAE SDK 2.12.0-5 Linux DFL ofs-2024.1-6.1-2 Python 3.6.8 cmake 3.15 GCC 8.5.0 perl 5.8.8The following server and Intel PAC card are required to run the examples in this guide:
The steps included in this guide have been verified in the Dell R740 and HPE ProLiant DL380 Gen10 servers.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#2-high-level-description","title":"2. High Level Description","text":"The FIM targets operation in the Intel\u00ae FPGA PAC D5005 card. The block diagram of the D5005 is shown below:
The key D5005 FPGA interfaces are:
The FPGA Interface Manager architecture is shown in the below diagram:
The FIM consists of the following components - PCIe Subsystem - Memory Subsystem - HSSI Subsystem - Platform Management Component Intercommunications (PMCI) - Board Peripheral Fabric (BPF) - AFU Peripheral Fabric (APF) - Port Gasket - AXI-S PF/VF Demux/Mux - Host Exerciser Modules - HE-MEM, HE-LB, HE-HSSI - FPGA Management Engine (FME)
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#22-fim-fpga-resource-usage","title":"2.2. FIM FPGA Resource Usage","text":"The FIM uses a small portion of the available FPGA resources. The table below shows resource usage for a base FIM built with 2 channels of external memory, a small AFU instantiated that has host CSR read/write, external memory test and Ethernet test functionality.
Entity ALMs Used % ALMS Used M20Ks % M20Ks used DSP Blocks Pins IOPLLs OFS_top 125009.4 13.0% 661 5.4% 0 630 15 afu_top 70522.7 7.0% 228 2.4% 0 0 1 auto_fab_0 1305.7 0.0% 9 0.1% 0 0 0 bpf_rsv_5_slv 0.6 0.0% 0 0.0% 0 0 0 bpf_rsv_6_slv 0.6 0.0% 0 0.0% 0 0 0 bpf_rsv_7_slv 0.4 0.0% 0 0.0% 0 0 0 bpf 241.9 0.0% 0 0.0% 0 0 0 emif_top_inst 10508.6 1.0% 0 0.0% 0 0 12 eth_ac_wrapper 6024.8 0.5% 9 0.1% 0 0 0 fme_top 615.5 0.2% 7 0.1% 0 0 0 pcie_wrapper 35424.7 3.5% 348 2.9% 0 0 1 pmci_top 318.5 0.1% 0 0.0% 0 0 0 rst_ctrl 40.2 0.0% 0 0.0% 0 0 0 sys_pll 0.5 0.0% 0 0.0% 0 0 1 Total ALMS 933,120 Total M20Ks 11,721 Summary FPGA Resource Utilization Logic utilization (in ALMs) 124,092 / 933,120 ( 13 % ) Total dedicated logic registers 282822 Total pins 630 / 912 ( 69 % ) Total block memory bits 3,425,120 / 240,046,080 ( 1 % ) Total RAM Blocks 661 / 11,721 ( 6 % ) Total DSP Blocks 0 / 5,760 ( 0 % ) Total eSRAMs 0 / 75 ( 0 % ) Total HSSI P-Tiles 17 / 48 ( 35 % ) Total HSSI E-Tile Channels 17 / 48 ( 35 % ) Total HSSI HPS 0 / 1 ( 0 % ) Total HSSI EHIPs 0 / 2 ( 0 % ) Total PLLs 36 / 104 ( 35 % )"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#23-ofs-directory-structure","title":"2.3. OFS Directory Structure","text":"The OFS Git OFS repository ofs-d5005 directory structure is shown below:
\u251c\u2500\u2500 eval_script\n| \u251c\u2500\u2500 ofs_d5005_eval.sh\n| \u2514\u2500\u2500 README_ofs_d5005_eval.txt\n\u251c\u2500\u2500 ipss\n\u2502 \u251c\u2500\u2500 hssi\n| \u251c\u2500\u2500 mem\n| \u251c\u2500\u2500 pcie\n| \u251c\u2500\u2500 pmci\n| \u251c\u2500\u2500 spi\n| \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 license\n\u2502 \u2514\u2500\u2500 quartus-0.0-0.01Intel OFS-linux.run\n\u251c\u2500\u2500 ofs-common\n| \u251c\u2500\u2500 scripts\n| \u251c\u2500\u2500 src\n| \u251c\u2500\u2500 verification\n| \u251c\u2500\u2500 LICENSE.txt\n| \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 sim\n| \u251c\u2500\u2500 bfm\n| \u251c\u2500\u2500 rp_bfm\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 scripts\n| \u251c\u2500\u2500 unit_test \u2502\u00a0\u00a0 \u2514\u2500\u2500 readme.txt\n\u251c\u2500\u2500 src\n\u2502 \u251c\u2500\u2500 afu_top\n\u2502 \u251c\u2500\u2500 includes\n\u2502 \u251c\u2500\u2500 pd_qsys\n\u2502 \u251c\u2500\u2500 README.md\n\u2502 \u2514\u2500\u2500 top\n\u251c\u2500\u2500 syn\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 setup\n\u2502 \u251c\u2500\u2500 syn_top\n\u2502 \u251c\u2500\u2500 readme.txt\n\u2502 \u2514\u2500\u2500 README\n\u251c\u2500\u2500 LICENSE.txt\n\u2514\u2500\u2500 README.md\nThe contents of each directory are described below:
Eval Script - Contains scripts for evaluation of OFS for D5005 including compiling FIM/AFU from source, unit level test. Also includes resources to report and setup D5005 development environment
ipss - Contains the code and supporting files that define or set up the IP subsystems (HSSI, PCIe, memory, PMCI, SPI, etc...) contained in the D5005 FPGA Interface Manager (FIM).
license - License file for the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core.
ofs-common - This directory contains resources that may be used across the board-specific repositories. This directory is referenced via a link within each of the FPGA-specific repositories.
sim - Contains the testbenches and supporting code for all the unit test simulations. - Bus Functional Model code is contained here. - Scripts are included for automating a myriad of tasks. - All of the individual unit tests and their supporting code is also located here.
src - SystemVerilog source and script files - Contains all of the structural and behavioral code for the FIM. - Scripts for generating the AXI buses for module interconnect. - Top-level RTL for synthesis is located in this directory. - Accelerated Functional Unit (AFU) infrastructure code is contained in this directory.
syn - This directory contains all of the scripts, settings, and setup files for running synthesis on the FIM.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#3-description-of-sub-systems","title":"3. Description of Sub-Systems","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#31-host-control-and-data-flow","title":"3.1. Host Control and Data Flow","text":"The host control and data flow are shown in the diagram below:
The control and data flow is composed of the following:
Peripherals are connected to one another using AXI:
Peripherals are presented to software as:
The peripherals connected to the peripheral fabric are primarily OPAE managed resources, whereas the peripherals connected to the AFU are \u201cprimarily\u201d managed by native OS drivers. The word \u201cprimarily\u201d is used since the AFU is not mandated to expose all its peripherals to OPAE. It can be connected to the peripheral fabric, but can choose to expose only a subset of its capability to OPAE.
OFS uses a defined set of CSRs to expose the functionality of the FPGA to the host software. These registers are described in Open FPGA Stack Reference Manual - MMIO Regions section.
If you make changes to the FIM that affect the software operation, then OFS provides a mechanism to communicate that information to the proper software driver. The Device Feature Header (DFH) structure provides a mechanism to maintain compatibility with OPAE software. Please see FPGA Device Feature List (DFL) Framework Overview for an excellent description of DFL operation from the driver perspective.
When you are planning your address space for your FIM updates, please be aware that the OFS FIM targeting Intel\u00ae FPGA PAC D5005, 256KB of MMIO region is allocated for external FME features and 128kB of MMIO region is allocated for external port features. Each external feature must implement a feature DFH, and the DFH needs to be placed at 4KB boundary. The last feature in the external feature list must have the EOL bit in its DFH set to 1 to mark the end of external feature list. Since the FPGA address space is limited, consider using an indirect addressing scheme to conserve address space.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#4-fim-development-flow","title":"4. FIM Development Flow","text":"OFS provides a framework of FPGA synthesizable code, simulation environment, and synthesis/simulation scripts. FIM designers can take the provided code and scripts and modify existing code or add new code to meet their specific product requirements.
FIM development for a new acceleration card consists of the following steps:
The FIM developer works closely with the hardware design of the target board, software development and system validation.
Understanding how the AFU developer utilizes the FIM is important for FIM development success. Please read Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs for a detailed description of AFU development.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#41-installation-of-ofs","title":"4.1. Installation of OFS","text":"In this section you set up a development machine for compiling the OFS FIM. These steps are separate from the setup for a deployment machine where the FPGA acceleration card is installed. Typically, FPGA development and deployment work is performed on separate machines, however, both development and deployment can be performed on the same server if desired. Please see Software Installation Guide: OFS for PCIe Attach FPGAs for instructions on installing software for deployment of your FPGA FIM, AFU and software application on a server.
Building the OFS FIM requires the development machine to have at least 64 GB of RAM.
The following is a summary of the steps to set up for FIM development:
ofs-d5005 repositoryQuartus Prime Pro version 23.4 is the currently verified version of Quartus used for building the FIM and AFU images for this release. Porting to newer versions of Quartus may be performed by developers. Download Quartus Prime Pro Linux version 23.4 from Quartus\u00ae Prime Pro Edition Linux.
Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.6 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\nCreate the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
Download your required Quartus Prime Pro Linux version here.
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are No patches for this release.
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\nFor example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\nVerify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\nAfter running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=$PATH:<Quartus install directory>/quartus/bin\nFor example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=$PATH:/home/intelFPGA_pro/23.4/quartus/bin\nVerify, Quartus is discoverable by opening a new shell:
which quartus\n## Output\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\nsudo dnf install libnsl\nsudo dnf install ncurses-compat-libs\nsudo ln -s /usr/bin/python3 /usr/bin/python\nYou will need to obtain a license for Quartus Prime Pro version 23.4 to compile the design. Additionally, OFS for Stratix\u00ae 10 FPGA requires a license for the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core. This license is required to generate a programming file using the provided OFS source code. The Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core license patch installer is provided in the ofs-d5005 git repository in the /license directory. After cloning the OFS release in step 4 below, you can install this IP license.
sudo dnf install git\n\u200b The OFS FIM source code is included in the GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_fim_build_root\ncd OFS_fim_build_root\nexport OFS_BUILD_ROOT=$PWD\ngit clone --recurse-submodules https://github.com/OFS/ofs-d5005.git\ncd ofs-d5005\ngit checkout tags/ofs-2024.1-1\ngit describe --tags\nofs-2024.1-1\ncd license\nchmod +x quartus-0.0-0.01Intel OFS-linux.run\nsudo ./quartus-0.0-0.01Intel OFS-linux.run\nquartus_sh --version\n##Output\nQuartus Prime Shell\nVersion 23.4 Pro Edition\nOFS provides a build script with the following FPGA image creation options:
The build scripts included with OFS are verified to run in a bash shell. Other shells have not been tested. Each build script step will take several hours to completes, Please note, building directly in Quartus GUI is not supported - you must build with the provided scripts.
The following sections describe how to set up the environment and build the provided FIM and AFU. Follow these steps as a tutorial to learn the build flow. You will use this environment and build scripts for the creation of your specialized FIM.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#421-setting-up-required-environment-variables","title":"4.2.1. Setting Up Required Environment Variables","text":"Set required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks so creating a simple script to set these variables saves time.
cd $OFS_BUILD_ROOT/ofs-d5005\nexport OFS_ROOTDIR=$PWD\n## Note, OFS_ROOTDIR is the directory where you cloned the repo, e.g. /home/MyProject/ofs-d5005 *\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\n## Note, QUARTUS_ROOTDIR is your Quartus installation directory, e.g. $QUARTUS_ROOTDIR/bin contains Quartus executuable*\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\nThe usage of the compile build script is shown below:
ofs-common/scripts/common/syn/build_top.sh [-p] target_configuration work_dir Usage: ofs-common/scripts/common/syn/build_top.sh [-k] [-p] <build target> [<work dir name>]\nBuild a FIM instance specified by <build target>. The target names an FPGA architecture, board and configuration.\n\nThe FIM is built in <work dir name>. If not specified, the target is ${OFS_ROOTDIR}/work.\n\nThe -k option preserves and rebuilds within an existing work tree instead of overwriting it.\n\nWhen -p is set, if the FIM is able then a partial reconfiguration template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable\n and uses only relative paths. See ofs-common/scripts/common/syn/generate_pr_release.sh for details.\n\nThe -e option runs only Quartus analysis and elaboration.\n\n* target_configuration - Specifies the project For example: d5005\n\n* work_dir - Work Directory for this build in the form a directory name. It is created in the <local repo directory>/ofs-d5005/<work_dir> - NOTE: The directory name must start with \"work\". If the work directory exists, then the script stops and asks if you want to overwrite the directory.\n - e.g.\n - ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005\n\nwork directory as a name will be created in <local repo directory>/ofs-d5005/work_d5005\n\nThe obmission of <work_dir> results in a default work directory (<local repo directory>/ofs-d5005/work)\n- compile reports and artifacts (.rpt, .sof, etc) are stored in <work_dir>/syn/syn_top/output_files\n\n- There is a log file created in ofs-d5005 directory. - [-p] Optional switch for creation of a relocatable PR build tree supporting the creation of a PR-able AFU workload. The \"-p\" switch invokes generate_pr_release.sh at the end of the FIM build and writes the PR build tree to the top of the work directory. More information on this option is provided below. Build the provided base example design:
```bash cd $OFS_BUILD_ROOT/ofs-d5005
ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005 ```
```bash ... build takes ~5 hours to complete
Compile work directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top Compile artifact directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top/output_files
*** OFS_PROJECT: d5005 *** Q_PROJECT: d5005 *** Q_REVISION: d5005 *** SEED: 03 *** Build Complete *** Timing Passed!
The build script copies the ipss, sim, src and syn directories to the specified work directory and then these copied files are used in the Quartus compilation process. Do not edit the files in the work directory, these files are copies of source files.\n\nSome of the key files are described below:\n\n<work_dir>/syn/syn_top == \n```bash\n\u251c\u2500\u2500 syn_top // D5005 Quartus build area with Quartus files used this build\n\u2502\u00a0\u00a0\u251c\u2500\u2500 d5005.ipregen.rpt // IP regeneration report states the output of IP upgrade\n\u2502\u00a0\u00a0\u251c\u2500\u2500 d5005.qpf // Quartus Project File (qpf) mentions about Quartus version and project revision\n\u2502 \u251c\u2500\u2500 d5005.qsf // Quartus Settings File (qsf) lists current project settings and entity level assignments\n\u2502\u00a0\u00a0\u251c\u2500\u2500 d5005.stp // Signal Tap file included in the d5005.qsf. This file can be modified as required if you need to add an Signal Tap instance\n\u2502\u00a0\u00a0\u251c\u2500\u2500 fme_id.mif // the fme id hex value is stored in a mif file format\n\u2502 \u251c\u2500\u2500 OFS_pr_afu.json // PR JSON file\n\u2502\u00a0\u00a0\u251c\u2500\u2500 OFS_pr_afu.qsf // PR AFU qsf file\n\u2502\u00a0\u00a0\u251c\u2500\u2500 OFS_pr_afu_sources.tcl // AFU source file list\n\u2502\u00a0\u00a0\u251c\u2500\u2500 ip_upgrade_port_diff_reports // IP upgrade report files for reference\nThe programming files consist of the Quartus generated d5005.sof and d5005.pof. The D5005 board hardware provides a 2 Gb flash device to store the FPGA programming files and a MAX10 BMC that reads this flash and programs the D5005 Stratix\u00ae 10 FPGA FPGA. The syn/build_top.sh script runs script file syn/syn_top/build_flash/build_flash.s which takes the Quartus generated d5005.sof and creates binary files in the proper format to be loaded into the 2 Gb flash device. You can also run build_flash.sh by yourself if needed. The build_flash script runs PACSign (if installed) to create an unsigned FPGA programming file that can be stored in the D5005 FPGA flash. Please note, if the D5005 has the root entry hash key loaded, then PACsign must be run with d5005_page1.bin as the input with the proper key to create an authenticated FPGA binary file. Please see Security User Guide: Open FPGA Stack for details on the security aspects of Open FPGA Stack.
The following table provides further detail on the generated bin files.
File Description d5005.sof This is the Quartus generated programming file created by Quartus synthesis and place and route. This file can be used to programming the FPGA using a JTAG programmer. This file is used as the source file for the binary files used to program the FPGA flash. d5005.bin This is an intermediate raw binary image of the FPGA d5005_page1.bin This is the binary file created from input file, d5005.sof. This file is used as the input file to the PACSign utility to generate d5005_page1_unsigned.bin binary image file. d5005_page1_unsigned.bin This is the unsigned PACSign output which can be programmed into the FPGA flash of an unsigned D5005 usign the OPAE SDK utility fpgasupdate mfg_d5005_reversed.bin A special programming file for a third party programming device used in board manufacturing. This file is typically not used.build/output_files/timing_report == Directory containing clocks report, failing paths and passing margin reports
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#423-relocatable-pr-directory-tree","title":"4.2.3. Relocatable PR Directory Tree","text":"If you are developing a FIM to be used by another team developing the AFU workload, scripts are provided that create a relocatable PR directory tree. ODM and board developers will make use of this capability to enable a broad set of AFUs to be loaded on a board using PR. The relocatable PR directory contains the Quartus *.qdb file that goes the FIM.
The creation of the relocatable PR directory tree requires a clone of the Basic Building Blocks (BBB) repository. The OFS_PLATFORM_AFU_BBB environment variable must point to the repository, for example.
cd $OFS_BUILD_ROOT\ngit clone https://github.com/OPAE/ofs-platform-afu-bbb\ncd ofs-platform-afu-bbb\nexport OFS_PLATFORM_AFU_BBB=$PWD\ncd $OFS_ROOTDIR\nYou can create this relocatable PR directory tree by either:
The generate_pr_release.sh has the following command structure:
./ofs-common/scripts/common/syn/generate_pr_release.sh -t <path to generated release tree> *Board Build Target* <work dir from build_top.sh>\n\nWhere:\n\n-t <path to generated release tree> = location for your relocatable PR directory tree\n*Board Build Target* is the name of the board target/FIM e.g. d5005\n<work dir from build_top.sh> ofs-common/scripts/common/syn/generate_pr_release.sh -t work_d5005/build_tree d5005 work_d5005\n**********************************\n********* ENV SETUP **************\n\nFIM Project:\n OFS_PROJECT = d5005\n OFS_FIM = .\n OFS_BOARD = .\n Q_PROJECT = d5005\n Q_REVISION = d5005\n Fitter SEED = 03\nFME id\n BITSTREAM_ID = 04010002c7cab852\n BITSTREAM_MD = 0000000002204283\n...\n...\n.\n\u251c\u2500\u2500 bin\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 afu_synth\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build_env_config\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 run.sh -> afu_synth\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 update_pim\n\u251c\u2500\u2500 hw\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 blue_bits\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 d5005_page1_unsigned.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 d5005.sof -> ../lib/build/syn/syn_top/output_files/d5005.sof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 lib\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-ifc-id.txt\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-platform-class.txt\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 platform\nThis build tree can be moved to a different location and used for AFU development of a PR capable AFU to be used with this board.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#424-unit-level-simulation","title":"4.2.4. Unit Level Simulation","text":"Unit level simulation of key components is provided. These simulations provide verification of the following areas:
These simulations use the Synopsys VCS simulator. Each simulation contains a readme file explaining how to run the simulation. Refer to UVM Simulation User Guide: OFS for Stratix\u00ae 10 PCIe Attach for details of simulation examples. Your simulation shell requires Python, Quartus, and VCS to run. To run a simulation of the dfh_walker that simulates host access to the internal DFH registers, perform the following steps:
Before running unit simulation, you must set environment variables as described below:
cd $OFS_BUILD_ROOT/ofs-d5005\nexport OFS_ROOTDIR=$PWD\n## *Note, OFS_ROOTDIR is the directory where you cloned the repo, e.g. /home/MyProject/ofs-d5005 *\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\n## *Note, QUARTUS_ROOTDIR is your Quartus installation directory, e.g. $QUARTUS_ROOTDIR/bin contains Quartus executuable*\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\nTo compile all IPs:
To Generate Simulation Files & compile all IPs, run the following command:
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\nsh gen_sim_files.sh d5005\nThe IPs are generated here:
$OFS_ROOTDIR/sim/scripts/qip_gen\n$OFS_ROOTDIR/sim/scripts/ip_flist.f\nTo run the simulation, run the following command:
cd $OFS_ROOTDIR/sim/unit_test/<Unit Test Name>/scripts\nsh run_sim.sh VCS=1\nTo view simulation waveform:
cd $OFS_ROOTDIR/sim/unit_test/<test_name>/script/sim/unit_test/<test_name>/scripts/sim_vcs\ndve -full64 -vpd vcdplus.vpd &\n********************************************\n Running TEST(0) : test_fme_dfh_walking\n********************************************\nREAD64: address=0x00000000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010000000\n\nFME_DFH\n Address (0x0)\nDFH value (0x4000000010000000)\nREAD64: address=0x00001000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000020000001\n\nTHERM_MNGM_DFH\n Address (0x1000)\nDFH value (0x3000000020000001)\nREAD64: address=0x00003000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000007\n\nGLBL_PERF_DFH\n Address (0x3000)\nDFH value (0x3000000010000007)\nREAD64: address=0x00004000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000c0001004\n\nGLBL_ERROR_DFH\n Address (0x4000)\nDFH value (0x30000000c0001004)\nREAD64: address=0x00010000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000000e\n\nSPI_DFH\n Address (0x10000)\nDFH value (0x300000010000000e)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000100000020\n\nPCIE_DFH\n Address (0x20000)\nDFH value (0x3000000100000020)\nREAD64: address=0x00030000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000100f\n\nHSSI_DFH\n Address (0x30000)\nDFH value (0x300000010000100f)\nREAD64: address=0x00040000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000500000009\n\nEMIF_DFH\n Address (0x40000)\nDFH value (0x3000000500000009)\nREAD64: address=0x00090000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010001005\n\nFME_PR_DFH\n Address (0x90000)\nDFH value (0x3000000010001005)\nREAD64: address=0x00091000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010001001\n\nPORT_DFH\n Address (0x91000)\nDFH value (0x4000000010001001)\nREAD64: address=0x00092000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000014\n\nUSER_CLOCK_DFH\n Address (0x92000)\nDFH value (0x3000000010000014)\nREAD64: address=0x00093000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000d0002013\n\nPORT_STP_DFH\n Address (0x93000)\nDFH value (0x30000000d0002013)\nREAD64: address=0x000a0000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000010000002010\n\nAFU_INTF_DFH\n Address (0xa0000)\nDFH value (0x3000010000002010)\nMMIO error count matches: x\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_fme_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\nThe simulation transcript is displayed while the simulation runs. The transcript is saved to the file transcript.out for review after the simulation completes. The simulation waveform database is saved as vcdplus.vpd for post simulation review. You are encouraged to run the additional simulation examples to learn about each key area of the OFS shell.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#43-compiling-the-ofs-fim-using-eval-script","title":"4.3. Compiling the OFS FIM using Eval Script","text":"The Evaluation Script provides resources to setup and report D5005 development environment. You can use the evaluation script to compile and simulate the FIM. Refer to README_ofs_d5005_eval.txt for details of using the evaluation script.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#44-debugging","title":"4.4. Debugging","text":"For debugging issues within the FIM, Signal Tap can be used to gain internal visibility into your design. This section describes the process of adding a Signal Tap instance to your FIM design
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#441-signal-tap-prerequisites","title":"4.4.1. Signal Tap Prerequisites","text":"To use Signal Tap with OFS, you will need the following:
The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware, the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized. These steps assume the use of the Intel\u00ae FPGA PAC D5005.
To acquire signals using SignalTap, first load the Signal Tap instrumented SOF file into your target board, open the STP file in the Signal Tap GUI and start the signal acquisition.
Avoid system hang during programming the sof file, mask AER regsiter using below steps
Find Root complex - End Point mapping using the below command
lspci -vt\n+-[0000:3a]-+-00.0-[3b-3c]----00.0 Intel Corporation Device bcce\n | +-05.0 Intel Corporation Sky Lake-E VT-d\n | +-05.2 Intel Corporation Sky Lake-E RAS Configuration Registers\n | +-05.4 Intel Corporation Sky Lake-E IOxAPIC Configuration Registers\n | +-08.0 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-09.0 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.0 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.1 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.2 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.3 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.4 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.5 Intel Corporation Sky Lake-E LM Channel 1\nUse the bus information from the lspci logs to mask the AER (Advanced Error Reporting) register
sudo su\n\nsetpci -s 0000:3b:00.0 ECAP_AER+0x08.L=0xFFFFFFFF setpci -s 0000:3b:00.0 ECAP_AER+0x14.L=0xFFFFFFFF\nsetpci -s 0000:3a:00.0 ECAP_AER+0x08.L=0xFFFFFFFF\nsetpci -s 0000:3a:00.0 ECAP_AER+0x14.L=0xFFFFFFFF\necho \"1\" > /sys/bus/pci/devices/0000:3b:00.0/remove\n\nexit\nsudo su\necho \"1\" > /sys/bus/pci/rescan\nsudo fpgainfo fme\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.13 Board Management Controller, MAX10 Build version: 2.0.8 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x40100022C164DB1\nBitstream Version : 4.0.1\nPr Interface Id : 210a4631-18bb-57d1-879f-2c3d59b26e37\nBoot Page : user\nquartus_stpw\nHardware: selection box select the cable \"Stratix10 Darby Creek [ JTAG cable number ]\" as shown below:File open your STP file. Your STP file settings will load. If not already set, you can create the trigger conditions, then start analysis with F5.An example of FIM modification is provided in this section. This example can be used in your specific application as a starting point. This example shows the basic flow and listing of files that are to be changed.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#51-hello-fim-example","title":"5.1. Hello FIM example","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example adds a simple DFH register set to the FIM. You can use this example as the basis for adding a new feature to your FIM.
The steps to add this simple DFH register are described below.
The following sections describe changes to add the hello_fim DFH example to the provided FPGA design.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#511-srctopiofs_topsv","title":"5.1.1. src/top/iofs_top.sv","text":"//*******************************\n// FME\n//*******************************\nfme_top fme_top(\n.clk (clk_1x ),\n.rst_n (rst_n_d_1x ),\n.pwr_good_n (ninit_done ),\n.i_pcie_error ('0 ),\n.axi_lite_m_if (bpf_fme_mst_if ),\n.axi_lite_s_if (bpf_fme_slv_if )\n);\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top hello_fim_top (\n.clk (clk_1x),\n.rst_n (rst_n_d_1x),\n.csr_lite_if (bpf_rsv_5_slv_if)\n);\n`endif\n//*******************************\n// AFU\n//*******************************\nYou will connect the Hello_FIM DFH register to the existing BPF reserved link 5. The provided OFS reference design includes 3 reserved BPF interfaces available for custom usage such as new OPAE controlled modules. The address map of BPF is shown below:
Address Size (Byte) Feature Master 0x00000 \u2013 0x0FFFF 64K FME (FME, Error, etc) Yes 0x10000 \u2013 0x1FFFF 64K PMCI Proxy (SPI Controller) Yes 0x20000 \u2013 0x2FFFF 64K PCIe CSR 0x30000 \u2013 0x3FFFF 64K HSSI CSR 0x40000 \u2013 0x4FFFF 64K EMIF CSR 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x6FFFF 64K Reserved 0x70000 \u2013 0x7FFFF 64K ReservedThe BPF reserved link 5 is connected to a dummy connection to prevent this link from being optimized out the design during synthesis. You will add a compiler `define that will cause this dummy connection to be removed when the variable INCLUDE_HELLO_FIM is defined by editing line 575 if iofs_top.sv as shown below:
// Reserved address response\n`ifndef INCLUDE_HELLO_FIM\nbpf_dummy_slv\nbpf_rsv_5_slv (\n.clk (clk_1x),\n.dummy_slv_if (bpf_rsv_5_slv_if)\n);\n`endif\nThe Hello_FIM DFH is inserted in the DFH link list after the EMIF CSR DFH and before the FME_PR DFH. The file ipss/d5005/emif/emif_csr.sv contains a parameter defining the next address for the next DFH in in the link list chain. You will change the next address offset to be 0x10000 so the reveserved BPF AXI lite link connected to the Hello_FIM DFH register is next in the DFH link list.
module emif_csr #(\nparameter NUM_LOCAL_MEM_BANKS = 1,\nparameter END_OF_LIST = 1'b0,\n`ifndef INCLUDE_HELLO_FIM\nparameter NEXT_DFH_OFFSET = 24'h05_0000\n`else\nparameter NEXT_DFH_OFFSET = 24'h01_0000//New for Hello_FIM, next offset now at 0x50000\n`endif\n)\nCreate hello_fim_top.sv, and store it in src/hello_fim directory. The main purpose of this RTL is to convert AXI4-Lite interface to a simple interface to interface with the registers in hello_fim_com.sv.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0xfff which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2021 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : Nov 2021\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'hfff,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h04_0000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic rst_n,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (rst_n),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.rst_n (rst_n ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\nCreate hello_fim_com.sv, and store it in src/hello_fim directory. This is the simple RTL to implement the Hello FIM registers. You may use this set of registers as the basis for your custom implementation.
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2021 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : Nov 2021\n// Module Name : hello_fim_com.sv\n// Project : IOFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'hfff,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput rst_n,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nreg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(posedge clk) if (!rst_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(posedge clk)\nif (!rst_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( posedge clk)\nif (!rst_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\nTo run a unit level simulation test for the updated RTL files, make modifications to your cloned /my_ofs_project/ofs-d5005/sim/d5005/unit_test/dfh_walker files. The following simulation files are updated to test the new hello_fim.
Edit sim/unit_test/dfh_walker/testbench/test_csr_defs.sv
typedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nSPI_DFH_IDX,\nPCIE_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX,//New for HELLO_FIM\nFME_PR_DFH_IDX,\nPORT_DFH_IDX,\nUSER_CLOCK_DFH_IDX,\nPORT_STP_DFH_IDX,\nAFU_INTF_DFH_IDX,\nMAX_FME_DFH_IDX\n} t_fme_dfh_idx;\n1. Edit function dfh_name line 78
function automatic dfh_name[MAX_FME_DFH_IDX-1:0] get_fme_dfh_names();\ndfh_name[MAX_FME_DFH_IDX-1:0] fme_dfh_names;\nfme_dfh_names[FME_DFH_IDX] = \"FME_DFH\";\nfme_dfh_names[THERM_MNGM_DFH_IDX] = \"THERM_MNGM_DFH\";\nfme_dfh_names[GLBL_PERF_DFH_IDX] = \"GLBL_PERF_DFH\";\nfme_dfh_names[GLBL_ERROR_DFH_IDX] = \"GLBL_ERROR_DFH\";\nfme_dfh_names[SPI_DFH_IDX] = \"SPI_DFH\";\nfme_dfh_names[PCIE_DFH_IDX] = \"PCIE_DFH\";\nfme_dfh_names[HSSI_DFH_IDX] = \"HSSI_DFH\";\nfme_dfh_names[EMIF_DFH_IDX] = \"EMIF_DFH\";\nfme_dfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\";//New for HELLO_FIM\nfme_dfh_names[FME_PR_DFH_IDX] = \"FME_PR_DFH\";\nfme_dfh_names[PORT_DFH_IDX] = \"PORT_DFH\";\nfme_dfh_names[USER_CLOCK_DFH_IDX] = \"USER_CLOCK_DFH\";\nfme_dfh_names[PORT_STP_DFH_IDX] = \"PORT_STP_DFH\";\nfme_dfh_names[AFU_INTF_DFH_IDX] = \"AFU_INTF_DFH\";\nreturn fme_dfh_names;\nendfunction\n1. Update get_fme_dfh_values
function automatic [MAX_FME_DFH_IDX-1:0][63:0] get_fme_dfh_values();\nlogic[MAX_FME_DFH_IDX-1:0][63:0] fme_dfh_values;\nfme_dfh_values[FME_DFH_IDX] = 64'h4000_0000_1000_0000;\nfme_dfh_values[THERM_MNGM_DFH_IDX] = 64'h3_00000_002000_0001;\nfme_dfh_values[GLBL_PERF_DFH_IDX] = 64'h3_00000_001000_0007;\nfme_dfh_values[GLBL_ERROR_DFH_IDX] = 64'h3_00000_00C000_1004; fme_dfh_values[SPI_DFH_IDX] = 64'h3_00000_010000_000e; fme_dfh_values[PCIE_DFH_IDX] = 64'h3_00000_010000_0020; fme_dfh_values[HSSI_DFH_IDX] = 64'h3_00000_010000_100f; fme_dfh_values[EMIF_DFH_IDX] = 64'h3_00000_010000_0009; //Update to link to Hello_FIM \nfme_dfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_040000_0FFF; //New for Hello_FIM\nfme_dfh_values[FME_PR_DFH_IDX] = 64'h3_00000_001000_1005; fme_dfh_values[PORT_DFH_IDX] = 64'h4000_0000_1000_1001;\nfme_dfh_values[USER_CLOCK_DFH_IDX] = 64'h3_00000_001000_0014;\nfme_dfh_values[PORT_STP_DFH_IDX] = 64'h3_00000_00D000_2013;\nfme_dfh_values[AFU_INTF_DFH_IDX] = 64'h3_00001_000000_2010; return fme_dfh_values;\nendfunction\nVLOG_OPT += +define+SIM_MODE +define+VCS_S10 +define+RP_MAX_TAGS=64 +define+INCLUDE_DDR4 +define+INCLUDE_SPI_BRIDGE +define+INCLUDE_USER_CLOCK +define+INCLUDE_HSSI +define+SIM_USE_PCIE_DUMMY_CSR +define+INCLUDE_HELLO_FIM\n$WORKDIR/src/hello_fim/hello_fim_com.sv\n$WORKDIR/src/hello_fim/hello_fim_top.sv\nAfter making these changes, run the unit level simulation using sim/unit_test/dfh_walker test. Before running, ensure your shell has the environment variables set properly as defined in Setting Up Required Environment Variables.
cd verification/scripts\ngmake -f Makefile_VCS.mk cmplib\ngmake -f Makefile_VCS.mk build run [DUMP=1]\nExpected output:
********************************************\n Running TEST(0) : test_fme_dfh_walking\n********************************************\nREAD64: address=0x00000000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010000000\n\nFME_DFH\n Address (0x0)\nDFH value (0x4000000010000000)\nREAD64: address=0x00001000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000020000001\n\nTHERM_MNGM_DFH\n Address (0x1000)\nDFH value (0x3000000020000001)\nREAD64: address=0x00003000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000007\n\nGLBL_PERF_DFH\n Address (0x3000)\nDFH value (0x3000000010000007)\nREAD64: address=0x00004000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000c0001004\n\nGLBL_ERROR_DFH\n Address (0x4000)\nDFH value (0x30000000c0001004)\nREAD64: address=0x00010000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000000e\n\nSPI_DFH\n Address (0x10000)\nDFH value (0x300000010000000e)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000100000020\n\nPCIE_DFH\n Address (0x20000)\nDFH value (0x3000000100000020)\nREAD64: address=0x00030000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000100f\n\nHSSI_DFH\n Address (0x30000)\nDFH value (0x300000010000100f)\nREAD64: address=0x00040000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000100000009\n\nEMIF_DFH\n Address (0x40000)\nDFH value (0x3000000100000009)\nREAD64: address=0x00050000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000400000fff\n\nHELLO_FIM_DFH\n Address (0x50000)\nDFH value (0x3000000400000fff)\nREAD64: address=0x00090000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010001005\n\nFME_PR_DFH\n Address (0x90000)\nDFH value (0x3000000010001005)\nREAD64: address=0x00091000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010001001\n\nPORT_DFH\n Address (0x91000)\nDFH value (0x4000000010001001)\nREAD64: address=0x00092000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000014\n\nUSER_CLOCK_DFH\n Address (0x92000)\nDFH value (0x3000000010000014)\nREAD64: address=0x00093000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000d0002013\n\nPORT_STP_DFH\n Address (0x93000)\nDFH value (0x30000000d0002013)\nREAD64: address=0x000a0000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000010000002010\n\nAFU_INTF_DFH\n Address (0xa0000)\nDFH value (0x3000010000002010)\nMMIO error count matches: x\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_fme_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\nEdit syn/syn_top/d5005.qsf
Add new macro \"INCLUDE_HELLO_FIM\" line 107
set_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\"\nAdd new line 211 to source TCL script with new hello_fim files
set_global_assignment -name SOURCE_TCL_SCRIPT_FILE ../../../syn/setup/hello_fim_design_files.tcl\nCreate \"hello_fim_design_files.tcl\" file and store in the syn/setup directory. This tcl file is called from d5005.qsf.
# Copyright 2021 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE src/hello_fim/hello_fim_top.sv\nWith the preceding changes complete, build the new hello_fim example using the following steps:
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh d5005 work_d5005_hello_fim\nVerify the design successfully compiled and timing closure is achieved by checking work_d5005_hello_fim/syn/syn_top/output_files/timing_report/clocks.sta.fail.summary - this file should be empty. If there are timing failures, then this file will list the failing clock domain(s).
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#519-test-the-hello_fim-on-a-d5005","title":"5.1.9. Test the hello_fim on a D5005","text":"Load the built FPGA binary file using an unsigned image. The FPGA image will be in work_d5005_hello_fim/syn/syn_top/output_files/d5005_page1_unsigned.bin
Provide the file d5005_page1_unsigned.bin on the server with the Intel\u00ae FPGA PAC D5005.
sudo fpgasupdate d5005_page1_unsigned.bin <D5005 PCIe B:D.F>\nsudo rsu bmcimg <D5005 PCIe B:D.F>\nsudo fpgainfo fme\n## Output\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.13 Board Management Controller, MAX10 Build version: 2.0.8 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x40100022C164DB1\nBitstream Version : 4.0.1\nPr Interface Id : 7d91e0d0-4dcd-58c3-a93d-b9295e6e29b0\nBoot Page : user\nUse the OPAE SDK tool opae.io to check default driver binding using your card under test PCIe B:D.F. The steps below will use 0000:12:00.0 as the card under test PCIe B:D.F.
sudo opae.io init -d 0000:12:00.0 $USER\n##Output\n[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: dfl-pci)\n sudo opae.io init -d 0000:12:00.0 $USER\n##Output\nopae.io 0.2.3\nUnbinding (0x8086,0xbcce) at 0000:12:00.0 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:12:00.0 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:12:00.0 is 35\nAssigning /dev/vfio/35 to $USER\nopae.io ls\n## Output\nopae.io 0.2.3\n[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: vfio-pci)\nopae.io walk -d 0000:12:00.0\n## Output\nopae.io 0.2.3\noffset: 0x0000, value: 0x4000000010000000\n dfh: id = 0x0, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x1000, value: 0x3000000020000001\n dfh: id = 0x1, rev = 0x0, next = 0x2000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x3000, value: 0x3000000010000007\n dfh: id = 0x7, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x4000, value: 0x30000000c0001004\n dfh: id = 0x4, rev = 0x1, next = 0xc000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x10000, value: 0x300000010000000e\n dfh: id = 0xe, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000100000020\n dfh: id = 0x20, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x30000, value: 0x300000010000100f\n dfh: id = 0xf, rev = 0x1, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x40000, value: 0x3000000100000009\n dfh: id = 0x9, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x50000, value: 0x3000000400000fff\n dfh: id = 0xfff, rev = 0x0, next = 0x40000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x90000, value: 0x3000000010001005\n dfh: id = 0x5, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x91000, value: 0x4000000010001001\n dfh: id = 0x1, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x92000, value: 0x3000000010000014\n dfh: id = 0x14, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x93000, value: 0x30000000d0002013\n dfh: id = 0x13, rev = 0x2, next = 0xd000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0xa0000, value: 0x3000010000002010\n dfh: id = 0x10, rev = 0x2, next = 0x0, eol = 0x1, reserved = 0x0, feature_type = 0x3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50000\nopae.io 0.2.3\n0x3000000400000fff\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0x0\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50038\nopae.io 0.2.3\n0x6626070150000034\n$ opae.io -d 0000:12:00.0 -r 0 poke 0x50038 0x123456789abcdef\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50038\nopae.io 0.2.3\n0x6626070150000034\n$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0x123456789abcdef\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0x123456789abcdef\n$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0xfedcba9876543210\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0xfedcba9876543210\n$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0x55550000aaaaffff\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0x55550000aaaaffff\nRelease the card under test from the vfio driver to re-bind to the dfl-pci driver:
sudo opae.io release -d 0000:12:00.0\n## Output\nopae.io 0.2.3\nReleasing (0x8086,0xbcce) at 0000:12:00.0 from vfio-pci\nRebinding (0x8086,0xbcce) at 0000:12:00.0 to dfl-pci\n$ sudo opae.io ls\nopae.io 0.2.3\n[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: dfl-pci)\nOFS enables modifications on the different subsystems that encompass the FIM. To customize the Memory Subsystem follow these instructions.
Set up the environment variables as described in section 4.2.1. Setting Up Required Environment Variables
Modify the NUM_MEM_CH parameter in src/afu_top/mux/top_cfg_pkg.sv Change NUM_MEM_CH from 4 to 2 as shown in below code
//=========================================================================================================================\n// OFS Configuration Parameters \n//=========================================================================================================================\nparameter NUM_MEM_CH = 2 ,// Number of Memory/DDR Channel \nNUM_HOST = 1 ,// Number of Host/Upstream Ports \nNUM_PORT = 4 ,// Number of Functions/Downstream Ports \nDATA_WIDTH = 512 ,// Data Width of Interface \nTOTAL_BAR_SIZE = 20 ,// Total Space for APF/BPF BARs (2^N) \n//------------+-------------+-------------+-----------------+ //--------------------------------------\n// VF Active | PF # | VF # | Mux Port Map | // PF/VF Mapping Parameters \n//------------+-------------+-------------+-----------------+ //--------------------------------------\nCFG_VA = 0 , CFG_PF = 0 , CFG_VF = 0 , CFG_PID = 3 , // Configuration Register Block \nHLB_VA = 1 , HLB_PF = 0 , HLB_VF = 0 , HLB_PID = 0 , // HE Loopback Engine \nPRG_VA = 1 , PRG_PF = 0 , PRG_VF = 1 , PRG_PID = 1 , // Partial Reconfiguration Gasket \nHSI_VA = 1 , HSI_PF = 0 , HSI_VF = 2 , HSI_PID = 2 ; // HSSI interface \nCompile a new FIM that incorporates the newly configured Memory Subsystem.
cd $OFS_BUILD_ROOT/ofs-d5005\nofs-common/scripts/common/syn/build_top.sh d5005 work_d5005_mem_2channel\n***********************************\n***\n*** OFS_PROJECT: d5005\n*** Q_PROJECT: d5005\n*** Q_REVISION: d5005\n*** SEED: 03\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\nProgram d5005_page1_unsigned.bin file using below command
sudo fpgasupdate d5005_page1_unsigned.bin 3b:00.0\nRun rsu command
sudo rsu bmcimg 3b:00.0\nCheck if binary was loaded correctly
fpgainfo fme\n## Output\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\nRun Host Excersiser to check Memory Subsystem performance
sudo host_exerciser mem\n## Output\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5365\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.054 GB/s\n Test mem(1): PASS\nVerify Memory controller placement in syn/syn_top/output_files/d5005.fit.place.rpt file. Open fitter place stage report in any text editor of your choice, find keyword emif in the file. You should see emif[0] & emif[1] for Memory channel 0 & 1 respectively.
|emif[0].ddr4_pr_freeze_sync| ; 0.4 (0.0) ; 0.5 (0.0) ; 0.1 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.4 (0.4) ; 0.5 (0.5) ; 0.1 (0.1) ;\n|emif[0].ddr4_softreset_sync| ; 0.5 (0.0) ; 0.7 (0.0) ; 0.2 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.5 (0.5) ; 0.7 (0.7) ; 0.2 (0.2) ;\n|emif[0].pr_frz_afu_avmm_if| ; 647.5 (647.5) ; 917.3 (917.3) ; 272.8 (272.8) ;\n|emif[1].ddr4_pr_freeze_sync| ; 0.4 (0.0) ; 0.8 (0.0) ; 0.4 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.4 (0.4) ; 0.8 (0.8) ; 0.4 (0.4) ;\n|emif[1].ddr4_softreset_sync| ; 0.4 (0.0) ; 1.0 (0.0) ; 0.6 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.4 (0.4) ; 1.0 (1.0) ; 0.6 (0.6) ;\n|emif[1].pr_frz_afu_avmm_if| ; 641.1 (641.1) ; 914.0 (914.0) ; 272.9 (272.9) ;\n|p[0].pr_frz_fn2mx_a_port| ; 435.4 (0.0) ; 476.2 (0.0) ; 40.8 (0.0) ;\n|r.axis_pl_stage[0].axis_reg_inst| ; 435.4 (435.4) ; 476.2 (476.2) ; 40.8 (40.8) ;\n|p[0].pr_frz_fn2mx_b_port| ; 434.6 (0.0) ; 494.3 (0.0) ; 59.6 (0.0) ;\nUsing the OFS reference design and OPAE SDK enables the rapid creation of market leading FPGA based Acceleration systems. OFS facilitates customization of the FIM area for your custom board or platforms.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/doc_modules/Glossary/","title":"Glossary","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/","title":"FPGA Interface Manager Technical Reference Manual: Open FPGA Stack for Stratix 10\u00ae FPGA","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1-overview","title":"1 Overview","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#11-about-this-document","title":"1.1 About this Document","text":"This document describes the hardware architecture of the\u200b Open FPGA Stack (OFS) targeting the Stratix 10 FPGA. After reviewing this document you should understand the features and functions of the components that comprise the FPGA Interface Manager (FIM), also known as the \"shell.\"
Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12-introduction-to-the-open-fpga-stack","title":"1.2 Introduction to the Open FPGA Stack","text":"The Open FPGA Stack (OFS) is a modular collection of hardware platform components, open source upstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS Provides a framework of FPGA synthesizable code, simulation environment and synthesis/simulation scripts. The key components of OFS include: - Target development platforms such as Intel-branded Programmable Acceleration Cards (PACs), Acceleration Development Platforms (ADPs) and third-party platforms.
The OFS hardware repository supports hardware development and simulation. Repositories for OFS high level design support and board management controller RTL and firmware source code are also provided. These repositories can be found in the Altera Opensource Technology GitHub location, which requires entitlement access. To request access, please contact your local Altera sales representative.
Table 1-2 OFS GitHub Repositories
OFS GitHub repositories can be found in the OFS.
Repository Contains ofs-fim-common Contains common modules shared by all OFS designs. This repository is a submodule of each platform repository. ofs-d5005 Contains FIM or shell RTL design, automated compilation scripts, unit tests.The OPAE software GitHub site is fully opensource and contains resources for both software and workload developers.
Table 1-3 OPAE Public Git Repositories
OPAE GitHub repositories can be found in the OFS.
OPAE Git Repository Folder Contains linux-dfl Contains OFS Linux drivers that are being upstreamed to the Linux kernel. linux-dfl-backport Backport versions of the linux-dfl to older kernel versions. opae-sdk Contains the files for building and installing OPAE SDK from source. opae-sim Contains an AFU/Workload simulator for software/hardware co-simulation. examples-afu Contains simple AFU tutorials.Providing the hardware and software source code and supporting test frameworks in a GitHub repository allows you to easily customize your own designs with the latest versions.
Most hardware and software ingredients are available in our OFS GitHub location. For access to the board management controller firmware and RTL or our security guide for OFS, please contact a local Altera sales representative.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#13-ofs-features","title":"1.3 OFS Features","text":"The OFS architecture within the FPGA comprises two partitions:
The FIM or shell provides platform management functionality, clocks, resets and interface access to the host and peripheral features of the acceleration platform. The FIM architecture along with the supporting OPAE software supports features such as partial reconfiguration and virtualization. The FIM provides a standard Arm* AMBA* 4 AXI4 datapath interface. The FIM resides in the static region of the FPGA.
The AFU partition is provided for custom acceleration workloads and may contain both static and partial reconfiguration regions.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#131-fpga-interface-manager-fim","title":"1.3.1 FPGA Interface Manager (FIM)","text":"The updated OFS architecture for Stratix\u00ae 10 FPGA devices improves upon the modularity, configurability and scalability of the first release of the OFS architecture while maintaining compatibility with the original design. The primary components of the FPGA Interface Manager or shell of the reference design are:
The AFU Region provides design space for custom workloads and contains both static and partial reconfiguration regions. Partial reconfiguration allows you to update your specific logic blocks or entire workload while the rest of your static design is still in operation.
Note that as discussed previously, the BMC RTL and firmware, the OFS OPAE software stack and support for building your own customer board support package are also provided in separate OFS repositories.
Figure 1-2 OFS for Stratix 10 Block Diagram
The table below details the features of the OFS release targeting the Stratix\u00ae 10 FPGA .
Table 1-4 Features
Key Feature OFS Update Comments PCIe H-tile PCIe Gen3x16 Interface Integrates PCIe TLP adapter for new data mover packet format. MSI-X vector and PBA tables are located in the PCIe subsystem. Interrupts from FME as well as four user interrupts coming from PF0.VF1 are supported. Memory Two Avalon Memory Mapped channels provided as default with capability to compile design with four channels support. - HSSI 1 Arm* AMBA* 4 AXI4-Stream channel of 10G Ethernet, using the low latency Ethernet 10G MAC FPGA IP interfacing to an E-tile PHY. - Manageability SPI interface to Board Management Controller targeting Intel FPGA PAC D5005 - CoreFIM Flexible configuration support using Arm* AMBA* 4 AXI4-Stream Physical Function/Virtual Function (PF/VF) Demux/Mux and AFU Peripheral Fabric (APF) and Board Peripheral (BPF) Fabric Interconnects. APF and BPF fabrics are Platform Designer generated IPs. The Arm* AMBA* 4 AXI4-Stream PF/VF Demux/Mux is a new component. Physical Function/Virtual 1 PF/3VF configuration is provided as an example but the architecture now supports full virtualization with the ability to expand to whatever the PCIe tile supports. - Partial Reconfiguration 1 Partial Reconfiguration region supported in hardware and software - Sample test PR AFUs Host exerciser modules provided to exercise interfaces. These modules are provided in both the flat and PR AFU examples. - OneAPI Yes Available Q1 2023 Software Support OFS software stack with support for full virtualization. -"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FIM contains only one FME, regardless of the number of host interfaces to the FIM. The FME provides management features for the platform and controls reset and loading of the AFU into the partial reconfiguration region of the FPGA.
Each FME feature exposes its capability to host software drivers through a device feature header (DFH) register found at the beginning of its control status register (CSR) space. The FME CSR maps to physical function 0 (PF0) Base address register 0 (BAR0) so that software can access it through a single PCIe link. For more information about DFHs, refer to the Device Feature Header (DFH) structure.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#streaming-datapath","title":"Streaming Datapath","text":"The FIM implements an AXI4-Stream bus protocol for data transfer in the FIM. AXI4-Stream channels send data packets to and from the host channel IP without data abstraction. Memory-mapped I/O (MMIO) CSR accesses are routed to the ST2MM module which converts the AXI4-Stream to an AXI4 memory mapped protocol.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#virtualization","title":"Virtualization","text":"This design supports virtualization by making use of the virtualization functionality in the PCIe Hard IP and mapping packets to the appropriate physical or virtual function through a PF/VF multiplexer. This reference FIM supports 1 PF and 3 VFs as an example; however, you may extend your configuration to whatever the PCIe Hard IP can support or what your application requires.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#132-afu","title":"1.3.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space. The port is part of the FPGA Interface Unit (FIU) that resides in the FIM.
You can compile your design in one of the following ways: * Your entire AFU resides in a partial reconfiguration region of the FPGA * The AFU is part of the static region and is compiled a flat design
In this design, PF0.VF1 and PF0.VF2 map to host exerciser modules (HEM) that map to HE-LB and HE-HSSI respectively.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#133-platform-interface-manager","title":"1.3.3 Platform Interface Manager","text":"The PIM provides a way to abstract the AXI4-Stream interface to the AFU by providing a library of shims that convert the host channel native packet into other protocols such as CCI-P, AXI4 memory-mapped, Avalon\u00ae streaming (Avalon-ST) or Avalon\u00ae memory-mapped (Avalon-MM). The FPGA or AFU developer implement these interface abstractions in the AFU region of the design.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#134-opae-sdk-fpga-platform-feature-discovery","title":"1.3.4 OPAE SDK FPGA Platform Feature Discovery","text":"The OPAE C library in the OPAE software development kit is built on top of the OPAE FPGA driver stack that abstracts the hardware and operating system specific details of the platform to the host. The FIM implements a DFH linked list to allow an FPGA platform driver running on the host to discover FME, port and AFU features. This model is similar to how PCIe enumeration occurs. You must implement a 64-bit DFH Device Feature Header register at the beginning (first 8B aligned address) of the feature CSR space for a new feature to be discovered or enumerated by a driver.
A driver starts the traversing by reading the DFH of the first feature from the first address on PF0 BAR0. Based on the information in the DFH, a driver can determine the CSR address range of the feature and other associated details of the feature. The end of the DFH contains a \"next DFH offset\" field that points the driver to the DFH of the next feature. The software must continue traversing the linked list until it sees the EOL (End-Of-List) bit set to 1 in the \"next DFH offset\" field it is inspecting. A 1 indicates this is the last feature in the feature set. Figure below gives a simple illustration of the feature discovery by traversing the DFH registers.
Figure 1-3 Device Feature Header Linked List Traversal
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#135-ofs-reference-design","title":"1.3.5 OFS Reference Design","text":"OFS provides FIM designs you can use as a starting point for your own custom design. These designs target a specific programmable acceleration card or development kit and exercise key FPGA device interfaces. The Stratix\u00ae 10 code line for OFS targets the Intel FPGA PAC D5005. FIM designs are released to OFS D5005 FIM Github Branch for evaluation and use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#136-fim-simulation","title":"1.3.6 FIM Simulation","text":"OFS provides a UVM environment for the FIM and a framework for new feature verification. UVM provides a modular, reusable, and scalable testbench structure by providing an API framework that can be deployed across multiple projects. The FIM testbench is UVM compliant and integrates third-party verification IPs from Synopsys that require license to use. Verification components include:
The verification infrastructure can be in the verification folder here OFS D5005 FIM Github Branch for evaluation and use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#2-ofs-high-level-architecture","title":"2 OFS High Level Architecture","text":"OFS provides distinct datapaths that simplifies the design and integration process for add or for removing interface modules:
Depending on your design goals, you can present peripherals to software as:
Figure 2-1 OFS Datapath Structure
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#3-pcie-interface","title":"3 PCIe Interface","text":"The FIM's H-tile PCIe* hard IP is a Gen3x16 design. The IP supports SR-IOV and is configured to provide one PF and three VFs. Native PCIe TLP packets are sent through the PCIe using Arm AMBA 4 AXI-4 Stream Protocol. Before they reach the AFU, however, the packets go through an adapter that converts any headers to a data mover format that is forward compatible with Agilex FPGA devices and beyond.
Figure 3-1 OFS FIM RX-TX Datapath
Some key features of the PCIe interface are:
Feature OFS for Stratix 10 Configuration Mode PCIe Gen3x16 Port Mode Native Endpoint SR-IOV 1 PF, 3 VFs MSI-X Support Yes Functional Mode Data Mover Profile Virtual+ TLP Bypass No Header Packing Scheme Simple Data Width 512-bit (64-byte) PLD Clock Frequency 250 MHz Tags Supported 128 Reordering No reordering of requests, no completion reordering Maximum Payload Size 256 Bytes Memory Requests Supported 1CL, 2CL, 4CL MMIO transaction Size 4B, 8B"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#31-receiver-datapath","title":"3.1 Receiver Datapath","text":"The path of data received by the FIM is as follows:
The PCIe Hard IP receives TLP packets from the host. Host response types can be:
* MMIO read or response * Memory write request * Interrupt response * Memory read request or response
The PCIe IP routes the TLP packets to the PCIe bridge interface in the FIM, where they get buffered into the RX FIFO.
The TLP checker in the bridge examines the packets and filters erroneous packets including packets with unsupported requests or fields. Errors are sent to the error logger and are logged as advance error reporting (AER). Error status is sent to the PCIe hard IP and to the FIM error status registers. Appropriate action is taken for the errors as described in the Reliability, Accessibility, Serviceability (RAS) and Error Handling section. The TLP checker also maintains RX buffer credits for TLP completions with data (CplD) that are received from the host in response to a memory read request (MRd request sent by the AFU). The TLP checker notifies the PCIe TX bridge when there are not enough RX buffer credits available in the PCIe RX bridge. If there are not enough credits, the PCIe TX bridge pauses MRd requests from the AFU to Host until there is availability.
The TLP checker forwards packets that pass TLP checking to an AXI4 adapter which moves the packets into an Arm AMBA 4 AXI4-Stream bus.
AXI4-Stream packets are sent downstream to datamover AXI4-Stream adapter to be modified into the new datamover packet format.
Datamover packets are sent to the PF/VF Mux in the AFU.
The AXI4-Stream to Memory Mapped (ST2MM) module in the AFU region routes MMIO requests targeting FIM CSRs (FME, peripherals and AFU).
The Transmit (TX) datapath refers to all TLP packet types that originate in the FPGA. A single Arm AMBA 4 AXI4-Stream channel at the port interface boundary of the FIM carries PCIe TLP packets and interrupt requests upstream from the AFU.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#33-data-handshaking","title":"3.3 Data Handshaking","text":"The Arm AMBA 4 AXI4 interfaces to the AFU use the VALID and READY signal for handshaking and backpressure management. The FIM holds the DATA and VALID asserted until the receiver asserts the READY signal. The AFU accepts the data when both the VALID and READY signals are asserted.
The table below shows the high-level signal mapping of the channels for the OFS for Stratix 10 FPGA. If you have previously used the OFS EA architecture, that mapping is provided as a comparison as well in this table.
Table 3-1 AXI4-Stream RX Channel
AXI4-Stream Signal Source OFS Stratix 10 Mapping OFS Early Access Mapping ACLK Clock Source PCLK = 250 MHz PCLK = 250 MHz AResetn Reset Source System Reset System Reset TVALID Master Data Valid Data Valid TREADY Slave Ready Ready TDATA Master Width=512 bits When packet includes a header (32 bytes) and data the packing scheme is: 16B data mover header: 4B prefix PF NumberVM Number VF Active Slot Number Memory Mapped Number8B Address/Metadata Data: 32B If the packet is only data then all 64 bytes comprise data [TLP_CH] [8n] n=49 (392 bits)TLP_CH=2 (2TLP data streams)Mapping of each TLP data stream [391:136] payload (32-byte data payload) [135:8]: hdr (16 byte header) [7:3]: rsvd0 (reserved bits)[2]: end of packet (eop) [1]: start of packet (sop) [0]: valid (TLP packet on data stream is valid) TLAST Master Set to 1'b1 when end of packet is sent; otherwise TLAST is 1'b0 Set to 1'b1 when end of packet is sent; otherwise TLAST is 1'b0 TKEEP Byte Qualifier Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. TUSER Master WIDTH = 10 Bit 0 is always equal to 1'b1 to indicate data mover header format [4:1] - Indicates header position on the TDATA bus and is always equal to 4'b0001 indicating that the header starts from Byte0.All other bits of TUSER are unused. [TLP_CH][u-1:0] u=21 Sideband of each TLP data stream[20]: ummio_rd (Unsupported MMIO request)[19:0]: destination routing ID, where:[19:17]= BAR offset[2:0][16:4]=VF number[12:0][3:1]=PF number[2:0][0]=vf_active, indicating if the virtual function feature is enabledFigure 3-3 AXI4-Stream RX Request Cycle Header Format
All Host requests sent to the AFU are memory-mapped I/O requests. Of the fields below, the following are not supported in the design: * Prefix * Slot number (set to 0) * Local Address
Figure 3-4 AXI4-Stream RX Completion Header Format
All completions in the RX direction are data completions. Of the fields below, the following are not supported in the design: * Prefix * MM mode * Slot number (set to 0) * Meta Data
Note that: * VF Active, VF Num and PF Num are obtained from TUSER. * Data packet responses (for memory read requests from the AFU) from the PCIe may come out of order when the size is greater than 64 bytes.
Table 3-2 AXI4-Stream TX Channel
AXI4-Stream Signal Source OFS Stratix 10 Mapping OFS Early Access Mapping ACLK Clock Source PCLK = 250 MHz PCLK = 250 MHz AResetn Reset Source System Reset System Reset TVALID Master Data Valid Data Valid TREADY Slave Ready ReadyOnly 1 ready signal for the two channels TDATA Master Width=512 bits When packet includes a header (32 bytes) and data the packing scheme is: 16B data mover header: 4B prefix PF NumberVM Number VF Active Slot Number Memory Mapped Number8B Address/Metadata Data: 32B If the packet is only data then all 64 bytes comprise data [TLP_CH] [8n] n=49 (392 bits)TLP_CH=2 (2TLP data streams)Mapping of each TLP data stream [391:136] payload (32 byte data payload) [135:8]: hdr (16 byte header) [7:3]: rsvd0 (reserved bits)[2]: end of packet (eop) [1]: start of packet (sop) [0]: valid (TLP packet on data stream is valid) TLAST Master Per protocol Set to 1'b1 TKEEP Byte Qualifier Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. TUSER Master WIDTH = 10 Bit 0 \u2013 Indicates Header Format: 0 \u2013 Power user mode header format 1 \u2013 Data mover header format Bit [4:1] - Indicates header position on the TDATA bus and is always equal to 4'b0001 indicating that the header starts from Byte0.All other bits of TUSER are unused. [TLP_CH][u-1:0] u=2Sideband of each TLP data stream[0]=vf_active, indicating if the virtual function feature is enabled[0]: afu_irq (AFU interrupt)Table 3-3 Interrupt Response Channel
AXI4-Stream Signal Source Mapping ACLK Clock Source PCLK AResetn Reset Source System Reset TVALID Master Valid TREADY Slave Ready TDATA [8n-1:0] n=3 (24 bits) Master [23:16]: 8-bit interrupt ID [15:0]: Requester IDFigure 3-5: AXI4-Stream TX Request Cycle Header Format
All requests in the TX direction are Memory Read/Write. The requester ID does not come from the AFU; the AXI-Stream adapter supplies it. The tag must come from the AFU. Of the fields below, the following are not used in the H-Tile PCIe subsystem design: * Prefix * MM Mode * Slot number (set to 0) * Local Address
Note that VF Active, VF Num and PF Num are obtained from the header packet.
Figure 3-4 AXI4-Stream TX Completion Header Format
All completions in the TX direction are for MMIO. Of the fields below, the following are not supported in the design: * Prefix * MM mode * Slot number (set to 0) * Meta Data
Note that: * VF Active, VF Num and PF Num are obtained from TUSER.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#4-platform-interface-manager","title":"4 Platform Interface Manager","text":"The FIM interfaces to an AFU through AXI4-Stream channels. This format allows the AFU to access the host channel's raw interface without any translation. As a FIM developer, you have the option to provide the raw data format associated with the host interface channel to the workload or AFU developer or you can provide an intermediate protocol using Platform Interface Manager Components or your own custom interface. If you expose the raw AXI4-Stream interface of the FIM, workload developers also have the option to convert to a desired protocol using the PIM resources as well.
Refer to https://github.com/OPAE/ofs-platform-afu-bbb for more information on options for implementing the PIM.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#5-afu-interface-handler","title":"5 AFU Interface Handler","text":"The AFU Interface Handler resides inline between the PCIe AXI4-Stream Adapter and the AXI4-Stream PF/VF Demux/Mux logic. Its main function is to provide: * Unique PCIe tags \u2013 Each PCIe transaction shares the 128 tags across all VFs in the AFU region * AFU error logging for all VFs in the AFU region
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#51-afu-error-handling","title":"5.1 AFU Error Handling","text":"In this OFS design, the AFU Interface Handler handles error logging for all VFs in the AFU. Errors handled are as follows
Checker Field Description AFU protocol checker (PCIe TLP) TxReqCounterOverflow Pending memory write or memory read requests exceed the predefined limit TxFifoOverflow Tx FIFO in the port that buffers TLP packets from AFU is overflow Malformed TLP AFU PCIe TLP contains unsupported format type MaxPayloadError AFU memory write payload size exceeds max_payload_length limit MaxReadReqSizeError AFU memory read payload size exceeds max_read_request_size limit MaxTagError AFU memory read request tag value exceeds the maximum supported tag count TagOccupiedErr AFU sends out memory read request using a tag that is already used for a pending memory read request UnalignedAddrErr The address field in AFU memory write/read request TLP is not DW-aligned. UnexpMMIOResp AFU is sending a MMIO read response with no matching MMIO read request. MMIOTimedOutAFU is not responding to a MMIO read request within the pre-defined response timeout period. MMIODataPayloadOverrunThe number of data payload sent by AFU for a MMIO response (cplD) is more than the data length specified in the response. MMIOInsufficientDataThe number of data payload sent by AFU for a MMIO response (cplD) is less than the data length specified in the response. TxMWrDataPayloadOverrun The number of data payload sent by AFU for a memory write request is more than the data length specified in the request. TxMWrInsufficientData The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU Protocol Checker (AXI4-Stream)TxValidViolationThree checkers are implemented in the FIM to catch errors and protocol violations.To view the CSR space for the AFU interface handle, go to the src/afu_top/AFU_INTF_CSR.xls file OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#6-interconnect-fabric","title":"6 Interconnect Fabric","text":"There are three types of interconnect fabric in the OFS FIM design: * AXI4-Stream mux/demux fabric * AFU Periheral Fabric (APF) * Board Peripheral Fabric (BPF)
Figure 6-1 Interonnect Fabric Diagram
TLP packets sent from upstream PCIe Subsystem on AXI4-Stream channel are demultiplexed in the AXI4-Stream PF/VF mux/demux fabric and routed to the respective PF/VF function based on the PF/VF information in the TLP header, such as vf_active or the PF/VF number. On the opposite direction, TLP packets from downstream PF/VF function are muxed in the fabric and sent to PCIe subsystem over AXI4-Stream channel.
All host MMIO requests targeting PF0 BAR0 are routed to the ST2MM module. The ST2MM converts MMIO TLP packets into AXI-Lite memory requests and places the requests onto AFU Peripheral Fabric (APF). AFU peripherals, such as OFS managed AFU features and ST2MM) and Board Peripheral Fabric (BPF) are interconnected by APF. The BPF is the interconnect fabric one hiearchy below APF which connects all the board peripherals. Both APF and BPF allow multiple AXI4-Lite master and slave interconnect topology.
The following table summarizes the mechanism for configuring PF/VF functions:
Table 6-1 Interconnect Configuration Methods
InterconnectConfiguration Mechanism APF or BPFEither:Use Platform Designer (PD) to generate the fabrics directly, or Specify desired cfg in iofs_dfl.txt and run dfh2tcl.pl script to generate the necessary HW TCL scripts as needed by PD. This PERL script also takes care of invoking PD in script mode to generate the end result of RTL design Verilog files. This is the preferred method for generation. AXI-S PF/VF Demux/MuxUpdate parameters in these RTL files: * src/includes/top_cfg_pkg.sv * src/common/pf_vf_mux.sv Then make the corresponding update to AFU top level instantiation and connections: * src/FIMs/.../afu_top.svNote:
In all cases, you must configure and regenerate the PCIe IP to match the new PF/VF configuration if it deviates from the reference FIM provided.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#61-fabric-generation-flow","title":"6.1 Fabric Generation Flow","text":"You have two options to generate APF/BPF IPs.
iofs_dfl.txt and automatically generates APF/BPF IPs. Both APF and BPF are generated from Platform Designer using hardware TCL scripts. To provide a more user friendly experience to OFS Rel 1 and AC ADP customers, a perl script dfh2tcl.pl has been developed to provide a higher level of abstraction. The below figure illustrates the high level flow:Figure 6-2 APF/BPF Generation
Note that the only input required is the iofs_dfl.txt text file which allows you to specify how many ports, which fabric, type of AXI4-Lite port (master, slave, or both), port addresses and sizes. Using iofs_dfl.txt and dfh2tcl.pl to generate the APF and BPF is the preferred method.
The following table describes the format of the iofs_dfl.txt input file:
Table 6-2 iofs_dfl.txt Format Types
ColumnDescription REGISTERName of the port on the APF/BPF fabrics. Note that each entry must be unique. FABRICAllows the user to specify which fabric this port is connected to, and also the type of AXI4-Lite port (master, slave, or both). The format is FABRIC-PORT_TYPE, where: * FABRIC = APF or PBF * PORT_TYPE = MST, SLV, or BID (Master, Slave, or Bi-Directional) BASE_ADDRSpecifies the address offset of the AXI4-Lite port. BAR_SIZE=2^N Specifies the size of the port. For simplicity reasons, all AXI4-Lite ports are configured as 64KB. i.e. 2^16Additional AXI4-Lite ports can be easily created by adding more rows in the iofs_dfl.txt file.
Note that there are several reserved ports in both APF and BPF so it might not be necessary for you to regenerate the fabrics if the design does not exceed the number of ports as implemented in the APF/BPF in the reference FIMs.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#63-afu-peripheral-fabric-apf","title":"6.3 AFU Peripheral Fabric (APF)","text":"The AFU Peripheral Fabric (APF) is a 64-bit AXI4-lite compliant interconnect fabric that connects AFU peripheral modules to board peripheral modules through the Board Peripheral Fabric (BPF). The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by the APF is listed below. All components are mapped to PF0 BAR0 and implement AXI-lite slave interface. The Master column indicates if a component also implements AXI4-lite master interface which can send request to APF.
Table 6-3 APF Address Mapping
AddressSize (Byte)FeatureMaster 0x00000 \u2013 0x7FFFF512KBoard Peripherals (See BPF address mapping) No AFU Peripherals 0x80000 \u2013 0x8FFFF64KST2MMYes (Send MMIO request to all the peripherals) 0x90000 \u2013 0x9FFFF64KPort GasketYes 4KPR Control & Status 4KUser clock 16KRemote STP 0xA0000 \u2013 0xAFFFF64KAFU Interface HandlerNo 0xB0000 \u2013 0xBFFFF64KRSV_b_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xC0000 \u2013 0xCFFFF64KRSV_c_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xD0000 \u2013 0xDFFFF64KRSV_d_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xE0000 \u2013 0xEFFFF64KRSV_e_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xF0000 \u2013 0xFFFFF64KRSV_f_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address.The five reserved regions have associated ports available in the APF for customer use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#64-board-peripheral-fabric-bpf","title":"6.4 Board Peripheral Fabric (BPF)","text":"The Board Peripheral Fabric is the 64-bit AXI4-Lite compliant interconnect fabric that connects board peripheral modules to APF. The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by BPF is listed below. All components are mapped to PF0 BAR0 and implement AXI4-lite slave interface. The Master column indicates if a component also implements AXI4-lite master interface which can send request to BPF.
Table 6-4 BPF Address Mapping
AddressSize (Byte)FeatureMaster 0x00000 \u2013 0x0FFFF64KFME (FME, Error, etc)Yes 0x10000 \u2013 0x1FFFF64KSPI ControllerYes 0x20000 \u2013 0x2FFFF64KPCIe CSR- 0x30000 \u2013 0x3FFFF64KHSSI CSR- 0x40000 \u2013 0x4FFFF64KEMIF CSR- 0x50000 \u2013 0x5FFFF64KReservedAvailable for customer use.Dependent on user programming 0x60000 \u2013 0x6FFFF64KReservedAvailable for customer use.Dependent on user programming 0x70000 \u2013 0x7FFFF64KReservedAvailable for customer use.Dependent on user programmingThe three reserved regions have associated ports available in the BPF for customer use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#65-axi4-stream-pfvf-muxdemux","title":"6.5 AXI4-Stream PF/VF Mux/Demux","text":"The AXI4-Stream PF/VF Mux/Demux routes the PCIe TLP packets from the PCIe subsytem AXI4-Stream RX channel to downstream PF/VF based on the pf_num and vf_num information in the PCIe TLP header.
The AXI4-Stream PF/VF mux arbitrates PCIe TLP packets from downstream PF/VF to the PCIe SS AXI-S TX channel. The PF/VF Mux/Demux is an M x N switch that allows any M port to target any N port, and any N port to target any M port, where M is the number of host/upstream ports, and N is the numbers functions/downstream ports. M and N values are parameterized in the RTL for adding, removing, or remapping of FPGA functional units/modules to PF/VF.
The fpga top package file, found in the src/includes/ofs_fim_cfg_pkg.sv file OFS D5005 FIM Github Branch contains these parameters as well as the mapping of N port\u2019s PF/VF.
Structurally, M x N switch is composed of M number of N:1 mux, and N number of M:1 mux. Each mux output has an arbiter that perform round robin priority arbitration of its inputs. At the mux output is a FIFO with depth greater than the handshake round trip delay. The FIFO allows the switch to arbitrarily insert pipeline/register stages for timing.
Note that M x N switch is design for AXI streaming, but it can be easily converted to AVST. The protocol signals pass through switch intact \u2013 only ready, valid, and last (common between AVST and AXI) effect switch operation. The data width of the switch is also parameterized in the src/includes/ofs_fim_cfg_pkg.sv file OFS D5005 FIM Github Branch.
The default mapping is shown below:
Table 6-5 PF/VF Mapping
DevicePhysical Function #Virtual Function #Switch Port ID APF/BPF0x3 HE Loopback000 Port Gasket011 HSSI022For information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#66-unified-tag-remapping","title":"6.6 Unified Tag Remapping","text":"When a FPGA function sends out a read cycle, it allocates a unique tag which is subsequently used to identify the read completion. The tag is considered busy; it cannot be assigned to another read cycle until read completion. While a tag may be unique within a unit, two different units could unknowingly send out two read cycles of the same tag. The PCIe subsystem requires unique tags for all read cycles irrespective of their origins. Therefore, a mechanism is needed to uniquify tag globally across different units.
OFS contains a tag remapper (tag_remap) that intercepts the read cycle, finds a globally unique tag, and replaces the original tag value. It also restores the original tag value when returning completion to the read requester. tag_remap is placed between the AXI4-Stream interface of the PCIE subsystem and the PF/VF Mux/Demux.
The logic is described as follows:
ST2MM implements the following key features: * Host MMIO bridge * Maps MMIO TLP packets received from the PCIe Subsystem over streaming interface to AXI4-Lite memory-mapped request. The memory-mapped request is sent to AFU or Board peripherals over APF and BPF. * Maps AXI4-lite MM response received from AFU or Board peripherals to TLP packets and send the packets over ST streaming channel to host HIA subsystem. * Sends MMIO response of all 0\u2019s for MMIO read to unused BAR region. * Interrupt * Sends interrupt packets to the PCIe subsystem when interrupt requests are received from the peripherals. Interrupts can be requested by a peripheral through a memory write to interrupt CSR registers in the ST2MM.
Figure 6-2 APF/BPF Generation
ST2MM implements both AXI4-lite master and slave interfaces that are connected to the designated slave and master port on APF. Host memory requests are sent on the ST2MM master interface to AFP where the requests are routed to the targeted peripherals.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#7-mmio-regions","title":"7 MMIO Regions","text":"The FIM and AFU expose their functionalities to the host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). An MMIO region is an address space within a base address register (BAR) region to which features are memory mapped. For example, when a feature is mapped to an MMIO region, the CSR registers of that feature are located within the address range of that region. There can be multiple MMIO regions within a BAR region.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#71-base-address-register-bar-layout","title":"7.1 Base Address Register (BAR) Layout","text":"The function, BAR and external feature region starting address are put into a platform specific parameter SystemVerilog package file src/includes/ofs_fim_cfg_pkg.sv file OFS D5005 FIM Github Branch.
You can modify the parameterization according to your platform requirements, however you must ensure the corresponding software driver is also updated to align with the new assignment.
Table 7-1 BAR Layouts
PF VF Feature BAR BAR Size PF0 - OFS Managed Peripherals BAR 0 512K AFU PeripheralsBoard Peripherals 256K256K VF0 HE-LB BAR0 4K VF1 HE-MEM in PR slot BAR0 4K VF2 HE-HSSI BAR0 4K"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#72-feature-region","title":"7.2 Feature Region","text":"A group of related CSRs can be categorized as a feature region. For example, a DMA engine has queue management function and quality of service (QoS) function; these are two different features of the DMA engine. A feature region is contained within a single PCIe BAR and cannot span across two BAR region boundaries. You can view the PF0 BAR0 MMIO mapping by referencing thesrc/common/fme/fme_csr_pkg.sv file OFS D5005 FIM Github Branch file.
A Device Feature Header (DFH) register marks the start of the feature region and sub-feature region, and you must place it at the first address of the region. Each DFH starts at 4KB boundary. A DFH register contains information that OPAE software requires to enumerate the feature. It also has an offset field that points to the next DFH in a feature list. OPAE software traverses the linked list of DFHs in each BAR region to discover all the features implemented on the platform. The EOL field in a DFH marks the end of a DFH list and is only set in the DFH of the last feature in the feature list. The feature type field in the DFH is used to differentiate between the different types of feature region. Basic building blocks (BBB) and private features are always a child of an AFU or FPGA Interface Unit (FIU) and must be contained within an AFU or FIU, respectively.
All DFHs must follow the following structure to be compatible with OPAE software.
Table 7-2: DFH Structure
Bitfield Name Range Access Description FeatureType 63:60 RO 4\u2019b0000 \u2013 Reserved 4\u2019b0001 \u2013 AFU4\u2019b0010 \u2013 BBB4\u2019b0011 \u2013 Private Feature4'b0100 \u2013 FIU/FIM Reserved 59:41 Rsvd Reserved EOL 40 RO End of DFH List1'b0=No other feature header beyond this one1'b1=This is the last feature header NextDFHByteOffset 39:16 RO Next DFH byte offsetNext DFH Address= Current DFH address + Next DFH byte offset. You can also use this value as an indication of the maximum size of the MMIO region occupied by this feature. FeatureRev 15:12 RO For AFU Feature type= AFU major version number that is user defined.All other feature types= Feature revision number FeatureID 11:0 RO For AFU feature type= CoreFIM version numberFor BBB feature type= Intel defined ID for BBBFor private feature type= User-defined ID to identify within an AFU For FIU type=ID for FIU unit (ex. 0 for FME, 1 for Port)You must increment a feature revision number if a feature changes. This change requires a corresponding change in the software to detect the new version and report mismatches between the hardware and software revision number.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#73-control-and-status-registers","title":"7.3 Control and Status Registers","text":"All the Control and Status Registers (CSRs) in the FIM are 64-bit registers with the following MMIO write and MMIO read support.
Table 7-3: CSR MMIO Read and Write Support
Request Memory Attribute Payload size Memory Ordering MMIO Write UC 4B or 8B Strongly ordered MMIO Read UC 4B or 8B Strongly orderedThe FIM does not reorder the MMIO requests or responses. For MMIO writes, there is no reordering of requests in FIM, and UC ordering rules are followed. Similarly, for MMIO reads, there is no re-ordering of requests or responses in the FIM. An AFU may opt to re-order the MMIO read responses but the FIM does not enforce read response ordering.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#731-software-access-to-registers","title":"7.3.1 Software Access to Registers","text":"In the following two cases, the FIM terminates MMIO Read requests by sending a completion with the data (CplD) specified below: * MMIO Timeout: This occurs when the AFU does not respond within a set timeout. The timeout value is currently configured to 512 pclks (clk_2x). In this case, the FIM returns all 1s.
Table 7-4: OFS Register Attribute Definitions
Attribute Expansion Description RW Read/Write This bit can be read or written by software. RO Read Only The bit is set by hardware only. Software can only read this bit. Writes do not have any effect. RW1C Read/ Write 1 to Clear Software can read or clear this bit. The software must write 1 to clear this bit. Writing zero to RW1C bit has no effect. Note that a multi-bit RW1C field may exist. In this case, all bits in the field are cleared if a 1 is written to any of the bits. RW1S Read/ Write 1 to Set Software can read this bit. Writing a 1 to the bit sets it to 1. Writing a 0 has no effect. It is not possible for software to set this bit to 0. The 1 to 0 transition can only be performed by HW. RW1CS Read/Write 1 to Clear Sticky Software can read and clear this bit. Writing a 1 to a bit clears it, while writing a 0 to a bit has no effect. This bit is only reinitialized to its default value by a power-on reset. RWD Read/Write Sticky across Hard Reset The bit can be read or written by SW. This bit is sticky or unchanged by any reset type, including Hard Reset. The bit gets cleared only with power on. *S Sticky across Soft Reset The bit will be sticky or unchanged by soft reset. These bits are only re-initialized to their default value by a power-on reset. *D Sticky across Hard Reset The bit is sticky or unchanged by or unchanged by any reset type, including hard reset. The bit gets cleared only with power on. Rsvd Reserved Reserved for future definitions. Currently don\u2019t care bits. RsvdP Reserved and Protected Reserved for future RW implementations. The software must preserve the value of this bit by read modify write. RsvdZ Reserved and Zero Reserved for future RW1C implementations. The software must write zero to this bit."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#733-csr-offset-in-bars","title":"7.3.3 CSR Offset in BARs","text":"The table below captures the FIM and AFU features in the supported BAR regions. The offset highlighted in red indicates the first DFH in the DFH list of a BAR region where device driver starts the DFH traversal.
Table 3-6: PF0 BAR0 Features
Offset Feature CSR set0x00000 FME 0x03000 Global Performance 0x04000 Global Error 0x10000 SPI Controller 0x20000 PCIe CSR Interface 0x30000 HSSI CSR Interface 0x40000 EMIF CSR Interface 0x80000 Reserved for ST2MM Bridge 0x90000 PR Control & Status (Port Gasket) 0x91000 Port CSRs (Port Gasket) 0x92000 User Clock (Port Gasket) 0x93000 Remote SignalTap (Port Gasket) 0xA000 AFU Errors (AFU Interface Handler) Table 3-7: PF0 BAR4 Features
Offset Feature CSR set 0x02000 MSI-X 0x03000 MSI-X PBA TablesTable 3-8: PF0-VF0 BAR0 Features
Offset Feature CSR set 0x00000 HE-LBKTable 3-9: PF0-VF1 BAR0 Features
Offset Feature CSR set 0x00000 HE-MEMTable 3-10: PF0-VF2 BAR0 Features
Offset Feature CSR set 0x00000 HE-HSSI"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#8-fim-clocks","title":"8 FIM Clocks","text":"The following table provides the clocks available in the OFS reference design that targets the Intel FPGA PAC D5005. Clocks that the high speed serial interface (HSSI) or external memory interface provide to the FIM may be different depending on if you modify your external features with different components.
Table 8-1: External Clock Source
Clock Frequency Description SYS_RefClk 100 MHz Reference clock to system IOPLL (sys_pll) which provides FIM system clocks. qsfp*_644_53125_clk 644.5312 5MHz HSSI reference clocks ddr4_mem*.ref_clk 150 MHz Reference clocks to DDR4 interfaces PCIE_REFCLK 100MHz PCIe reference clockTable 8-2: Internal Clocks
Clock Frequency Description clk_1x 250 MHz Generated by the system IOPLL (sys_pll). This clock drives CoreFIM datapath and the AFU interface. clk_div2 125 MHz Generated by the system IOPLL, synchronous to clk_1x. This clock drives IM datapath and AFU interface. clk_100 100 MHz Generated by the system IOPLL, synchronous to clk_1x. This clock also supplies the HSSI reconfiguration clock. avl_clk 250 MHz PCIe H-tile Hard IP clock output. This clock is not synchronous to Clk_* DDR4x_USERCLK 299.76 MHz Each of the four DDR interfaces generates one of these clocks, which provides the the clock to the DDR4* Avalon Memory Mapped interfaces. Your memory clock output may be different depending on the memory interface you are implementing in your design. uclk_usr User defined Provides an AFU user clock running at a user specified frequency. Generated by user IOPLL. Not synchronous to clk_*. uclk_usr_div2 uclk_usr/2 Second user clock to AFU running at half the frequency of uclk_usr. Synchronous to uclk_usr. Generated by user IOPLL. hssi[*].f2a_tx_parallel_clk_x1 156.2 MHz 1x TX clock generated from fPLL in the HSSI module in FIM, used to clock the HSSI TX datapath in AFU. hssi[*].f2a_tx_parallel_clk_x2 312.5 MHz 2x TX clock generated from fPLL in the HSSI module in FIM, used to clock the HSSI TX datapath in AFU. hssi[*].f2a_rx_clkout 322.265625MHz RX parallel clock from the HSSI PHY channels, used to clock the HSSI RX data datapath in AFU."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#9-reset","title":"9 Reset","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#91-reset-signals","title":"9.1 Reset Signals","text":"The system reset of OFS reference platform is driven by nPERST pin, pcie_reset_status signal from the PCIe hard IP, the INIT_DONE and nCONFIG pins of the FPGA, and the locked signal of the SYS IOPLL that provides system clocks to FIM.
Upon power-on, the reset module in the FIM holds the FIM in reset until all the reset conditions are de-activated:
nPERST signal is asserted.INIT_DONE pin is driven high to indicate core configuration is complete.The reset module places the FIM back into reset if any of these conditions becomes active again. The only way to invoke a system reset to the FIM after power-up is to deassert the nPERST pin either by performing a warm reboot or through PCIe driver intervention. There are soft reset signals set aside to allow software to reset the Port, AFU and partial reconfiguration IP.
Table 9-1: FIM System Resets
Reset DescriptionnPERST pin Active low PCIe reset pin that serves as the system reset pin on the platform. nCONFIG pin Active low input to the FPGA that causes the FPGA to lose its configuration data, enter a reset state, and tri-state all I/O pins. Host software must reload the FPGA FIM after nCONFIG is activated. ninit_done Active low signal derived from the INIT_DONE pin which indicates the FPGA core configuration is complete and has entered usermode. pcie_reset_status Active high reset status from PCIe hard IP. When driven high, this signal indicates that the PCIe IP core is not ready for usermode. pll_locked Active high SYS IOPLL locked signal Table 9-2: Soft Resets
Soft Reset Bitfield Register DescriptionPortSoftReset PORT_CONTROL[0] Resets Port and AFU. FlrPortReset PORT_CONTROL[3] PCIe function level reset that resets Port and AFU when SR-IOV is enabled. PRReset FME_PR_CTRL[0] Resets the partial reconfiguration (PR) controller."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#92-platform-power-up-sequence","title":"9.2 Platform Power up Sequence","text":"Upon power up, the HSSI interfaces of the FIM go through an internal reset and calibration sequence. After the nPERST is de-activated, the PCIe interface and EMIF interfaces are first released from reset, followed by SYS IOPLL when the ninit_done is de-asserted. The rest of the FIM logic is still being hold in reset. The nPOR signal to the PCIe hard IP de-activates following nPERST assertion, which releases PCIe hard IP from reset. The PCIe hard IP asserts pld_clk_inuse to indicate to the application layer that the HIP transaction layer is using the pld_clk as its clock and is ready for operation with the Application layer (pld_clk is stable). Finally, reset_status from PCIe IP is de-asserted. When SYS IOPLL is locked and the reset_status from the PCIe interface is de-asserted, the FIM is released from reset while the Port and AFU is still held in reset by the PortSoftReset register bit (PORT_CONTROL[0]) that is held high until software writes a 0 to this bit to de-activate port reset. At this point, the platform is fully released from reset and PCIe link is going through link training and PCIe enumeration. Once the platform is successfully enumerated, driver can then be loaded to start the device feature discovery process and provision the platform for AFU application usage.
The OFS platform supports interrupt through MSI-X feature. The OFS reference platform supports at least 4 FME interrupts (PF only) and 4 AFU interrupts (PF and VF).
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#101-msi-x","title":"10.1 MSI-X","text":"In the default implementation the MSI-X feature that handles FME and AFU interrupts is inside the PCIe Subsystem. The MSI-X vector table and Pending Bit Array (PBA) table for PF0 and PF0/VF1 are provided as an example. FME interrupts are primarily used to notify the host of error events occurring in the FIM.
All interrupt requests arrive inband through the AXI4-Stream interface to the TX AXI4-stream adapter inside the PCIe Subsystem.
An AFU sends an interrupt to the MSI-X module on the AXI interrupt request channel. After the interrupt request is serviced, the MSI-X module sends an interrupt response back to the AFU on the AXI interrupt response channel. The AFU has the flexibility to define the use of each AFU interrupt.
The following interrupts are supported: PF supports 7 interrupt vectors: - 0-3: User AFU triggered - 4: Port error triggered - 6: FME error triggered
VF supports 5 interrupt vectors: - 0-3: User AFU triggered - 4: Port error triggered
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#11-external-memory-interface-emif","title":"11 External Memory Interface (EMIF)","text":"There are four DDR4 external memory interfaces on the OFS EA FIM that targets the Intel FPGA PAC D5005 FIM for each of the four DDR4 banks (DDR4a, DDR4b, DDR4c, and DDR4d). Two of the DDR4 external memory interfaces and the associated clocks are directly exposed to AFU except for two Avalon Memory Mapped pipeline bridges to facilitate timing closure across PR boundary. The Avalon Memory Mapped interfaces of each external memory interface are connected to an Avalon-MM pipeline bridge (avmm_bridge) in the FIM, which is then connected to another Avalon-MM pipeline bridge in the PR or AFU region. An AFU should use the USER_CLK associated with a memory interface when interfacing with the memory interface for better timing performance.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#111-emif-csr","title":"11.1 EMIF CSR","text":"The CSR for the EMIF feature is memory mapped to the FME BAR region. Following table captures the EMIF CSR registers.
Table 9-1: EMIF CSR Registers
EMIF_DFH 0x40000 0x3000000050000009 EMIF Management DFH FIELD NAMERANGEACCESSDEFAULT DESCRIPTION FeatureType [63:60] RO 0x3 Feature Type = Private Feature Reserved40 [59:40] RsvdZ 0x0 Reserved NextDfhByteOffset [39:16] RO 0x050000 Next DFH Byte offset FeatureRev [15:12] RO 0x0 Feature Revision FeatureID [11:0] RO 0x9 Feature Id EMIF_STAT 0x40008 0x0000000000000000 EMIF Status FIELD NAMERANGEACCESSDEFAULT DESCRIPTION Reserved [63:16] RsvdZ 0x0 Reserved Reserved [15:12] RsvdZ 0x0 Reserved EmifCalFail [11:8] RO 0x0 EMIF PHY Calibration Failure (1 bit per interface) Reserved [7:4] RsvdZ 0x0 Reserved EmifCalSuccess [3:0] RO 0x0 EMIF PHY Calibration Successful (1 bit per interface)\u200b \u200b
EMIF_CTRL 0x40010 0x0000000000000000 EMIF Control FIELD NAMERANGEACCESSDEFAULT DESCRIPTION Reserved [63:0] RsvdZ 0x0 Reserved"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#92-afu-emif-interface","title":"9.2 AFU EMIF Interface","text":"The FIM exposes 576-bits of Avalon Memory-Mapped data to the AFU, with 512-bit data and additional 64 bits that can either be used for additional metadata, parity or ECC. The AFU has the flexibility to decide the use of the extra 64 bits of data. The ECC soft IP is not enabled in the EMIF IP to allow for the afore-mentioned flexibility. AFU developers can implement the ECC logic in the AFU by making use of the extra 64-bit of data. Avalon Memory-Mapped is the native interface protocol used by EMIF IP. AFU developers who desire other interface protocol in their designs over Avalon Memory-Mapped, such as AXI4 Memory-Mapped, can leverage the bridge in the PIM library.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12-hssi-subsystem","title":"12 HSSI Subsystem","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#121-hssi-subsystem-overview","title":"12.1 HSSI Subsystem Overview","text":"The high speed serial interface (HSSI) subsystem architecture provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack. This reference FIM contains the Low Latency Ethernet 10G MAC IP and provides a Linux driver that can be leveraged for customization.
The HSSI design is leveraged from our OFS EA release so prior customers can easily maintain compatibility with past designs.
A host exerciser, named he-hssi, is provided in the pr_slot of the AFU partition. The Ethernet interface to the AFU has an AXI4-Stream data and sideband interface. The HSSI control and status registers in the FIM are accessible by the AXI4-Stream memory mapped interface.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#122-ofs-hssi-subsystem-interfaces","title":"12.2 OFS HSSI Subsystem Interfaces","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1221-hssi-subsystem-fim-interfaces","title":"12.2.1 HSSI Subsystem FIM Interfaces","text":"There are three interfaces to the HSSI Subsystem that is part of the FIM:
The PCIe subystem uses AXI Memory Mapped accesses to read and write HSSI Control and Status Registers in the FIM. The Ethernet MAC interface typically has a data streaming interface which is mapped to standard AXI4-Stream.
Figure 12-1: HSSI Subsystem
Additionally, the Ethernet MAC interface has an interface for status and flow control. Status and flow control signals vary across IPs and operating modes so for this design we group the signals into a AXI4-Stream sideband interface which provides a standard interface to the AFU along with platform customizations if needed. The Avalon to Arm AMBA 4 AXI4 bridge (av_axi_st_bridge) converts native Avalon interfaces to an AXI4 Stream interface.
The following flow control are implemented in the Ethernet MAC: * IEEE 802.3 flow control: this flow control implements the IEEE 802.3 Annex 31B standard to manage congestion. When the Low Latency Ethernet 10G MAC IP experiences congestion, the core sends a pause frame to request its link partner to suspend transmission for a given period of time. This flow control is a mechanism to manage congestion at the local or remote partner. When the receiving device experiences congestion, it sends an XOFF pause frame to the emitting device to instruct the emitting device to stop sending data for a duration specified by the congested receiver. Data transmission resumes when the emitting device receives an XON pause frame (pause quanta = zero) or when the timer expires.
\u2022 Priority-based flow control (PFC): this flow control implements the IEEE 802.1Qbb standard. PFC manages congestion based on priority levels. It supports up to 8 priority queues. When the receiving device experiences congestion on a priority queue, it sends a PFC frame requesting the emitting device to stop transmission on the priority queue for a duration specified by the congested receiver. When the receiving device is ready to receive transmission on the priority queue again, it sends a PFC frame instructing the emitting device to resume transmission on the priority queue
To use flow control, set the following registers:
On the TX datapath: 1. Set tx_pfc_priority_enable[7:0] (Address :0x0046 -> 11A0) to 0 to disable the PFC. The rest of the bits are unused. 2. Set tx_pauseframe_enable[0] (Address :0x0044 -> 1142) to 1 to enable the flow control.
On the RX datapath:
\u2022 Set rx_pfc_control[7:0] (Address :0x00C0 -> 818) to 1 to disable the PFC. The rest of the bits are mostly unused. \u2022 Set the IGNORE_PAUSE (Address :0x00AC -> 800) bit in the rx_frame_control register to 0 to enable the flow control.
To use Priority-Based Flow Control Follow these steps to use the priority-based flow control (PFC): 1. Enable the Priority-based flow control (PFC) parameter and specify the number of priority levels using the Number of PFC priorities parameter. You can specify between 2 to 8 PFC priority levels. 2. Set the following registers.
On the TX datapath: * Set tx_pauseframe_enable (Address :0x0044 -> 1142) to 0 to disable the flow control. * Set tx_pfc_priority_enable[n] (Address :0x0046 -> 11A0) to 1 to enable the PFC for priority queue n.
On the RX datapath: * Set the IGNORE_PAUSE bit in the rx_frame_control (Address :0x00AC -> 800) register to 1 to disable the flow control. * Set the rx_pfc_control[7:0] (Address :0x00C0 -> 818) register bits to 0 to enable the PFC. Most of the rest of the bits are unused. * Set PFC Quanta bit for the appropriate queue. Eg: pfc_pause_quanta_0 (0x048 -> 1180) for queue0 and so on. * Connect the avalon_st_tx_pfc_gen_data signal to the corresponding RX client logic and the avalon_st_rx_pfc_pause_data signal to the corresponding TX client logic. * You have the option to configure the MAC RX to forward the PFC frame to the client by setting the rx_pfc_control[16] register to 1. By default, the MAC RX drops the PFC frame after processing it
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1222-hssi-subsystem-afu-interfaces","title":"12.2.2 HSSI Subsystem AFU Interfaces","text":"The HSSI subsystem provides the following interfaces to the AFU region:
The he-hssi uses the APF interface for HSSI CSR (MMIO) accesses. The AXI4-Stream Ethernet data and side band interface along with Ethernet clocks communicate directly to the he-hssi module in the AFU region through platform independent data structures provided by the PIM. Even if you implement a different MAC you typically can leverage these data structures defined in the hssi/inc/ofs_fim_eth_avst_if.sv file here without modification.
While the platform-independent interfaces in ofs_fim_eth_if.sv are convenient containers for passing data streams through the design hierarchy, both the MAC and AFU traffic generator require platform-specific data types. The payloads of the streams in ofs_fim_eth_if.sv are defined in platform-specific structures, with fields that are MAC-specific. In this 10GbE reference design, the payload datatypes are defined in the ipss/hssi/s10/includes/ofs_fim_eth_plat_if_pkg.sv file in the OFS D5005 FIM Github Branch. Implementers connecting a new MAC should generally edit only ofs_fim_eth_plat_if_pkg.sv when defining payloads.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1223-hssi-sideband-interface","title":"12.2.3 HSSI Sideband Interface","text":"The AXI4-Stream sideband interface has been been defined to allow for time sensitive status and flow control information. It does not have the optional tready signal and assumes the slave always accepts information. The Ethernet sideband interface varies widely across IPs and operating modes and device generations. In OFS Stratix 10 FIM, the Ethernet sideband signals are mapped to the AXI4-Stream interface and interface variations can be accommodated by customizing the tdata signals of the sideband interface in platform specific interface packages using the PIM.
As an example, please refer to ofs_fim_eth_sideband_tx_axis_if interface in the ipss/hssi/inc/ofs_fim_eth_if.sv found OFS D5005 FIM Github Branch.
The t_axis_eth_sideband_tx and t_axis_eth_sideband_rx structures are found in the ipss/hssi/s10/includes/ofs_fim_eth_plat_if_pkg.sv file OFS D5005 FIM Github Branch.
Platform specific details for the 10GbE example are from the ipss/hssi/s10/includes/ofs_fim_eth_plat_if_pkg.sv file OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1224-reconfiguration-interfaces","title":"12.2.4 Reconfiguration Interfaces","text":"The reconfiguration interface in the OFS EA design consists of abstracted and consolidated memory-mapped transceiver reconfiguration interfaces that are exposed to the HSSI CSRs. The reconfiguration interface directly exposes the address space of the MAC and PHY IPs in order to allow a software driver to perform dynamic reconfiguration of those IPs (i.e. read/write access to the Native PHY CSRs). Therefore, to use this interface you must be familiar with the CSR memory maps of the corresponding IP cores.
The table below summarizes all the ports associated with Reconfiguration Interfaces.
Name Width Domain Description i_xcvr_reconfig_cmd 2 i_reconfig_clk Command port used to specify a read or write access operation to a MAC/PHY CSRs on a selected CSR_interface i_xcvr_reconfig_addr 20 i_reconfig_clkCSR access address of MAC/PHY IP. Note that only the lower 16 bits are used, i_xcvr_reconfig_addr[15:0]. The remaining upper bits, i_xcvr_reconfig_addr[20:16], should be set to zero. i_xcvr_reconfig_writedata 32 i_reconfig_clk CSR data to be written on a write command. o_xcvr_reconfig_readdata 32 i_reconfig_clk CSR data that was read on a read command. o_xcvr_reconfig_ack 1 i_reconfig_clk Reconfiguration acknowledgement bit. Asserted when Reconfiguration Controller is finished performing an Avalon-MM read or write request (i.e. waitrequest is low). Deasserted when i_xcvr_reconfig_cmd is all-zero (i.e. command is invalidated). Note that the controller assumes a valid command when i_xcvr_reconfig_cmd is non-zero."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12241-reconfiguration-sequence","title":"12.2.4.1 Reconfiguration Sequence","text":"The diagram below explains the sequence of operation and handshaking between software and a memory-mapped dynamic reconfiguration interface.
Figure 12-2: Sequence of Operation and Handshaking between Software and a Memory-Mapped Dynamic Reconfiguration Interface.
(0) Idle state \u2013 command and address buses are cleared (all-zero). 1. Software sets a desired non-zero command and address to initiate reconfiguration.
a. Memory-Mapped Reconfiguration Controller (MM CTRL) converts the command and address to a single Avalon Memory Mapped read or write request and handles Avalon Memory Mapped protocol details.\n\nb. MM CTRL completes the Avalon Memory Mapped transaction when the `waitrequest` signal of a given Avalon Memory Mapped interface is deasserted.\n\nc. MM CTRL sets the reconfiguration acknowledgment bit and `readdata` (in case of a read command) back to the FME and waits for command and address ports to be cleared by software. \n1.\nMeanwhile, software continuously polls the reconfiguration acknowledgment bit and waits for it to get asserted. Assertion of the acknowledgment bit confirms that the MM CTRL has completed the current Avalon Memory Mapped read/write request.
Software reads readdata from the HSSI_RCFG_DATA CSR that was returned by the MM CTRL (in case of a read command).
Software clears the command and address buses to communicate back to the MM CTRL that the operation is finished from the CPU\u2019s perspective.
a. MM CTRL gets cleared (all-zero) command and address signals from the HSSI_CSR. b. MM CTRL clears (deasserts) the reconfiguration acknowledgment bit back to the HSSI_CSR and is finished / back to idle state. 5. Meanwhile, software continuously polls the reconfiguration acknowledgment bit and waits for it to get deasserted. Deassertion of the acknowledgment bit confirms that the MM CTRL has completed its handshake and is now back to idle state.
NOTE
Reads and writes cannot be performed at the same time. Remember that when multiple CSRs are at the same address, a Read-Modify-Write operation may be required to change the desired CSR without changing the CSRs in the same address.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1225-hssi-control-and-status-register-csr-map","title":"12.2.5 HSSI Control and Status Register (CSR) Map","text":"The HSSI CSR Map structure is designed to scale according to IP capabilities.
Example: If Num_CSR_interface=2 & Num_channels_CSR_interface=2 then channel(0,1) are behind CSR interface 0 handled by HSSI_RCFG_CMD0/DATA0 , channel (2,3) are behind CSR interface 1 handled by HSSI_RCFG_CMD1/DATA1 HSSI_CTRL, HSSI_STATUS0, HSSI_STATUS1 provide control and status information and can scale upto 8 channels. HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers are for the reconfiguration interface, and additional mailbox registers could be added depending on the number of CSR interfaces the design exposes.
The HSSI CSR Map can be found in the ipss/hssi/s10/hssi_ss_csr.xls file OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1226-hssi-host-exercisier-he-hssi","title":"12.2.6 HSSI Host Exercisier (HE-HSSI)","text":"HE-HSSI is an Ethernet AFU that handles client side ethernet traffic. The reference HE-HSSI has following features:
The HE-HSSI Ethernet block diagram is below.
Figure 12-6: HE-HSSI Block Diagram Block Diagram
Figure 12-7: 10G Ethernet AFU Clock Domains
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12261-he-hssi-csr-map","title":"12.2.6.1 HE-HSSI CSR Map","text":"The reference HSSI AFU contains the following registers and a similar arrangement of register space can be implemented for other usecase specific HSSI AFUs. * AFU DFH Register: Device feature header for the AFU (AFU_DFH) * AFU ID Registers: 128-bit UUID for the AFU which occupies two 64-bit registers (AFU_ID_L, AFU_ID_H) * Mailbox Registers: Command and Data register for traffic controller register access. It follows the standard access method defined for OFS. Access method and implementation is same as Reconfiguration Interface defined for the HSSI FIM. (TRAFFIC_CTRL_CMD, TRAFFIC_CTRL_DATA) * Channel Select Register: Channel select register for traffic controller mailbox access. It is used in cases where more than one channel is in the AFU, else it defaults to zero, meaning channel-0 is selected. (TRAFFIC_CTRL_PORT_SEL) * Scratchpad Register: Scratchpad register for CSR access checking. (AFU_SCRATCHPAD)
The CSR excel for the 10G HSSI reference AFU can be found ipss/hssi/s10/hssi_ss_csr.xls OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#123-hssi-software","title":"12.3 HSSI Software","text":"There are two pieces of software related to running the HSSI Subsystem and the HE-HSSI host exerciser: The Linux* dfl network driver and a user space application.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1231-hssi-linux-driver","title":"12.3.1 HSSI Linux Driver","text":"The HSSI subystem is exposed as a feature in the PCIe PF BAR0 region. It has a Device Feature Header (DFH) indicating the HSSI interface. The feature ID in the DFH causes the following driver to be instantiated for the HSSI interface: drivers/net/ethernet/intel/s10hssi.c Kernel Driver Branch
The primary functionality of the driver is to interact with the ethernet MAC and PHY through an indirect register access mailbox implemented by the HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers described above. To aid in RTL bringup, the driver provides a debugfs interface directly to the indirect register access mailbox. For each HSSI interface in the system there would be a directory with the following form containing two files, regaddr and regval: /sys/kernel/debug/dfl-fme.X.Y
To read a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then read the value as string out of regval file. To write a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then write the value as a C hex string to regval file.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1232-hssi-user-space-tool","title":"12.3.2 HSSI User Space Tool","text":"The HSSI user space application exports a control interface to the HSSI AFU's packet generator logic. Context-sensitive help is given by the --help option, doc/src/fpga_tools/hssi/hssi.md, OPAE SDK Branch.
$ hssi --help\n\n"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#124-user-guidelines","title":"12.4 User Guidelines","text":"You can either leverage Ethernet example designs from platform designer or use your own custom IP\u2019s. However below recommendations would help leverage the infrastructure of the OFS stack:\n* Follow the Ethernet-GBS interface standard, customize platform specific sideband and clock intefaces.\n* Follow the reconfiguration interface example and reuse the hssi_csr block by modifying the memory map (change address decoder map accordingly as well.)
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#13-partial-reconfiguration","title":"13 Partial Reconfiguration","text":"Partial Reconfiguration (PR) is an Altera FPGA technology that allows user to reconfigure parts of the FPGA device dynamically, while the remainder of the device continues to operate. In a non-partial reconfiguration flow, any change to the design requires full reprogramming of the entire configuration RAM (CRAM) arrays in the device. With partial reconfiguration, you can dynamically reprogram one or more CRAM frames. A partial reconfiguration design has a static region, and one or more PR regions, which can be modified to implement new logic. The portion of the CRAM on the chip to be reconfigured is contained within a PR region.\nFor the PR flow, the design should be partitioned into static region and reconfigurable region. The static region is the area of your FPGA that is not reconfigured without reprogramming the entire FPGA. An area of the chip that you plan to partially reconfigure is a PR region.
\nThe Port Gasket contains all the PR specific modules and logic, such as PR slot reset/freeze control, user clock, remote STP etc. For this reference example only one PR slot is supported.\nThe following figure depicts the high level view of the Port Gasket:
\nFigure 13-1 Partial Reconfiguration Gasket
\n\n\nThe isolation logic is provided on the output signals of the PR region to ensure they don\u2019t glitch and affect the interfacing logic in the Static Region (SR). The isolation logic is controlled by the PR Freeze logic during PR operation.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14-reliability-accessibility-serviceability-ras-and-error-handling","title":"14 Reliability, Accessibility, Serviceability (RAS) and Error Handling","text":"\n- Downstream AFU checker: Identifies AFU violations. For example, this checker flags violations of the interface specification.
\n- Upstream software or PCIe link checker: Identifies invalid traffic from PCIe that violates either FIM specifications or PCIe specifications. For example, this checker flags an application sending traffic if it violates the FIM specification or creates a PCIe link issue by causing completion timeout or malformed TLP.
\n- FIM - Checks for bugs in the FIM fabric.
\n
\nErrors reported by the checker are logged in either the FME error registers or Port error registers, or both, as shown in the table below. For more details on each of the registers, please refer to src/common/protocol_checker/protocol_checker_csr.xml file OFS FIM_COMMON Github Branch or the SystemVerilog file src/common/fme/xls/d5005/FME_CSR.xls found OFS FIM_COMMON Github Branch .
\nTable 14-1: Error Registers
\nMMIO Region\nArea\nRegister\nDescription\nFME\nCoreFIM\nFME_ERROR\nFME Error Status Register 0. Registers parity errors, underflow or overflow errors and access mismatches.\nFME\nCoreFIM\nFME_ERROR0_MASK\nFME Error Mask Register 0. Write a 0 to mask errors in the FME Error Status Register 0.\nFME\nExternal\nPCIE0_ERROR\nPCIe0 Error Status Register.\nFME\nExternal\nPCIE0_ERROR_MASK\nPCIe0 Error Mask Register 0. Write a 0 to mask errors in the PCIe0 Error Status Register 0.\nFME\nCoreFIM\nFME_FIRST_ERROR\nFirst FME Error Register.\nFME\nCoreFIM\nFME_NEXT_ERROR\nFME Next Error Register.\nFME\nCoreFIM\nRAS_NOFAT_ERR_STAT\nReliability/Accessibility/Serviceability (RAS) Non-Fatal Error Status Register.\nFME\nCoreFIM\nRAS_NOFAT_ERR_MASK\nRAS Non-Fatal Error Mask Register. Write 0 to mask error fields in RAS_NOFAT_ERR_STAT Register.\nFME\nCoreFIM\nRAS_CATFAT_ERR_STAT\nRAS Catastrophic and Fatal Errors Status Register.\nFME\nCoreFIM\nRAS_CATFAT_ERR_MASK\nRAS Catastrophic and Fatal Errors Mask Register. Write 0 to mask error fields in the RAS_CATFAT_ERR_STAT Register.\nFME\nCoreFIM\nRAS_ERROR_INJ\nRAS error Injection Register.\nPORT\nCoreFIM\nPORT_ERROR\nPort Error Status Register.\nPORT\nCoreFIM\nPORT_FIRST_ERROR\nPort First Error Register .\nPORT\nCoreFIM\nPORT_MALFORMED_REQ0\nPort Malformed Request Register 0. Provides malformed request header LSBs.\nPORT\nCoreFIM\nPORT_MALFORMED_REQ1\nPort Malformed Request Register 1. Provides malformed request header MSBs."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#141-fme-errors","title":"14.1 FME Errors","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1411-fme_error0","title":"14.1.1 FME_ERROR0","text":"The FME_ERROR0 register flags CoreFIM FME errors in the Global Error (GLBL_ERROR) private feature. The error bits in this register are sticky bits. You can only clear these bits through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in FME_ERROR0_MASK register masks the error.
\nTable 14-2: FME Error Types
\nError Type\nDescription\nFabric errors\nFIFO underflow/overflow condition in CoreFIM. These errors only occur if you have introduced bugs into the FIM or during very rare single event upset (SEU) or SEU-like events.\nInvalid port access\nA port can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the Port. If it finds a PF is trying to access a port that is mapped to a VF or vice-versa, an error will be reported.\nInvalid AFU access\nAn AFU can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the AFU associated with the Port. If it finds a PF is trying to access an AFU that is mapped to a VF or vice-versa, an error is reported and a fake response is sent back to the requester to avoid a completion timeout on the host."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1412-pcie0_error","title":"14.1.2 PCIE0_ERROR","text":"The PCIe Avalon-ST to AXI4-Stream bridge monitors the PCIe link for errors and logs any such errors in the PCIE0_ERROR register (in PCIE0 feature region) and PCIE0_ERROR register in the GLBL_ERR private feature. The error bits in the PCIE0_ERROR register are sticky bits that you can only clear through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in PCIE0_ERROR0_MASK masks the error.
\nIf you have other external FME features, you can add similar _ERROR registers to this space. Please refer to the following spreadsheet, bbs/csr/stratix10/pac_d5005/fme_csr_pcie.xls, OFS D5005 FIM Github Branch or the SystemVerilog file the SystemVerilog file src/common/fme/xls/d5005/FME_CSR.xls found OFS FIM_COMMON Github Branch for more details on this register. \n
NOTE
\nThe PCIE0_ERROR register is located in both the Global Error external feature memory space and a separate PCIe external feature memory space. OPAE software supports the PCIe external feature memory space beginning at offset 0x40000 for OFS EA and going forward. PCIe registers beginning at 0x4000 in the Global Error external feature memory space is there for backward compatibility to the Intel FPGA PAC D5005 v2.0.1 Acceleration Stack.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1413-fme_first_error-fme_next_error","title":"14.1.3 FME_FIRST_ERROR, FME_NEXT_ERROR","text":"The FME_FIRST_ERROR register flags which of the FME error reporting registers, such as FME_ERROR0, PCIE0_ERROR0, has reported the first error occurrence. The error fields of the first error register are then continuously logged into the FME_FIRST_ERROR register until a system reset or software clears all the errors in that first error register.\nLikewise, the FME_NEXT_ERROR indicates which of the FME error reporting registers (except the first error register) has reported the next occurrence of error after the first error register. The error fields of the next error register are continuously logged into the FME_NEXT_ERROR register until a system reset or software clears all the errors in the second error register.
\nPlease refer tobbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls, OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv, OFS D5005 FIM Github Branch
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1414-reliability-accessibility-serviceability-ras-error-status","title":"14.1.4 Reliability, Accessibility, Serviceability (RAS) Error Status","text":"The RAS feature in CoreFIM labels errors as non-fatal, fatal or catastrophic based on their impact to the system. \n* A non-fatal error usually originates from software or an AFU. With a non-fatal error, the user application may still be able to recover from the error by performing a soft reset on the AFU, fixing the user application software if necessary, and clearing the error. On the other hand, a fatal or catastrophic error is non-recoverable and requires the platform to be reset.\n* Non-fatal errors are logged in the RAS_NOFAT_ERR_STAT register and fatal or catastrophic errors are logged in the RAS_CATFAT_ERR_STAT register.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14141-non-fatal-errors","title":"14.1.4.1 Non-Fatal Errors","text":"The RAS_NOFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It logs the high-level status of non-fatal errors in the hardware. Unlike the error bits in the PCIE0_ERROR and FME_ERROR0 registers which are RW1C (software can write a 1 to clear the error), the error bits in this register are read-only and can only be cleared by system reset. Software has an option to mask the error using RAS_NOFAT_ERR_MASK.\nPlease refer to bbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls, OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv, OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14142-catastrophic-fatal-errors","title":"14.1.4.2 Catastrophic & Fatal Errors","text":"The RAS_CATFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It captures the high-level status of errors that can only be recovered with a system reset. Therefore, the error bits in the RAS_CATFAT_ERR_STAT register are read-only and can only be cleared by system reset or masked through RAS_CATFAT_ERR_MASK.\nPlease refer to bbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls, OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1415-ras-error-injection","title":"14.1.5 RAS Error Injection","text":"For software testing purposes, you can inject non-fatal, fatal and catastrophic errors into the platform through the RAS_ERROR_INJ register. These errors are reported in the RAS_CATFAT_ERR_STAT and RAS_NOFAT_ERR_STAT registers.\nPlease refer to bbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1416-fme-error-interrupts","title":"14.1.6 FME Error Interrupts","text":"In an event of an FME error, the MSI-X module in the FIM generates an interrupt so the host can decide on the next course of action. The FIM does not stall upstream and downstream traffic after the FME error. However, a port error triggered by invalid request from a user AFU stalls all the traffic going from AFU to PCIe.\nThe interrupt capability is discoverable by querying the NumbSuppInterrupt field of the PORT_CAPABILITY register in the Port private feature. The MSI-X vector number is recorded in the InterruptVectorNumber field of the GLBL_ERROR_CAPABILITY register of the Global Error external feature.
\nAn FME error interrupt is generated in response to the FME_ERROR0, PCIE0_ERROR0, RAS_NOFAT_ERR_STAT or RAS_CATFAT_ERR_STAT registers recording a new, unmasked, error per the rules defined by CoreFIM interrupt feature.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14161-msi-x-masking-pending-bit-array-pba-clearing","title":"14.1.6.1 MSI-X Masking & Pending Bit Array (PBA) Clearing","text":"If the MSI-X vector corresponding to the FME error interrupt is masked, events are recorded in the PBA array. Clearing the FME error status registers clears the corresponding MSI-X PBA entries. If only some events are cleared, the normal interrupt triggering rules apply and a new pending interrupt is registered.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1417-fme-error-handling","title":"14.1.7 FME Error Handling","text":"When the host receives an FME error interrupt, it must take the recommended actions described below to bring the system back to its normal operation.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14171-catastrophicfatal-error","title":"14.1.7.1 Catastrophic/Fatal Error","text":"A system reset is mandatory for any catastrophic or fatal error.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14172-non-fatal-error","title":"14.1.7.2 Non-Fatal Error","text":"When software receives a non-fatal error interrupt which does not require a system reset, it can take the following steps to clear the error after software handles the error:\n1. Set the *_ERROR_MASK register to all 1\u2019s to mask all errors\n2. Clear the *_FIRST_ERROR register\n3. Clear the *_ERROR register\n4. Set *_ERROR_MASK register to all 0\u2019s to enable all errors
\n\n- Result: The *_ERROR & *_FIRST_ERROR registers begin capturing new errors.
\n
\nNOTE
\nA system reset can only clear RAS Error status registers.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#142-mmio-requests","title":"14.2 MMIO Requests","text":"The FIM is designed to gracefully handle MMIO request scenarios.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1421-unsupported-functions-and-bars","title":"14.2.1 Unsupported Functions and BARs","text":"The OFS FIM EA has only one PCIe link and all MMIO requests from the host are sent through this link. The PCIe hard IP in the FIM guarantees that only TLP packets for the functions and BARs supported by the FIM (as configured in PCIe HIP IP instantiation) are sent to the FIM on the PCIe Avalon Streaming interface.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1422-mmio-request-decoding","title":"14.2.2 MMIO Request Decoding","text":"The packet router and memory decoder in the FIM ensure that only legal MMIO requests are forwarded to the targeted MMIO region. Full address and BAR decoding is done both in the packet router and the memory decoder to ensure the requests are forwarded to the designated CSR region as defined in the MMIO Regions chapter. Any unsolicited/illegal MMIO request is dropped, and an error is reported back to host through the FME error register.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1423-unused-fmeport-csr-regions","title":"14.2.3 Unused FME/Port CSR Regions","text":"All the CSR slaves in FIM which are mapped to the FME or Port CSR regions must always respond to MMIO read requests targeting its associated CSR region. A CSR slave must return all 0s for MMIO reads to its unused CSR region such as a reserved space or a region where no CSR register is implemented for the address.\nThe FIM ensures MMIO reads to FME or Port CSR regions that are not mapped to any CSR slave always gets a response of all 0s. The memory decoder module and fake responder module in the FIM provide this guaranteed response.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1424-unsupported-mmio-request","title":"14.2.4 Unsupported MMIO Request","text":"Any MMIO request targeting FME or Port CSR regions with a length or address alignment that are not supported by the FIM is dropped, and an error is logged in PCIE0_ERROR register. The MMIO checker module in the FIM guarantees this response. When an unsupported MMIO read request to the FIM CSR region is detected, the FIM sends back a CPL (completion without data) with error status (UR) back to host.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1425-afu-access-violation","title":"14.2.5 AFU Access Violation","text":"AFU access violations refer to the scenarios where a PF is attempting to access the MMIO region of an AFU bound to a VF (virtualization enabled), or when a VF is trying to access the MMIO region of an AFU bound to a PF (virtualization disabled). When such a violation is detected, the FIM drops the request and logs the error in the FME_ERROR0 register. If the request is an MMIO read request, the FIM returns a fake response to the host.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1426-afu-mmio-response-timeout","title":"14.2.6 AFU MMIO Response Timeout","text":"An AFU MMIO Response timeout functions in the same manner described in the MMIO Response Timeout section.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#15-design-guidance","title":"15 Design Guidance","text":"The OFS FIM is designed with configurability and scalability in mind. At a high level, these are the necessary steps for a user to customize the design. Please refer to the FPGA Interface Manager Developer Guide: Open FPGA Stack for Stratix 10
\nTable 15-1 Features
\nStep\n Description\n Comments \n 1\n Re-configure PCIe HIP for additional VFs (if necessary)\n * PF0 is mandatory for running OPAE software* Only modification of the VFs (added or subtracted) by the user is recommended.\n\u2022 Default configuration supports 1 PF and 3 VFs.\n \n 2\n Update AXI4-Stram PF/VF MUX-DEMUX configuration (if necessary)\n\n * The PF/VF MUX-DEMUX is parameterized for flexibility. You can change, add or delete PF or VF functions by updating the top_cfg_pkg.sv file.\n* You also have the option of keeping the default configuration and tying off the unused VFs if needed.\n\n 3\n Update top level and AFU level as necessary\n\n * If you integrating additional external interfaces, make the edits at the top level (iofs_top.sv) and propagate the interface down to the AFU level (afu_top.sv)\n 4\n Add user implemented function(s) in AFU\n * All of your implemented functions must have the required AXI4-Stream interface for both the data path and the MMIO control path to CSRs. * All CSRs in the user implemented function must have the required DFH layout. * See host exerciser CSRs for reference.\n \n \n\n\nFor more information on modifying the FIM, refer to the [Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs](https://ofs.github.io/ofs-2024.2-1/hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/).\n\n\n\n## Notices & Disclaimers\n\nIntel\u00ae technologies may require enabled hardware, software or service activation.\nNo product or component can be absolutely secure. \nPerformance varies by use, configuration and other factors.\nYour costs and results may vary. \nYou may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.\nNo license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document.\nThe products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.\nIntel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.\nYou are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \n\u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others. \n\nOpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122. \n\n\n\n\n\n\n\n\n*[AFU]: Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region\n*[BBB]: Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID.\n*[BKC]: Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against.\n*[BMC]: Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors.\n*[DFL]: Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\n*[FIM]: FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring.\n*[FME]: FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform.\n*[HEM]: Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc.\n*[JTAG]: Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology.\n*[OFS]: Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs.\n*[OPAE SDK]: Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE.\n*[PIM]: Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols.\n*[PR]: Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page.\n*[RSU]: Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration.\n*[UVM]: Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework.\n*[TB]: Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output.\n*[Intel VT-d]: Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization.\n*[SR-IOV]: Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance.\n*[MMIO]: Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators.\n*[VFIO]: Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace.\n*[IOCTL]: Input/Output Control, System calls used to manipulate underlying device parameters of special files.\n*[AER]: Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting.\n*[PAC]: Programmable Acceleration Card: FPGA based Accelerator card"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/","title":"Platform Evaluation Script: Open FPGA Stack for Stratix 10 FPGA","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#1-overview","title":"1 Overview","text":""},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the checkout and evaluation of an Intel\u00ae FPGA PAC D5005 development platform using Open FPGA Stack (OFS). After reviewing the document, you will be able to:
- Set-up and modify the script to your environment
- Compile and simulate an OFS reference design
- Run hardware and software tests to evaluate the complete OFS flow
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#12-table-software-version-summary","title":"1.2 Table : Software Version Summary","text":"Component Version Description FPGA Platform Intel\u00ae FPGA PAC D5005 Stratix 10 platform you can use for your custom board development OFS FIM Source Code Branch: ofs-d5005, Tag: ofs-2024.1-1 OFS Shell RTL for Stratix 10 FPGA (targeting Intel\u00ae FPGA PAC D5005) OFS FIM Common Branch: ofs-2024.1-1, Tag: ofs-2024.1-1 Common RTL across all OFS-based platforms AFU Examples Branch: examples-afu , Tag: ofs-examples-ofs-2024.1-1 Tutorials and simple examples for the Accelerator Functional Unit region (workload region) OPAE SDK Branch: 2.12.0-5, Tag: 2.12.0-5 Open Programmable Acceleration Engine Software Development Kit Kernel Drivers Branch: ofs-2024.1-6.1-2, Tag: ofs-2024.1-6.1-2 OFS specific kernel drivers OPAE Simulation Branch: opae-sim, Tag: 2.12.0-5 Accelerator Simulation Environment for hardware/software co-simulation of your AFU (workload) Quartus Prime Pro Edition Design Software 23.4 Quartus\u00ae Prime Pro Edition Linux Software tool for FPGA Development Operating System RHEL 8.6 Operating system on which this script has been tested A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae FPGA PAC D5005 can be found on the OFS 2024.1-1 official release drop on GitHub.
Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#2-introduction-to-ofs-evaluation-script","title":"2 Introduction to OFS Evaluation Script","text":"By following the setup steps and using the OFS evaluation script you can quickly evaluate many features that the OFS framework provides and also leverage this script for your own development.
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#21-pre-requisites","title":"2.1 Pre-Requisites","text":"This script uses the following set of software tools which should be installed using the directory structure below. Tool versions can vary.
- Quartus\u00ae Prime Pro Software : The software can be downloaded here. For details on installation of required patches and quartus installation, refer to section 4.1 of the Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
- Synopsys\u00ae VCS Simulator : The simulator can be downloaded in the Synopsys page here
- Siemens\u00ae Questa\u00ae Simulator : The simulator can be downloaded in the Siemens page here
Figure 2-1 Folder Hierarchy for Software Tools
- Download tar(scripts_ofs-2024.1-1_external.tar.gz) from the assets tab of the release page
- Untar to folder using the following command
tar -xvf scripts_ofs-2024.1-1_external.tar.gz\n````\n\n3. The folder structure containing the clone script (ofs-clone_repositories.sh) and evaluation script (ofs_d5005_eval.sh) is as shown in the figure 2-2\n\n**Figure 2-2 Directory Structure for the clone script**\n\n``` sh\n## ofs_scripts\n## -> host_chan_mmio.stp\n## -> ofs_d5005_eval.sh\n## -> README_ofs_d5005_eval.txt\n## ofs-clone_repositories.sh\n
- Open the clone script ofs-clone_repositories.sh in any text editor
- Please enter the location of your proxy server to allow access to external internet to build software packages. (lines 6-8)
Note: Failing to add proxy server will prevent cloning of repositories and the user will be unable to build the OFS framework.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
-
Please enter the license file locations (lines 11-13) for the following tool variables
export LM_LICENSE_FILE=<user_license>\n export DW_LICENSE_FILE=<user_license>\n export SNPSLMD_LICENSE_FILE=<user_license>\n
-
Add the name of the directory where you want the platform repositories to be cloned (line 19)
export OFS_RELEASE=ofs-2024.1-1\n
-
After setting the required variables, source the clone script based on which platform you want to test
source ofs-clone_repositories.sh\n
- The script ofs-clone_repositories.sh has different platforms to choose from the menu to clone as shown in the figure 2-3.
Figure 2-3 Directory Structure for OFS Project
-
Once the repositories are cloned, navigate to the directory where you cloned
cd ofs-2024.1-1\n
-
After cloning, the OFS repositories cloned will look as per Figure 2-4.
Figure 2-4 Directory Structure for OFS Project
## ofs-2024.1-1\n## d5005(OFS platform)\n## -> examples-afu\n## -> ofs-d5005\n## -> oneapi-asp\n## -> oneAPI-samples\n## -> opae-sim\n## -> opae-sdk\n## -> linux-dfl\n## -> ofs_d5005_eval.sh\n## -> README_ofs_d5005_eval.txt\n## -> host_chan_mmio.stp\n
- Once the repositories are cloned, in the platform specific evaluation script, for e.g., in ofs_d5005_eval.sh, please follow the instructions below which explains which line numbers to change to adapt this script to the user environment.
a) Set-Up Proxy Server (lines 67-69)
Please enter the location of your proxy server to allow access to external internet to build software packages.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
b) Tools Location (line 87, 88, 89, 90)
Set Location of Quartus, Synopsys, Questasim and oneAPI Tools
export QUARTUS_TOOLS_LOCATION=/home\nexport SYNOPSYS_TOOLS_LOCATION=/home\nexport QUESTASIM_TOOLS_LOCATION=/home\nexport ONEAPI_TOOLS_LOCATION=/opt\n
In the example above /home is used as the base location of Quartus, Synopsys and Questasim tools, /opt is used for the oneAPI tools
c) PCIe (Bus Number)
The Bus number must be entered by the user after installing the hardware in the chosen server, in the example below \"b1\" is the Bus Number for a single card as defined in the evaluation script.
export OFS_CARD0_BUS_NUMBER=b1\n
The evaluation script uses the bus number as an identifier to interrogate the card. The command below will identify the accelerator card plugged into a server.
lspci | grep acc\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\nb1:00.1 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.2 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\nb1:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
The result identifies the card as being assigned \"b1\" as the bus number so the entry in the script changes to
export OFS_CARD0_BUS_NUMBER=b1\n
The user can also run the following command on the ofs_d5005_eval.sh script to automatically change the bus number to b1 in the ofs_d5005_eval.sh script.
grep -rli 'b1' * | xargs -i@ sed -i 'b1' @
if the bus number is 85 for example
85:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\n85:00.1 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.2 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\n85:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
the command to change to 85 in the evaluation script would be
grep -rli 'b1' * | xargs -i@ sed -i '85' @
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#ofs-platform-example-n6000-n6001-fseries-dk-iseries-dk-custom_board-choice-line-173","title":"OFS Platform (Example:= n6000, n6001, fseries-dk, iseries-dk, custom_board) choice (line 173)","text":"The script is designed to accommodate many OFS platform choices eg, n6000, n6001, fseries-dk, iseries-dk or a custom_board of your choice. By default the script defaults to n6001 as shown below and is set by a variable on line 173 of the ofs_d5005_eval.sh script.
export OFS_PLATFORM=n6001\n
but the user can switch platform by changing the OFS_PLATFORM variable, so for example if the user wants to switch to the i-series development kit the command would be
export OFS_PLATFORM=iseries-dk\n
The ofs_d5005_eval.sh script has now been modified to the server set-up and the user can proceed to build, compile and simulate the OFS stack
Figure 3-1 ofs_d5005_eval.sh Evaluation Menu
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#311-tools-menu","title":"3.1.1 TOOLS MENU","text":"By selecting \"List of Documentation for ADP Intel\u00ae FPGA PAC D5005 Project,\" a list of links to the latest OFS documentation appears. Note that these links will take you to documentation for the most recent release which may not correspond to the release version you are evaluating. To find the documentation specific to your release, ensure you clone the OFS tag that corresponds to your OFS version.
By selecting \"Check Versions of Operating System and Quartus Premier Design Suite\", the tool verifies correct Operating System, Quartus version, kernel parameters, license files and paths to installed software tools.
Menu Option Example Output 1 - List of Documentation for ADP d5005 Project Open FPGA Stack Overview Guides you through the setup and build steps to evaluate the OFS solution https://ofs.github.io 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS) Checking Linux release Linux version 6.1.78-dfl Checking RedHat release Red Hat Enterprise Linux release RHEL 8.6 Checking Ubuntu release cat: /etc/lsb-release: No such file or directory Checking Kernel parameters BOOT_IMAGE=(hd0,msdos1)/vmlinuz-6.1.78-dfl-2023.4-1 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 Checking Licenses LM_LICENSE_FILE is set to port@socket number:port@socket number DW_LICENSE_FILE is set to port@socket number:port@socket number SNPSLMD_LICENSE_FILE is set to port@socket number:port@socket number Checking Tool versions QUARTUS_HOME is set to /home/intelFPGA_pro/23.4/quartus QUARTUS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus IMPORT_IP_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../ip QSYS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../qsys/bin Checking QPDS Patches Quartus Prime Shell Version 23.4"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#312-hardware-menu","title":"3.1.2 HARDWARE MENU","text":"Identifies card by PCIe number, checks power, temperature and current firmware configuration.
Menu Option Example Output 3 - Identify Acceleration Development Platform (ADP) d5005 Hardware via PCIe PCIe card detected as 86:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) Host Server is connected to SINGLE card configuration 4 - Identify the Board Management Controller (BMC) Version and check BMC sensors Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** BMC SENSORS ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e 5 - Identify the FPGA Management Engine (FME) Version Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** FME ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4eBoot Page : user 6 - Check Board Power and Temperature Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** POWER ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e( 1) VCCERAM Voltage : 0.90 Voltsetc ......................Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** TEMP ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e( 1) VCCT Temperature : 57.00 Celsiusetc ...................... 7 - Check Accelerator Port status //****** PORT ******// Object Id : 0xEF00000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00 8 - Check MAC and PHY status Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** MAC ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4eMAC address : 64:4c:36:f:44:1fIntel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** PHY ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#313-fimpr-build-menu","title":"3.1.3 FIM/PR BUILD MENU","text":"Builds FIM, Partial Reconfiguration Region and Remote Signal Tap
Menu Option Description 9 - Check ADP software versions for ADP Intel\u00ae FPGA PAC D5005 Project OFS_ROOTDIR is set to /home/user_area/ofs-2024.1-1/ofs-d5005OPAE_SDK_REPO_BRANCH is set to release/2.12.0-5 OPAE_SDK_ROOT is set to /home/user_area/ofs-2024.1-1/ofs-d5005/../opae-sdk LD_LIBRARY_PATH is set to /home/user_area/ofs-2024.1-1/ofs-d5005/../opae-sdk/lib64: 10 - Build FIM for Intel\u00ae FPGA PAC D5005 Hardware This option builds the FIM based on the setting for the $ADP_PLATFORM, $FIM_SHELL environment variable. Check these variables in the following file ofs_d5005_eval.sh 11 - Check FIM Identification of FIM for Intel\u00ae FPGA PAC D5005 Hardware The FIM is identified by the following file fme-ifc-id.txt located at $OFS_ROOTDIR/$FIM_WORKDIR/syn/syn_top/ 12 - Build Partial Reconfiguration Tree for Intel\u00ae FPGA PAC D5005 Hardware This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the OneAPI build flow 13 - Build Base FIM Identification(ID) into PR Build Tree template This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and OneAPI workloads 14 - Build Partial Reconfiguration Tree for Intel\u00ae FPGA PAC D5005 Hardware with Remote Signal Tap This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the OneAPI build flow and for the Remote Signal Tap flow 15 - Build Base FIM Identification(ID) into PR Build Tree template with Remote Signal Tap This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree for Remote Signal Tap to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and OneAPI workloads"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#314-hardware-programmingdiagnostic-menu","title":"3.1.4 HARDWARE PROGRAMMING/DIAGNOSTIC MENU","text":"The following submenu allows you to: * Program and check flash * Perform a remote system update (RSU) of the FPGA image into the FPGA * Bind virtual functions to VFIO PCIe driver * Run host exerciser (HE) commands such as loopback to test interfaces VFIO PCI driver binding * Read the control and status registers (CSRs) for bound modules that are part of the OFS reference design.
Menu Option Description 16 - Program BMC Image into Intel\u00ae FPGA PAC D5005 Hardware The user must place a new BMC flash file in the following directory $OFS_ROOTDIR/bmc_flash_files. Once the user executes this option a new BMC image will be programmed. A remote system upgrade command is initiated to store the new BMC image 17 - Check Boot Area Flash Image from Intel\u00ae FPGA PAC D5005 Hardware This option checks which location area in FLASH the image will boot from, the default is user1 Boot Page : user1 18 - Program FIM Image into user1 area for Intel\u00ae FPGA PAC D5005 Hardware This option programs the FIM image \"d5005_page1_unsigned.bin\" into user1 area in flash 19 - Initiate Remote System Upgrade (RSU) from user1 Flash Image into Intel\u00ae FPGA PAC D5005 Hardware This option initiates a Remote System Upgrade and soft reboots the server and re-scans the PCIe bus for the new image to be loaded 2022-12-13 07:31:33,244 - [[pci_address(0000:86:00.0), pci_id(0x8086, 0xbcce, 0x8086, 0x138d)]] performing RSU operation 2022-12-13 07:31:33,249 - [[pci_address(0000:85:00.0), pci_id(0x8086, 0x2030, 0x1590, 0x00ea)]] removing device from PCIe bus 2022-12-13 07:31:34,333 - waiting 10.0 seconds for boot 2022-12-13 07:31:44,344 - rescanning PCIe bus: /sys/devices/pci0000:85/pci_bus/0000:85 2022-12-13 07:31:44,377 - RSU operation complete 20 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 21 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 22 - Create Virtual Functions (VF) and bind driver to vfio-pci Intel\u00ae FPGA PAC D5005 Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 20 to check that the new drivers are bound 23 - Run HE-LB Test This option runs 5 tests 1) checks and generates traffic with the intention of exercising the path from the AFU to the Host at full bandwidth 2) run a loopback throughput test using one cacheline per request 3) run a loopback read test using four cachelines per request 4) run a loopback write test using four cachelines per request 5) run a loopback throughput test using four cachelines per request 24 - Run HE-MEM Test This option runs 2 tests 1) Checking and generating traffic with the intention of exercising the path from FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host 2) run a loopback throughput test using one cacheline per request 25 - Run HE-HSSI Test This option runs 1 test HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic 1) Send traffic through the 10G AFU 26 - Read from CSR (Command and Status Registers) for Intel\u00ae FPGA PAC D5005 Hardware This option reads from the following CSR's HE-LB Command and Status Register Default Definitions HE-MEM Command and Status Register Default Definitions HE-HSSI Command and Status Register Default Definitions"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#315-hardware-afu-testing-menu","title":"3.1.5 HARDWARE AFU TESTING MENU","text":"This submenu tests partial reconfiguration by building and loading an memory-mapped I/O example AFU/workload, executes software from host, and tests remote signal tap.
Menu Option Description 27 - Build and Compile host_chan_mmio example This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB_EXTERNAL/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) binary file ready for hardware programming 28 - Execute host_chan_mmio example This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test 29 - Modify host_chan_mmio example to insert Remote Signal Tap This option inserts a pre-defined host_chan_mmio.stp Signal Tap file into the OFS code to allow a user to debug the host_chan_mmio AFU example 30 - Build and Compile host_chan_mmio example with Remote Signal Tap This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB_EXTERNAL/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) binary file ready for hardware programming with Remote Signal tap enabled 31 - Execute host_chan_mmio example with Remote Signal Tap This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test. The user must open the Signal Tap window when running the host code to see the transactions in the Signal Tap window"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#316-hardware-afu-bbb-testing-menu","title":"3.1.6 HARDWARE AFU BBB TESTING MENU","text":"This submenu tests partial reconfiguration using a hello_world example AFU/workload, executes sw from the host
Menu Option Description 32 - Build and Compile hello_world example This option builds the hello_ world example from the following repo $FPGA_BBB_CCI_SRC/tutorial/afu_types/01_pim_ifc/$AFU_BBB_TEST_NAME, where AFU_BBB_NAME=hello_world. This produces a GBS (Green Bit Stream) file ready for hardware programming 33 - Execute hello_world example This option builds the host code for hello_world example and programs the GBS file and then executes the test"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#317-adp-oneapi-project-menu","title":"3.1.7 ADP ONEAPI PROJECT MENU","text":"Builds OneAPI kernel, executes the software from host and runs diagnostic tests
Menu Option Result 34 - Check oneAPI software versions for Intel\u00ae FPGA PAC D5005 Project This option checks the setup of the oneAPI software and adds the relevant oneAPI environment variables to the terminal. This option also informs the user to match the oneAPI software version to the oneAPI-samples version 35 - Build and clone shim libraries required by oneAPI host This option builds the oneAPI directory structure 36 - Install OneAPI Host Driver This option Installs the oneAPI Host driver at the following location /opt/Intel/OpenCLFPGA/oneAPI/Boards/, and requires sudo permission 37 - Uninstall One API Host Driver This option Uninstall's the oneAPI Host driver, and requires sudo permissions 38 - Diagnose oneAPI Hardware This option Checks ICD (Intel Client Driver) and FCD (FPGA Client Driver), oneAPI library locations and detects whether oneAPI BSP is loaded into the FPGA 39 - Build oneAPI BSP ofs-d5005 Default Kernel (hello_world) This option Builds the oneAPI BSP using hello_world kernel 40 - Build oneAPI MakeFile Environment This option Builds the oneAPI environment using a Makefile for kernel insertion 41 - Compile oneAPI Sample Application (board_test) for Emulation This option compiles the board_test kernel for Emulation 42 - Run oneAPI Sample Application (board_test) for Emulation This option executes the board_test kernel for Emulation 43 - Generate oneAPI Optimization report for (board_test) This option generates an optimization report for the board_test kernel 44 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 45 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 46 - Create Virtual Function (VF) and bind driver to vfio-pci Intel\u00ae FPGA PAC D5005 Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 45 to check that the new drivers are bound 47 - Program oneAPI BSP ofs-d5005 Default Kernel (hello_world) This option programs the FPGA with a aocx file based on the hello_world kernel 48 - Compile oneAPI Sample Application (board_test) for Hardware This option compiles the board_test kernel for Hardware 49 - Run oneAPI Sample Application (board_test) for Hardware This option builds the host code for board_test kernel and executes the program running through kernel and host bandwidth tests"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#318-unit-test-project-menu","title":"3.1.8 UNIT TEST PROJECT MENU","text":"Builds, compiles and runs standalone simulation block tests. More unit test examples are found at the following location ofs-d5005/sim/unit_test
Menu Option Result 50 - Generate Simulation files for Unit Test This option builds the simulation file set for running a unit test simulation 51 - Simulate Unit Test dfh_walker and log waveform This option runs the dfh_walker based on the environment variable \"UNIT_TEST_NAME=dfh_walker\" in the evaluation script. A user can change the test being run by modifying this variable"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#319-adp-uvm-project-menu","title":"3.1.9 ADP UVM PROJECT MENU","text":"Builds, compiles and runs full chip simulation tests. The user should execute the options sequentially ie 52, 53, 54 and 55
Menu Option Description 52 - Check UVM software versions for Intel\u00ae FPGA PAC D5005 Project DESIGNWARE_HOME is set to /home/synopsys/vip_common/vip_Q-2020.03A UVM_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm VCS_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel VERDIR is set to /home/user_area/ofs-2024.1-1/ofs-d5005/verification VIPDIR is set to /home/user_area/ofs-2024.1-1/ofs-d5005/verification 53 - Compile UVM IP This option compiles the UVM IP 54 - Compile UVM RTL and Testbench This option compiles the UVM RTL and Testbench 55 - Simulate UVM ofs_mmio_test and log waveform This option runs the dfh_walking test based on the environment variable \"UVM_TEST_NAME=dfh_walking_test\" in the evaluation script. A user can change the test being run by modifying this variable 56 - Simulate all UVM test cases (Regression Mode) This option runs the Intel\u00ae FPGA PAC D5005 regression mode, cycling through all UVM tests defined in /ofs-d5005/verification/tests/test_pkg.svh file"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#3110-adp-build-all-project-menu","title":"3.1.10 ADP BUILD ALL PROJECT MENU","text":"Builds the complete OFS flow, good for regression testing and overnight builds
For this menu, a user can run a sequence of tests (compilation, build and simulation) and executes them sequentially. After the script is successfully executed, a set of binary files is produced which a you can use to evaluate your hardware. Log files are also produced which checks whether the tests passed.
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 57 from the main menu the script will execute 23 tests ie (main menu options 2, 9, 10, 11, 12, 13, 14, 15, 27, 29, 30, 32, 34, 35, 39, 40, 48, 50, 51, 52, 53, 54 and 55. These 23 menu options are chosen to build the complete OFS flow covering build, compile and simulation.
Menu Option Result 57 - Build and Simulate Complete Intel\u00ae FPGA PAC D5005 Project Generating Log File with date and timestamp Log file written to /home/user_area/ofs-2024.1-1/log_files/d5005_log_2022_11_10-093649/ofs-d5005_eval.log Definition of Multi-Test Set-up
Menu Option 57 above in the evaluation script can be refined to tailor the number of tests the users runs. The set-up is principally defined by the variable below
MULTI_TEST[A,B]=C
where
A= Total Number of menu options in script B= Can be changed to a number to select the test order C= Menu Option in Script
Example 1 MULTI_TEST[57,0]=2
A= 57 is the total number of options in the script B= 0 indicates that this is the first test to be run in the script C= Menu option in Script ie 2- List of Documentation for ADP Intel\u00ae FPGA PAC D5005 Project
Example 2 MULTI_TEST[57,0]=2 MULTI_TEST[57,1]=9
In the example above two tests are run in order ie 0, and 1 and the following menu options are executed ie 2- List of Documentation for ADP Intel\u00ae FPGA PAC D5005 Project and 9 - Check ADP software versions for ADP Intel\u00ae FPGA PAC D5005 Project
The user can also modify the build time by de-selecting options they do not wish to use, see below for a couple of use-case scenarios.
Default User Case
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 57 from the main menu the script will execute 23 tests ie (main menu options 2, 9, 10, 11, 12, 13, 14, 15, 27, 29, 30, 32, 34, 35, 39, 40, 48, 50, 51, 52, 53, 54 and 55. All other tests with an \"X\" indicates do not run that test
User Case for ADP FIM/PR BUILD MENU
In the example below when the user selects option 57 from the main menu the script will only run options from the ADP FIM/PR BUILD MENU (7 options, main menu options 9, 10, 11, 12, 13, 14 and 15). All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#4-common-test-scenarios","title":"4 Common Test Scenarios","text":"This section will describe the most common compile build scenarios if a user wanted to evaluate an acceleration card on their server. The Pre-requisite column indicates the menu commands that must be run before executing the test eg To run Test 5 then a user needs to have run option 10, 12 and 13 before running options 20, 21, 22, 27 and 28.
Test Test Scenario Pre-Requisite Menu Option Menu Option Test 1 FIM Build - 10 Test 2 Partial Reconfiguration Build 10 12 Test 3 Program FIM and perform Remote System Upgrade 10 18, 19 Test 4 Bind PF and VF to vfio-pci drivers - 20, 21, 22 Test 5 Build, compile and test AFU on hardware 10, 12, 13 20, 21, 22, 27, 28 Test 6 Build, compile and test AFU Basic Building Blocks on hardware 10, 12, 13 20, 21, 22, 32, 33 Test 7 Build, compile and test oneAPI on hardware 10, 12, 13 34, 35, 36, 39, 40, 44, 45, 46, 47, 48, 49 Test 8 Build and Simulate Unit Tests - 50, 51 Test 9 Build and Simulate UVM Tests - 52, 53, 54, 55"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/","title":"Getting Started Guide: Open FPGA Stack for Intel Stratix 10","text":"Last updated: November 19, 2024
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#10-introduction","title":"1.0 Introduction","text":"This document helps users get started in evaluating Open FPGA Stack (OFS) for Stratix\u00ae 10 FPGA targeting the Intel\u00ae FPGA PAC D5005. After reviewing the document a user shall be able to:
- Set up a development environment with all OFS ingredients
- Build and install the OFS Linux Kernel drivers on the host
- Build and install the Open Programmable Acceleration Engine Software Development Kit (OPAE SDK) on the host
- Flash an OFS FIM binary onto the Intel\u00ae FPGA PAC D5005
- Verify the functionality of OFS on an Intel\u00ae FPGA PAC D5005 board
- Know where to find additional information on all OFS ingredients
The following flow charts show a high level overview of the initial bring-up process, split into three sequential diagrams.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#diagram-1-installing-the-opae-sdk","title":"Diagram 1: Installing the OPAE SDK","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#diagram-2-installing-the-linux-dfl-drivers","title":"Diagram 2: Installing the Linux DFL Drivers","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#diagram-3-bringing-up-the-intel-d5005","title":"Diagram 3: Bringing up the Intel D5005","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#11-intended-audience","title":"1.1 Intended Audience","text":"The information in this document is intended for customers evaluating the Open FPGA Stack for Stratix\u00ae 10 FPGA on the Intel PAC D5005. This document will cover key topics related to initial setup and development, with links for deeper dives on the topics discussed therein.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#12-terminology","title":"1.2 Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#13-reference-documents","title":"1.3 Reference Documents","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#14-component-version-summary","title":"1.4 Component Version Summary","text":"The OFS 2024.1-1 Release targeting the Stratix\u00ae 10 FPGA is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions which comprise this release.
The following table highlights the hardware which makes up the Best Known Configuration (BKC) for the OFS 2024.1-1 release.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-1-2-hardware-bkc","title":"Table 1-2: Hardware BKC","text":"Component 1 x Intel\u00ae FPGA PAC D5005 1 x Supported Server Model 1 x Intel FPGA Download Cable II (Optional, only required if loading images via JTAG) The following table highlights the versions of the software which comprise the OFS stack. The installation of the user-space OPAE SDK on top of the kernel-space linux-dfl drivers is discussed in subsequent sections of this document.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-1-3-software-version-summary","title":"Table 1-3: Software Version Summary","text":"Component Version FPGA Platform Intel\u00ae FPGA PAC D5005 OPAE SDK Tag: 2.12.0-5 Kernel Drivers Tag: ofs-2024.1-6.1-2 OFS FIM Source Code Branch: ofs-2024.1-1 Intel Quartus Prime Pro Edition Design Software 23.4 [Intel\u00ae Quartus\u00ae Prime Pro Edition Linux] Operating System RHEL 8.6 A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae FPGA PAC D5005 can be found on the OFS 2024.1-1 official release drop on GitHub.
Note: If you wish to freeze your Red Hat operating system version on the RHEL 8.6, refer to the following solution provided in the Red Hat customer portal.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#15-host-bios","title":"1.5 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
- Intel VT for Directed I/O (VT-d) must be enabled
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#16-host-server-kernel-and-grub-configuration","title":"1.6 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
- OS: RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.6
- Kernel: 6.1.78
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
OFS is a blanket term which can be used to collectively refer to all ingredients of the OFS reference design, which includes the core hardware components discussed below and software.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM) or 'shell' provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The FIM is implemented in a static region of the FPGA device.
The primary components of the FIM reference design are:
- PCIe Subsystem
- Transceiver Subsystem
- Memory Subsystem
- FPGA Management Engine
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from host or AFU
- Interface to Board Management Controller (BMC)
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration.
For more information on the FIM and its external connections, please refer to the Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs, and the Intel FPGA Programmable Acceleration Card D5005 Data Sheet. Below is a high-level block diagram of the FIM.
Figure 2-1 FIM Overview
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required for partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capabilities to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your AFU resides in a partial reconfiguration (PR) region of the FPGA.
- Your AFU is a part of the static region (SR) and is a compiled flat design.
- Your AFU contains both static and PR regions.
The AFU provided in this release is comprised of the following functions:
- AFU interface handler to verify transactions coming from the AFU region.
- PV/VF Mux to route transactions to and from corresponding AFU components, including the ST2MM module, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), and Memory Host Exerciser (HE-MEM).
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads).
- Port gasket and partial reconfiguration support.
For more information on the Platform Interface Manager (PIM) and AFU development and testing, please refer to the [OFS AFU Development Guide].
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#22-ofs-software-overview","title":"2.2 OFS Software Overview","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#221-kernel-drivers-for-ofs","title":"2.2.1 Kernel Drivers for OFS","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on the DFL Wiki page.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#30-intel-fpga-pac-d5005-card-installation-and-server-requirements","title":"3.0 Intel FPGA PAC D5005 Card Installation and Server Requirements","text":"Currently OFS for Stratix\u00ae 10 FPGA targets the Intel\u00ae FPGA PAC D5005. Because the Intel\u00ae FPGA PAC D5005 is a production card, you must prepare the card in order to receive a new non-production bitstream. For these instructions, please contact an Intel representative.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#40-ofs-dfl-kernel-drivers","title":"4.0 OFS DFL Kernel Drivers","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#41-ofs-dfl-kernel-driver-installation","title":"4.1 OFS DFL Kernel Driver Installation","text":"All OFS DFL kernel driver code resides in the Linux DFL GitHub repository. This repository is open source and does not require any permissions to access. It includes a snapshot of the latest best-known configuration (BKC) Linux kernel with the OFS driver included in the drivers/fpga/* directory. Downloading, configuration, and compilation will not be discussed in this document. Please refer to the Software Installation Guide: OFS for PCIe Attach FPGAs for guidelines on environment setup and build steps for all OFS stack components.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.1-1 Release Page.
It is recommended you boot into your operating system's native 4.18.x kernel before attempting to upgrade to the dfl enabled 6.1.78 You may experience issues when moving between two dfl enabled 6.1.78 kernels.
This installation process assumes the user has access to an internet connection in order to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#42-ofs-dfl-kernel-driver-installation-environment-setup","title":"4.2 OFS DFL Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.6 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 4.4 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
-
It is recommended you lock your Red Hat release version to RedHatEnterprise Linux\u00ae (RHEL) 8.6 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.6\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
-
Install the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#43-building-and-installing-the-ofs-dfl-kernel-drivers-from-source","title":"4.3 Building and Installing the OFS DFL Kernel Drivers from Source","text":"It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl\ncd /home/OFS/linux-dfl\ngit checkout tags/ofs-2024.1-6.1-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n
2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nofs-2024.1-6.1-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n
3. Copy an existing kernel configuration file from /boot and apply the minimal required settings changes.
```bash\ncd /home/OFS/linux-dfl\ncp /boot/config-`uname -r` .config\ncat configs/dfl-config >> .config\necho 'CONFIG_LOCALVERSION=\"-dfl\"' >> .config\necho 'CONFIG_LOCALVERSION_AUTO=y' >> .config\nsed -i -r 's/CONFIG_SYSTEM_TRUSTED_KEYS=.*/CONFIG_SYSTEM_TRUSTED_KEYS=\"\"/' .config\nsed -i '/^CONFIG_DEBUG_INFO_BTF/ s/./#&/' .config\necho 'CONFIG_DEBUG_ATOMIC_SLEEP=y' >> .config\nexport LOCALVERSION=\n```\n
Note: If you wish to add an identifier to the kernel build, edit .config and make your additions to the line CONFIG_LOCALVERSION=\"\".
4. The above command may report errors resembling symbol value 'm' invalid for CHELSIO_IPSEC_INLINE. These errors indicate that the nature of the config has changed between the currently executing kernel and the kernel being built. The option \"m\" for a particular kernel module is no longer a valid option, and the default behavior is to simply turn the option off. However, the option can likely be turned back on by setting it to 'y'. If the user wants to turn the option back on, change it to 'y' and re-run \"make olddefconfig\":
```bash\ncd /home/OFS/linux-dfl\necho 'CONFIG_CHELSIO_IPSEC_INLINE=y' >> .config\n```\n\n*Note: To use the built-in GUI menu for editing kernel configuration parameters, you can opt to run `make menuconfig`.*\n
5. Linux kernel builds take advantage of multiple processors to parallelize the build process. Display how many processors are available with the nproc command, and then specify how many make threads to utilize with the -j option. Note that number of threads can exceed the number of processors. In this case, the number of threads is set to the number of processors in the system.
```bash\ncd /home/OFS/linux-dfl\nmake -j $(nproc)\n```\n
6. You have two options to build the source:
- Using the built-in install option from the kernel Makefile.\n- Locally building a set of RPM/DEP packages.\n\nThis first flow will directly install the kernel and kernel module files without the need to create a package first:\n\n```bash\ncd /home/OFS/linux-dfl\nsudo make modules_install -j $(nproc)\nsudo make install\n```\n\nIn this second flow, the OFS Makefile contains a few options for package creation:\n\n- rpm-pkg: Build both source and binary RPM kernel packages\n- binrpm-pkg: Build only the binary kernel RPM package\n- deb-pkg: Build both source and binary deb kernel packages\n- bindeb-pkg: Build only the binary kernel deb package\n\nIf you are concerned about the size of the resulting package and binaries, they can significantly reduce the size of the package and object files by using the make variable INSTALL_MOD_STRIP. If this is not a concern, feel free to skip this step. The below instructions will build a set of binary RPM packages:\n\n```bash\ncd /home/OFS/linux-dfl\nmake INSTALL_MOD_STRIP=1 binrpm-pkg -j `nproc`\n```\n\nIf the kernel development package is necessary for other software you plan on installing outside of OFS, you should instead use the build target `rpm-pkg`.\n\nBy default, a directory is created in your home directory called `rpmbuild`. This directory will house all the kernel packages which have been built. You need to navigate to the newly built kernel packages and install them. The following files were generated using the build command executed in the previous step:\n\n```bash\ncd ~/rpmbuild/RPMS/x86_64\nls\nkernel-4.18.0_dfl.x86_64.rpm kernel-headers-6.1.78_dfl.x86_64.rpm\nsudo dnf localinstall kernel*.rpm\n```\n
7. The system will need to be rebooted in order for changes to take effect. After a reboot, select the newly built kernel as the boot target. This can be done pre-boot using the command grub2-reboot, which removes the requirement for user intervention. After boot, verify that the currently running kernel matches expectation.
```bash\nuname -r\n6.1.78-dfl\n```\n
8. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/6.1.78-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\n
If an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n
9. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n
10. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n
11. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n
12. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-6.1.78-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\n
A list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#44-installing-the-ofs-dfl-kernel-drivers-from-pre-built-packages","title":"4.4 Installing the OFS DFL Kernel Drivers from Pre-Built Packages","text":"To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page. You can choose to either install using the SRC RPMs, or to use the pre-built RPM packages targeting the official supported release platform.
tar xf kernel-6.1.78_dfl-1.x86_64-*.tar.gz\n\nsudo dnf localinstall kernel-6.1.78_dfl_*.x86_64.rpm \\\nkernel-devel-4.18.0_dfl_*.x86_64.rpm \\\nkernel-headers-4.18.0_dfl_*.x86_64.rpm\n\n### OR\nsudo dnf localinstall kernel-6.1.78_dfl_*.src.rpm\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#50-opae-software-development-kit","title":"5.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE GitHub. This repository is open source.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#51-opae-sdk-installation","title":"5.1 OPAE SDK Installation","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 5.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#52-opae-sdk-installation-environment-setup","title":"5.2 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package. -
Remove any currently installed OPAE packages.
sudo dnf remove opae*\n
-
Initialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.12.0-5\n
-
Verify that the correct tag/branch have been checkout out.
git describe --tags\n2.12.0-5\n
-
Set up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\n
The following packages will be built in the same directory as create:
-
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\n
-
Check that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.12.0-5.el8.x86_64.rpm\nopae-debuginfo-2.12.0-5.el8.x86_64.rpm\nopae-debugsource-2.12.0-5.el8.x86_64.rpm\nopae-devel-2.12.0-5.el8.x86_64.rpm\nopae-devel-debuginfo-2.12.0-5.el8.x86_64.rpm\nopae-extra-tools-2.12.0-5.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.12.0-5.el8.x86_64.rpm\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#53-installing-the-opae-sdk-with-pre-built-packages","title":"5.3 Installing the OPAE SDK with Pre-Built Packages","text":"You can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.12.0-5.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.12.0-5.x86_64-*.tar.gz\n
For a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#54-opae-tools-overview","title":"5.4 OPAE Tools Overview","text":"The OPAE SDK user-space tools sit upon the kernel-space DFL drivers. In order to use OPAE SDK functionality the user needs to have installed both the OPAE SDK and Linux DFL driver set. You must have at least one D5005 card with the appropriate FIM present in your system. The steps to read and load a new FIM version are discussed in section 7.1 Programming the OFS FIM. After both the DFL kernel-space drivers have been installed and the FIM has been upgraded, you may proceed to test the OPAE commands discussed below.
This section covers basic functionality of the commonly used OPAE tools and their expected results. These steps may also be used to verify that all OFS software installation has been completed successfully. A complete overview of the OPAE tools can be found on the OPAE GitHub and in your cloned GitHub repo at <your path>/opae-sdk/doc/src/fpga_tools. More commands are listed than are defined in the list below - most of these are called by other tools and do not need to be called directly themselves.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#541-fpgasupdate","title":"5.4.1 fpgasupdate","text":"The fpgasupdate tool updates the Intel Max10 BMC image and firmware, root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate will only accept images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then the image will also need to be signed using the correct keys. Please refer to the [Security User Guide: Intel Open FPGA Stack] for information on created signed images and on programming and managing the root entry hash.
The Intel FPGA PAC ships with a factory and user programmed image for both the FIM and BMC FW and RTL on all cards.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-5-1-fpgasupdate-overview","title":"Table 5-1: fpgasupdate Overview","text":"Synopsis:
fpgasupdate [--log-level=<level>] file [bdf]\n
Description: The fpgasupdate command implements a secure firmware update.
Command args (optional) Description --log-level Specifies the log-level which is the level of information output to your command tool. The following seven levels are available: state, ioctl, debug, info, warning, error, critical. Setting --log-level=state provides the most verbose output. Setting --log-level=ioctl provides the second most information, and so on. The default level is info. file Specifies the secure update firmware file to be programmed. This file may be to program a static region (SR), programmable region (PR), root entry hash, key cancellation, or other device-specific firmware. bdf The PCIe address of the PAC to program. bdf is of the form [ssss:]bb:dd:f, corresponding to PCIe segment, bus, device, function. The segment is optional. If you do not specify a segment, the segment defaults to 0000. If the system has only one PAC you can omit the bdf and let fpgasupdate determine the address automatically."},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#542-fpgainfo","title":"5.4.2 fpgainfo","text":"Synopsis:
fpgainfo [-h] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\n{errors,power,temp,fme,port,bmc,mac,phy,security}\n
Description: Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Command args (optional) Description --help, -h Prints help information and exits. --version, -v Prints version information and exits. -S, --segment PCIe segment number of resource. -B, --bus PCIe bus number of resource. -D, --device PCIe device number of resource. -F, --function PCIe function number of resource. errors {fme, port, all} --clear, -c First agument to the errors command specifies the resource type to display in human readable format. The second optional argument clears errors for the given FPGA resource. power Provides total power in watts that the FPGA hardware consumes temp Provides FPGA temperature values in degrees Celsius port Provides information about the port fme Provides information about the FME bmc Provides BMC sensors information mac Provides information about MAC ROM connected to FPGA security Provides information about the security keys, hashes, and flash count, if available. Note: Your Bitstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo.
$ sudo fpgainfo fme\n\nOpen FPGA Stack Platform\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\n
$ sudo fpgainfo bmc\n\nOpen FPGA Stack Platform\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** BMC SENSORS ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\n( 1) VCCERAM Voltage : 0.90 Volts\n( 2) VCCT Temperature : 29.00 Celsius\n( 3) 12v Backplane Voltage : 12.17 Volts\n( 4) VCCERAM Current : 0.18 Amps\n( 5) FPGA Transceiver Temperature : 36.50 Celsius\n( 6) QSFP1 Supply Voltage : 0.00 Volts\n( 7) 3.3v Temperature : 29.00 Celsius\n( 8) 12v Backplane Current : 2.28 Amps\n( 9) RDIMM3 Temperature : 25.50 Celsius\n(10) VCCR Voltage : 1.12 Volts\n(11) Board Inlet Air Temperature : 24.50 Celsius\n(12) 1.8v Temperature : 27.50 Celsius\n(13) 12v AUX Voltage : 12.14 Volts\n(14) VCCR Current : 0.55 Amps\n(15) RDIMM0 Temperature : 24.50 Celsius\n(16) FPGA Core Voltage : 0.88 Volts\n(17) VCCERAM Temperature : 27.50 Celsius\n(18) 12v AUX Current : 1.19 Amps\n(19) QSFP0 Temperature : N/A\n(20) VCCT Voltage : 1.12 Volts\n(21) FPGA Core Current : 11.60 Amps\n(22) FPGA Core Temperature : 42.50 Celsius\n(23) 12v Backplane Temperature : 24.00 Celsius\n(24) VCCT Current : 0.14 Amps\n(25) RDIMM1 Temperature : 24.00 Celsius\n(26) 3.3v Voltage : 3.30 Volts\n(27) VCCR Temperature : 33.50 Celsius\n(28) 1.8v Voltage : 1.80 Volts\n(29) 3.3v Current : 0.32 Amps\n(30) Board Exhaust Air Temperature : 26.00 Celsius\n(31) 12v AUX Temperature : 25.00 Celsius\n(32) QSFP0 Supply Voltage : 0.00 Volts\n(33) QSFP1 Temperature : N/A\n(34) 1.8v Current : 0.54 Amps\n(35) RDIMM2 Temperature : 26.00 Celsius\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#543-rsu","title":"5.4.3 rsu","text":"The rsu performs a Remote System Update operation on a device, given its PCIe address. A rsu operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either the BMC or FPGA.
The Intel FPGA PAC contains a region of flash the user may store their FIM image. After an image has been programmed with fpgasupdate the user may choose to perform rsu to update the image on the device.
Note: The D5005 platform only supports storing and configuring a single user image from flash for the FPGA. It does not include support for the user1/user2 partitions as shown in other OFS related acceleration boards.
rsu Overview
Synopsis
rsu [-h] [-d] {bmc,bmcimg,retimer,sdm,fpgadefault} [PCIE_ADDR]\n
rsu bmc --page=(user) [PCIE_ADDR]\nrsu retimer [PCIE_ADDR]\nrsu sdm [PCIE_ADDR]\n
Perform RSU (remote system update) operation on PAC device given its PCIe address. An RSU operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either BMC, Retimer, SDM, (on devices that support these) or the FPGA.
Note: As a result of using the rsu command, the host rescans the PCI bus and may assign a different Bus/Device/Function (B/D/F) value than the originally assigned value.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#544-pacsign","title":"5.4.4 PACsign","text":"PACSign is an OPAE utility which allows users to insert authentication markers into bitstreams targeted for the platform. All binary images must be signed using PACSign before fpgasupdate can use them for an update. Assuming no Root Entry Hash (REH) has been programmed on the device, the following examples demonstrate how to prepend the required secure authentication data, and specify which region of flash to update. More information, including charts detailing the different certification types and their required options, are fully described in the PACsign python/pacsign/PACSign.md OPAE GitHub on GitHub.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-5-4-pacsign-overview","title":"Table 5-4: PACSign Overview","text":"Synopsis:
PACSign [-h] {FIM,SR,SR_TEST,BBS,BMC,BMC_FW,BMC_FACTORY,AFU,PR,PR_TEST,GBS,FACTORY,PXE,THERM_SR,THERM_PR} ...\n\nPACSign <CMD> [-h] -t {UPDATE,CANCEL,RK_256,RK_384} -H HSM_MANAGER [-C HSM_CONFIG] [-s SLOT_NUM] [-r ROOT_KEY] [-k CODE_SIGNING_KEY] [-d CSK_ID] [-R ROOT_BITSTREAM] [-S] [-i INPUT_FILE] [-o OUTPUT_FILE] [-b BITSTREAM_VERSION] [-y] [-v]\n
Description: The PACSign utility inserts authentication markers into bitstreams.
Command args (optional) Description (required) -t, --cert_type TYPE The following operations are supported: UPDATE, CANCEL, RK_256, RK_348 (required) -H, --HSM_manager MODULE The module name for a module that interfaces to a HSM. PACSign includes both the openssl_manager and pkcs11_manager to handle keys and signing operations. -C, --HSM_config CONFIG The argument to this operation is passed verbatim to the specified HSM. For pkcs11_manager, this option specifies a JSON file describing the PKCS #11 capable HSM\u2019s parameters. -r, --root_key KEY_ID The key identifier that the HSM uses to identify the root key to be used for the selected operation. -k, --code_signing_key KEY_ID The key indentifier that the HSM uses to identify the code signing key to be used for the selected operation -d, --csk_id CSK_NUM Only used for type CANCEL. Specifies the key number of the code signing key to cancel. -s, --slot_num For bitstream types with multiple slots (i.e. multiple ST regions), this option specifies which of the slots to which this bitstream is to be acted upon -b, --bitstream_version VERSION User-formatted version information. This can be any string up to 32 bytes in length. -S, --SHA384 Used to specify that PACSign is to use 384-bit crypto. Default is 256-bit -R, --ROOT_BITSTREAM ROOT_BITSTREAM Valid when verifying bitstreams. The verification step will ensure the generated bitstream is able to be loaded on a board with the specified root entry hash programmed. -i, --input_file FILE Only to be used for UPDATE operations. Specifies the file name containing data to be signed -o, --output_file FILE Specifies the file name for the signed output bitstream. -y, --yes Silently answer all queries from PACSign in the affirmative. -v, --verbose Can be specified multiple times. Increases the verbosity of PACSign. Once enables non-fatal warnings to be displayed. Twice enables progress information. Three or more occurrences enables very verbose debugging information. -h Prints help information and exits {FIM, SR, SR_TEST, BBS, BMC, BMC_FW, BMC_FACTORY, AFU, PR, PR_TEST, GBS, FACTORY, PXE, THERM_SR, THERM_PR} Bitstream type identifier. PACSign can be run on images that have previously been signed. It will overwrite any existing authentication data.
The following example will create an unsigned SR image from an existing signed SR binary update image.
PACSign SR -t UPDATE -s 0 -H openssl_manager -i d5005_page1_unsigned.bin -o new_image.bin\n#output\nNo root key specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo CSK specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo root entry hash bitstream specified. Verification will not be done. Continue? Y = yes, N = no: y\n2022-07-20 10:13:54,954 - PACSign.log - WARNING - Bitstream is already signed - removing signature blocks\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#545-bitstreaminfo","title":"5.4.5 bitstreaminfo","text":"Displays authentication information contained with each provided file on the command line. This includes any JSON header strings, authentication header block information, and a small portion of the payload. The binary is installed by default at /usr/bin/bitstreaminfo.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#546-hssi","title":"5.4.6 hssi","text":"The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode-specific options. Only the hssi_10g MODE is currently supported. An example of this command's output can be found in section 5.2.9 Running the Host Exerciser Modules. The binary is installed by default at /usr/bin/hssi.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#547-opaeio","title":"5.4.7 opae.io","text":"Opae.io is a interactive Python environment packaged on top of libopaevfio.so, which provides user space access to PCIe devices via the vfio-pci driver. The main feature of opae.io is its built-in Python command interpreter, along with some Python bindings that provide a means to access Configuration and Status Registers (CSRs) that reside on the PCIe device. opae.io has two operating modes: command line mode and interactive mode. An example of this command's output can be found in section 5.2.9 Running the Host Exerciser Modules. The binary is installed by default at /usr/bin/opae.io.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#548-host_exerciser","title":"5.4.8 host_exerciser","text":"The host exerciser is used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. An example of this command's output can be found in section 5.2.9 Running the Host Exerciser Modules. The binary is installed by default at /usr/bin/host_exerciser. For more information refer to - Host Exerciser
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#549-running-the-host-exerciser-modules","title":"5.4.9 Running the Host Exerciser Modules","text":"The reference FIM and unchanged compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc.
Note: Before continuing, if huge pages are not set refer to section 4.2, step 7.
There are three HEMs present in the OFS FIM - HE-LPBK, HE-HSSI, and HE-MEM. These exercisers are tied to three different VFs that must be enabled before they can be used. The user should enable the VF for each HEM using the below steps:
1. Determine the BDF of the Intel\u00ae FPGA PAC D5005 card.
The PCIe BDF address is initially determined when the server powers on. The user can determine the addresses of all Intel\u00ae FPGA PAC D5005 boards using lspci:
lspci -d :bcce\n\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
Note: Before continuing, if you updated your OFS installation, please also update your PAC FIM to run HEMs.
2. Enable three VFs.
In this example, the BDF address is 0000:3b:00.0. With this information the user can now enable three VFs with the following:
sudo pci_device 0000:3b:00.0 vf 3\n
3. Verify that all three VFs have been created.
lspci -s 3b:00\n\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.1 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.2 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.3 Processing accelerators: Intel Corporation Device bccf (rev 01)\n
4. Bind the 3 VFs to the vfio-pci driver.
sudo opae.io init -d 0000:3b:00.1 $USER\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:3b:00.1 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:3b:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.1 is 142\nAssigning /dev/vfio/142 to $USER:$USER\nChanging permissions for /dev/vfio/142 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.2 $USER\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:3b:00.2 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:3b:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.2 is 143\nAssigning /dev/vfio/143 to $USER:$USER\nChanging permissions for /dev/vfio/143 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.3 $USER\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:3b:00.3 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:3b:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.3 is 144\nAssigning /dev/vfio/144 to $USER:$USER\nChanging permissions for /dev/vfio/144 to rw-rw----\n
5. Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
$ sudo fpgainfo port\n\n//****** PORT ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x603B000000000000\nPCIe s:b:d.f : 0000:3B:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0x403B000000000000\nPCIe s:b:d.f : 0000:3B:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x203B000000000000\nPCIe s:b:d.f : 0000:3B:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-5-5-vf-to-hem-mappings","title":"Table 5-5 VF to HEM Mappings","text":"VF BDF HEM BBBB:DD.1 HE-LB BBBB:DD.2 HE-MEM BBBB:DD.3 He-HSSI HE-MEM / HE-LB
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise the DDR interface; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. The following commands are supported by the HE-LB/HE-MEM OPAE driver program. They may need to be run using sudo privileges, depending on your server configuration.
Basic operations:
$ sudo host_exerciser lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5342\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.067 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode lpbk lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5358\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.058 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode write lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 0\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2592\nTotal number of Reads sent: 0\nTotal number of Writes sent: 1024\nBandwidth: 6.321 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode trput lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 3384\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 4.842 GB/s\n Test lpbk(1): PASS\n
Number of cachelines per request 1, 2, and 4. The user may replace --mode lpbk with read, write, trput. The target lpbk can be replaced with mem:
$ sudo host_exerciser --mode lpbk --cls cl_1 lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5475\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 2.993 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode lpbk --cls cl_2 lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5356\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.059 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode lpbk --cls cl_4 lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 4481\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.656 GB/s\n Test lpbk(1): PASS\n
Interrupt tests (only valid for mode mem):
$ sudo host_exerciser --interrupt 0 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5140\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.188 GB/s\n Test mem(1): PASS\n\n$ sudo host_exerciser --interrupt 1 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5079\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.226 GB/s\n Test mem(1): PASS\n\n$ sudo host_exerciser --interrupt 2 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5525\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.439 GB/s\n Test mem(1): PASS\n\n$ sudo host_exerciser --interrupt 3 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 4735\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.460 GB/s\n Test mem(1): PASS\n
HE-HSSI
HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G ethernet AFU and includes a 10G traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic. Context sensitive information is given by the hssi --help command. Help for the 10G specific test is given by hssi hssi_10g --help Example useage:
$ sudo hssi --pci-address 3b:00.3 hssi_10g --eth-ifc s10hssi0 --eth-loopback on --he-loopback=off --num-packets 100\n10G loopback test\nport: 0\neth_loopback: on\n he_loopback: off\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth: s10hssi0\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#60-compiling-ofs-fim","title":"6.0 Compiling OFS FIM","text":"Pre-Compiled FIM binaries are at OFS 2024.1-1 release page and to compile the OFS FIM for Intel\u00ae FPGA PAC D5005 follow the below steps :
1) Compile OFS FIM manually - Steps are provided in the developer guide to compile FIM and generate binaries. Refer to Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
2) Compile OFS FIM using evaluation script - The script guides you to the steps required for compilation via selecting options from the menu. Refer to Automated Evaluation User Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#70-programming-the-ofs-fim-and-bmc","title":"7.0 Programming the OFS FIM and BMC","text":"Instructions surrounding the compilation and simulation of the OFS FIM have fully moved into the Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#71-programming-the-ofs-fim","title":"7.1 Programming the OFS FIM","text":"In order to program the OFS FIM, both the OPAE SDK and DFL drivers need to be installed on the host system. Please complete the steps in sections 4.0 OFS DFL Kernel Drivers and 5.0 OPAE Software Development Kit. The OFS FIM version can be identified using the OPAE tool fpgainfo. A sample output of this command is included below.
$ sudo fpgainfo fme\n\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\n
Use the value under PR Interface ID to identify that FIM that has been loaded. Refer to the table below for a list of previous FIM releases:
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-7-1-previous-fim-releases","title":"Table 7-1 Previous FIM Releases","text":"PR Release PR Interface ID 2023.2 edad864c-99d6-5831-ab67-62bfd81ec654 2022.2 (tag 1.3.0) bf531bcf-a896-5171-ab31-601a4ab754b6 2022.1 Beta (tag: 1.2.0-beta) 2fae83fc-8568-53aa-9157-8f75e9c0ba92 OFS 2.1 Beta (tag: 1.1.0-beta) 99160d37e42a 3f8b586f-c275-594c-92e2-d9f2c23e94d1 OFS 1.0 (tag: ofs-1.0.0) b5f6a71e-daec-59c3-a43a-85567b51fd3f Intel Acceleration Stack for Intel\u00ae FPGA PAC D5005 2.0.1 9346116d-a52d-5ca8-b06a-a9a389ef7c8d If the user's card does not report a PR Interface ID which matches the above table, then a new FIM will need to be programmed.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#711-programming-the-fim","title":"7.1.1 Programming the FIM","text":"1. Download the file d5005_page1_unsigned.bin from OFS 2024.1-1 release page.
2. Run PACSign to create an unsigned image with added header for use by fpgasupdate
$ PACSign SR -y -v -t UPDATE -s 0 -H openssl_manager -i d5005_page1_unsigned.bin -o d5005_PACsigned_unsigned.bin\n
3. Run fpgasupdate to load the image into the user location of the Intel\u00ae FPGA PAC D5005 FPGA flash, NOTE: use \"sudo fpgainfo fme\" command to find the PCIe address for your card.
$ sudo fpgasupdate d5005_PACsigned_unsigned.bin 3B:00.0\n
4. Run RSU command.
$ sudo rsu bmcimg 0000:3B:00.0\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#72-programming-the-bmc","title":"7.2 Programming the BMC","text":"1. Download intel-fpga-bmc images(To download OFS Stratix 10 BMC binaries contact Intel Technical Sales Representative)
2. The file unsigned_bmc_fw.bin has the newly binary format. This bitstream is programmed with remote system update (RSU) and the bitstream must be signed with PACSign tool to generate.
3. Run PACSign to create an unsigned image with added header for use by fpgasupdate
$ PACSign BMC -y -v -t UPDATE -s 0 -H openssl_manager -i unsigned_bmc_fw.bin -o PACsigned_unsigned_bmc_fw.bin\n\n2022-04-22 03:07:05,626 - PACSign.log - INFO - OpenSSL version \"OpenSSL 1.1.1k FIPS 25 Mar 2021\" matches \"1.1.1\"\n2022-04-22 03:07:05,648 - PACSign.log - INFO - Bitstream not previously signed\n2022-04-22 03:07:05,648 - PACSign.log - INFO - platform value is '688128'\n2022-04-22 03:07:05,745 - PACSign.log - INFO - Starting Block 0 creation\n2022-04-22 03:07:05,745 - PACSign.log - INFO - Calculating SHA256\n2022-04-22 03:07:05,747 - PACSign.log - INFO - Calculating SHA384\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Done with Block 0\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Root Entry creation\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Calculating Root Entry SHA\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Code Signing Key Entry creation\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Calculating Code Signing Key Entry SHA\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Code Signing Key Entry done\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Block 0 Entry creation\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Calculating Block 0 Entry SHA\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Block 0 Entry done\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Block 1 creation\n2022-04-22 03:07:05,750 - PACSign.log - INFO - Block 1 done\n2022-04-22 03:07:05,757 - PACSign.log - INFO - Writing blocks to file\n2022-04-22 03:07:05,758 - PACSign.log - INFO - Processing of file 'PACsigned_unsigned_bmc_fw.bin' complete\n
4. Run fpgasupdate to perform an upgrade of the BMC.
$ sudo fpgasupdate PACsigned_unsigned_bmc_fw.bin 3B:00.0\n\n[2022-04-22 03:08:34.15] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-22 03:08:34.15] [INFO ] updating from file pacsign_unsigned_bmc_fw.bin with size 819968\n[2022-04-22 03:08:34.15] [INFO ] waiting for idle\n[2022-04-22 03:08:34.15] [INFO ] preparing image file\n[2022-04-22 03:09:02.18] [INFO ] writing image file\n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [819968/819968 bytes][Elapsed Time: 0:00:13.01]\n[2022-04-22 03:09:15.20] [INFO ] programming image file\n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:00:29.03]\n[2022-04-22 03:09:44.24] [INFO ] update of 0000:3B:00.0 complete\n[2022-04-22 03:09:44.24] [INFO ] Secure update OK\n[2022-04-22 03:09:44.24] [INFO ] Total time: 0:01:10.08\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/","title":"UVM Simulation User Guide: Open FPGA Stack for Intel Stratix\u00ae 10 FPGA","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#1-overview","title":"1 Overview","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the UVM simulation tool using OFS. After reviewing the document, you will be able to:
- Set-up the UVM verification tool suite
- Run pre-existing UVM unit tests and also create new UVM tests for your design
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#2-introduction-to-uvm","title":"2 Introduction to UVM","text":"OFS (Open FPGA Stack) provides a UVM (Universal Verification Methodology) environment for the FIM (FPGA Interface Manager) with a modular, reusable, and scalable testbench structure via an API framework.
The framework consists of a FIM Testbench which is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this Testbench. UVM RAL (Register Abstaction Layer) is used for CSR (Command and Status Registers) verification.
The qualified verification IPs will help to detect incorrect protocol behavior, help to focus on FIM features and accelerate the verification process.
Verification components include:
- FIM monitor to detect correct design behavior
- FIM assertions for signal level integrity testing
- Arm AMBA Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity
- FIM coverage to collect functional data
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#3-testbench-architecture","title":"3 Testbench Architecture","text":"The testbench connects to the full chip that includes major RTL blocks depicted in Figure 1.
Figure 1 Testbench Diagram
The major interface is between the Xeon and FPGA where PCIe Verification IP is connected to PCIe Subsystem. Therefore, as a full chip simulation environment, PCIe host VIP is the sole VIP/BFM used. PCIe host VIP connects to PCIe device which resides in FPGA in serial mode.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#4-testbench-infrastructure","title":"4 Testbench Infrastructure","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#41-traffic-flow","title":"4.1 Traffic Flow","text":"PCIe Host, as the master of FPGA, initiates MMIO read/write requests to FPGA to program registers. The PCIe host also passively receives memory requests from FPGA to read from or write to host memory.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#42-link-up-and-enumeration","title":"4.2 Link Up and Enumeration","text":"With serial mode connection between PCIe host and device, link training and enumeration has to be done before the regular traffic starts.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#421-link-up","title":"4.2.1 Link Up","text":"Linkup sequence(pcie_device_bring_up_link_sequence) is part of configure sequence(ofs_config_seq), which is started in UVM configure phase.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#422-enumeration","title":"4.2.2 Enumeration","text":"PCIe host driver needs to retrieve information from the device hard IP and program necessary configuration space registers, such as PF/VF BAR values. This is done in enumerate_seq, which follows link up sequence in configure sequence(ofs_config_seq).
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#423-pfvf-bar","title":"4.2.3 PF/VF BAR","text":"PF0 BAR0 is set in the base sequence and can be randomized. During enumeration, PF0 BAR0, along with PCIe device hard IP configuration, derives other PF and VF BAR values. These BAR values are stored into base sequence variables and can be used throughout any test sequences that extend the base sequence.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#43-mmio-apis","title":"4.3 MMIO APIs","text":"The base sequence provides APIs for 32-bit and 64-bit MMIO read/write accesses, as well as blocking or non-blocking for MMIO read as described in Table 1. The users can use MMIO APIs without knowing the underlining PCIe sequence items.
Name API 32-bit MMIO Write task mmio_write32(input bit [63:0] addr_, input bit [31:0] data_); 64-bit MMIO Write task mmio_write64(input bit [63:0] addr_, input bit [63:0] data_); 32-bit MMIO Read task mmio_read32(input bit [63:0] addr_, output bit [31:0] data_, input blocking_ = 1); 64-bit MMIO Read task mmio_read64(input bit [63:0] addr_, output bit [63:0] data_, input blocking_ = 1); Table 1 MMIO APIs
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#44-ral","title":"4.4 RAL","text":"UVM RAL is integrated in the testbench providing alternative ways of accessing CSRs in test sequences. RAL is generated from an excel format CSR specification where register name, field, offset, bitmap, attribute, and description are specified.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#45-vip-dut-connection","title":"4.5 VIP DUT Connection","text":"PCIe host verification IP and DUT connection is achieved by connecting 16 bits lanes. The module for connection from VIP is svt_pcie_device_agent_serdes_x16_x8g_hdl.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#5-test-plan","title":"5 Test Plan","text":"The test plan consists of four major categories: MMIO path, HE-LB, HE-MEM, HE-HSSI and interrupt tests.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#51-mmio-path","title":"5.1 MMIO Path","text":"The tests under this category exercise MMIO path including all destination functions or blocks as well as PF/VF mux and different fabrics.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#52-he-lb","title":"5.2 HE-LB","text":"The tests under this category target HE-LB function only. Software which test sequences needs to configure HE-LB CSRs before starting it. These CSRs include SRC_ADDR, DST_ADDR, DSM_ADDR, NUM_LINES, CFG etc.
If HE-LB is configured to have memory read transactions, PCIe host memory has to be initialized before HE-LB is started. This is done by svt_pcie_mem_target_service sequence. In other words, PCIe host VIP programs its internal memory model entries in backdoor way. The same process applies to DSM memory entry.
Once HE-LB is started, HE-LB will function based on what it is programmed to do. When HE-LB is done with all necessary memory transactions, it will perform a final memory write to DSM memory entry. Since the software does not know when hardware is done, software polls DSM memory entry periodically until the DSM status bit is asserted.
For loopback mode, data is compared between source buffer and destination buffer in host memory.
RTL statistic counters are also compared against the corresponding variables inside the test sequence at the end of the simulation.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#53-he-mem","title":"5.3 HE-MEM","text":"HE-MEM tests are duplicates from HE-LB with MMIO to CSRs targeting HE-MEM instead of HE-LB.
The DDR simulation model is inside memory controller IP when being generated.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#54-he-hssi","title":"5.4 HE-HSSI","text":"HE-HSSI has indirect registers that are associated with HSSI subsystem, MMIO for indirect registers is different from other functions.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#541-indirect-registers","title":"5.4.1 Indirect Registers","text":"To obtain access to indirect registers, either reading or writing, a MMIO write must be performed.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#542-tx-loopback","title":"5.4.2 TX Loopback","text":"In TX loopback, HE-HSSI initiates ethernet packets to HSSI subsystem and the packets are looped back to HE-HSSI. The loopback is achieved by hard-wiring HSSI TX and RX lanes. This is done inside RTL and for simulation purposes only.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#55-interrupt-test","title":"5.5 Interrupt Test","text":"The test plan covers the basic interrupt flow for FME error, PORT error and user AFU interrupts. The MSI-X table must be programmed in PF0 BAR4. Corresponding PBA bit is expected to be asserted.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#56-performance-test","title":"5.6 Performance Test","text":"Performance tests are derived from HE-LB tests and they are directed tests. At the end of the simulation, performance number is calculated and printed to terminal and a log file.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#57-csr-test","title":"5.7 CSR Test","text":"CSR consists of two parts.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#571-reset-value-check","title":"5.7.1 Reset Value Check","text":"Front-door MMIO read data is compared against RAL register reset value out of reset.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#572-rw-attribute-csr","title":"5.7.2 RW Attribute CSR","text":"MMIO write-read-compare is performed after reset value check for RW attribute CSRs.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#6-checking-mechanism","title":"6 Checking Mechanism","text":"Since there is only PCIe host verification component in testbench, data checking is done by a self-checking mechanism in the test sequence.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#61-protocol-violation","title":"6.1 Protocol Violation","text":"PCIe host VIP has built-in protocol checking on TLP received from FPGA. Abnormal responses are also flagged by the VIP.
Internal AXI Streaming interface has integrated RTL assertion to check AXI Streaming protocol violations.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#62-data-checking","title":"6.2 Data Checking","text":"Data checking is done by self-checking inside a test sequence. MMIO write/read/compare to read-writable CSRs is done inside a sequence.
For memory transactions initiated by functions, backdoor reads from host memory on source buffer and destination buffer is done inside a sequence. Data is compared in case of loopback mode.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#63-counter-checking","title":"6.3 Counter Checking","text":"RTL statistic counters records the number of transactional information that can be read at the end of the simulation and compared against the test expected number.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#64-afu-error-csr","title":"6.4 AFU Error CSR","text":"AFU interface handler provides an error log for illegal transactions that can be read at the end of the simulation.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#7-uvm-set-up","title":"7 UVM set-up","text":"To run the tutorial steps in this guide requires the following development environment:
Item Version Intel Quartus Prime Pro Intel Quartus Prime Pro 23.4 Simulator (VCS) Synopsys VCS S-2021.09-SP1 or newer for UVM simulation of top level FIM Simulator (Questasim) Questasim 2023.4 or newer for UVM simulation of top level FIM"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#71-uvm-prerequisite","title":"7.1 UVM Prerequisite","text":"Retrieve OFS repositories.
The OFS FIM source code is included in the GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories. Cloning the repo using the HTTPS method requires a personal access token. Please see this blog post for information about obtaining a personal access token Token authentication requirements for Git operations.
Navigate to the location for storage of OFS source, create the top-level source directory and clone OFS repositories.
$ mkdir ofs-2024.1-1\n$ cd ofs-2024.1-1\n$ export OFS_BUILD_ROOT=$PWD\n$ git clone --recurse-submodules https://github.com/OFS/ofs-d5005\n\nCloning into 'ofs-d5005' ...\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\n\n$ cd ofs-d5005\n$ git checkout tags/ofs-2024.1-1\n
Verify that the correct tag/branch have been checked out
$ git describe --tags\n\n$ ofs-2024.1-1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#72-license-requirements","title":"7.2 License Requirements","text":"The FIM Testbench is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this TB. UVM RAL (Register Abstraction Layer) is used for CSR Verification.
The Qualified Verification IPs will help to detect incorrect protocol behavior easily, help to focus on BBS features and accelerate the verification process.
- VCS & DVE
- SNPS-Assertions
- Verdi
- VerdiCoverage
- VerdiSimDB
- VerdiTransactionDebugUltra
- VIP-AMBA-AXI-SVT
- VIP-AMBA-STREAM-SVT
- VIP-PCIE-SVT
- VIP-PCIE-TS-SVT
- VIP-PCIE-G3-OPT-SVT
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#73-software-tools-requirements","title":"7.3 Software Tools Requirements","text":"The following tools are required for successful UVM set-up
- Python 3.6.8
- Synopsys PCIE and AMBA AXI UVM VIP Q-2020.03A License
- Synopsys Verdi R-2020.12-SP2 License Note: Makefile can be modified to use DVE instead of Verdi
- VCS R-2020.12-SP2 License
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#74-creating-a-software-tools-script","title":"7.4 Creating a Software Tools Script","text":"The UVM tool set-up is best done by creating a simple set-up script so all applicable tools are sourced before running the tests.
The following environment variables can be pasted into a script and used prior to running the UVM verification environment
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#license-files","title":"License Files","text":"export LM_LICENSE_FILE=\nexport SNPSLMD_LICENSE_FILE=\n
The license environment variables LM_LICENSE_FILE and SNPSLMD_LICENSE_FILE can point to a server license on your system.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#general-environment-variables","title":"General Environment Variables","text":"export OFS_BUILD_ROOT=$PWD\nexport OFS_ROOTDIR=<user_path>/ofs-d5005\nexport WORKDIR=$OFS_ROOTDIR\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#quartus-tools","title":"Quartus Tools","text":"export QUARTUS_HOME=<user_path>/intelFPGA_pro/23.4/quartus\nexport QUARTUS_ROOTDIR=$QUARTUS_HOME\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-verification-tools","title":"Synopsys Verification Tools","text":"export DESIGNWARE_HOME=<user_path>/synopsys/vip_common/vip_Q-2020.03A\nexport PATH=$DESIGNWARE_HOME/bin:$PATH\nexport UVM_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm\nexport VCS_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel\nexport PATH=$VCS_HOME/bin:$PATH\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim-verification-tools","title":"QuestaSIM Verification Tools","text":"export MTI_HOME=<user_path>/mentor/questasim/2023.4/linux64\nexport PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\nexport QUESTA_HOME=$MTI_HOME\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#8-running-a-uvm-simulation-test-and-analysing-results","title":"8 Running a UVM Simulation Test and Analysing Results","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#81-simulation","title":"8.1 Simulation","text":"The default simulator used in this document is Synopsys VCS-MX but there will be references to Questasim. Users can refer to the options and adopt the options for other simulators.
The script is a makefile that calls vlogan, vcs and simv for compilation, elaboration and simulation, respectively
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#82-file-structure","title":"8.2 File Structure","text":"After cloning the repo, the verification and ofs-common directories contain all UVM verification related files. The directory structure is shown in Figure 2 below.
Figure 2 UVM Verification Directory File Structure
ofs-d5005/testbench has a testbench, uvm env, virtual sequencer, RAL etc.
ofs-d5005/verification/tests contains all uvm tests and sequences.
Users can run the simulation under \"ofs-d5005/verification/scripts\" directory and the simulation result is outputted to a \"sim\" directory for Synopsys VCS or \"sim_msim\" for Questasim.
The simulation result folder is named after the test name with increasing suffix number. If user runs the same test multiple times, the suffix is incremented by 1 each time.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#83-uvm-test-suite","title":"8.3 UVM Test Suite","text":"The UVM environment contains a variety of tests that have been developed to test out the FIM portion of OFS.
The table below lists out the \"Test Name\" which will be used on the command line to execute the test, the \"Test Scenario\" and the \"Checking Criteria\".
Tests are located at ofs-d5005/verification/tests
Test Name DUT Scope Test Scenario Checking Criteria dfh_walking_test DFH DHF walking offset checking, eol checking flr_reset_test FLR Reset FLR reset to all PFs Reset checking flr_vf0_reset_test FLR Reset FLR reset to VF0 Reset checking flr_vf1_reset_test FLR Reset FLR reset to VF1 Reset checking flr_vf2_reset_test FLR Reset FLR reset to VF2 Reset checking fme_csr_test FME CSR CSR accesses data checking fme_hemem_intr_test Interrupt FME and HE MEM interrupt Interrupts assertion, PBA bits check fme_intr_test Interrupt FME error interrupt Interrupts assertion, PBA bits check he_hssi_csr_test HE-HSSI CSR accesses for HSSI data checking he_hssi_err_test HE-HSSI Error Cases counter checking he_hssi_rx_lpbk_test HE-HSSI RX loopback data checking he_hssi_tx_lpbk_test HE-HSSI TX loopback counter checking he_lpbk_cont_test HE-LPBK Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_lpbk_long_rst_test HE-MEM Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_lpbk_long_test HE-MEM Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_lpbk_port_rst_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len with port rst data checking he_lpbk_rd_cont_test HE-LPBK Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_rd_test HE-LPBK Read only mode. Randomize num_lines, addresses, req_len counter checking he_lpbk_reqlen1_test HE-LPBK Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_lpbk_reqlen2_test HE-LPBK Loopback mode. 128 CLs, req_len = 2CL, random addresses. data checking, counter checking he_lpbk_reqlen4_test HE-LPBK Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_lpbk_reqlen8_test HE-LPBK Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_lpbk_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_thruput_contmode_test HE-LPBK Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses req_len data checking, counter checking he_lpbk_thruput_test HE-LPBK Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_lpbk_wr_cont_test HE-LPBK Write only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_wr_test HE-LPBK Write only mode. Randomize num_lines, addresses, req_len counter checking he_mem_cont_test HE-MEM Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_mem_lpbk_long_rst_test HE-LPBK Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_mem_lpbk_long_test HE-LPBK Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_mem_lpbk_reqlen1_test HE-MEM Loopback mode. 128 CLs, req_len = 1CL, random addresses. data checking, counter checking he_mem_lpbk_reqlen2_test HE-MEM Loopback mode. 128 CLs, req_len = 2CL, random addresses. data checking, counter checking he_mem_lpbk_reqlen4_test HE-MEM Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_mem_lpbk_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_cont_test HE-MEM Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_test HE-MEM Read only mode. Randomize num_lines, addresses, req_len counter checking he_mem_thruput_contmode_test HE-MEM Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_thruput_test HE-MEM Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_mem_wr_cont_test HE-MEM Write only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_wr_test HE-MEM Write only mode. Randomize num_lines, addresses, req_len counter checking he_random_long_test All HE's Enable all HEs and randomize modes for multiple iterations data checking if in lpbk mode, counter checking he_random_test All HEs Enable all HEs and randomize modes data checking if in lpbk mode, counter checking hehssi_csr_test HE-HSSI CSR accesses for Traffic Control Mail box registers data checking helb_csr_test HE-LPBK CSR accesses data checking helb_rd_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Read only mode data checking, counter checking helb_rd_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Read only mode data checking, counter checking helb_rd_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Read only mode data checking, counter checking helb_thruput_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Thruput mode data checking, counter checking helb_thruput_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Thruput mode data checking, counter checking helb_thruput_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Thruput mode data checking, counter checking helb_wr_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Write only mode data checking, counter checking helb_wr_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Write only mode data checking, counter checking helb_wr_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Write only mode data checking, counter checking hemem_csr_test HE-MEM CSR accesses data checking hemem_intr_test Interrupt HE MEMN Interrupt Interrupts assertion, PBA bits check malformedtlp_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. maxpayloaderror_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. MaxTagError_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3.Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. mini_smoke_test All HEs shorter simpler version of random test for turn-in sanity check data checking if in lpbk mode, counter checking mmio_64b_bar_test PCIe MMIO Path 64-bit bar addess for MMIO data checking mmio_stress_nonblocking_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux with non-blocking MMIO reads data checking mmio_stress_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux data checking mmio_test PCIe MMIO Path MMIO targeting PF0(ST2MM, FME, PMCI, HSSI SS), PF1, PF1.VF1, PF1.VF2 data checking mmio_unimp_test PCIe MMIO Path MMIO acccess to unimplemented addresses MMIO checking MMIODataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. MMIOInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. MMIOTimedout_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. msix_csr_test MSIX CSR CSR accesses data checking pmci_csr_test PMCI CSR CSR accesses data checking port_gasket_csr_test PORT GASKET Port Gasket CSR test port csr checking TxMWrDataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing TxMWrInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing UnexpMMIORspErr_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be retuened on resds. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing The next section describes how to compile and build the UVM environment prior to running each UVM test and analyzing the results in the log files
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#84-ip-compile","title":"8.4 IP Compile","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs","title":"Synopsys VCS","text":"To compile all IPs for the Synopsys VCS simulater:
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk cmplib\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim","title":"Questasim","text":"To compile all IPs for the Questasim simulater:
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk cmplib
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#85-rtl-test-bench-compile","title":"8.5 RTL & Test Bench Compile","text":"The RTL file list for compilation is located here: verification/scripts/rtl_comb.f
The TB file list for compilation is located here: verification/scripts/ver_list.f
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_1","title":"Synopsys VCS","text":"To compile RTL and Testbench for the Synopsys VCS simulater
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk build DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_1","title":"Questasim","text":"To compile RTL and Testbench for the Questasim simulater
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk build DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#86-ip-and-rtl-test-bench-compile","title":"8.6 IP and RTL & Test Bench Compile","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_2","title":"Synopsys VCS","text":"If the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS then follow the procedure below
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk build_all DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_2","title":"Questasim","text":"If the user wants to compile all IPs and RTL Testbench in one command for Questasim then follow the procedure below
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk build_all DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_3","title":"Synopsys VCS","text":"To run a simulation for Synopsys VCS:
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_3","title":"Questasim","text":"To run a simulation for Questasim:
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk run TESTNAME=mmio_test DUMP=1
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_4","title":"Synopsys VCS","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation.
ofs-d5005/verification/scripts gmake -f Makefile_VCS.mk build DUMP=1\nofs-d5005/verification/scripts gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> DUMP=1\n
Or ofs-d5005/verification/scripts gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_4","title":"Questasim","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Questasim build and simulation.
ofs-d5005/verification/scripts gmake -f Makefile_MSIM.mk build DUMP=1\nofs-d5005/verification/scripts gmake -f Makefile_MSIM.mk run TESTNAME=<test_case_name> DUMP=1\n
Or ofs-d5005/verification/scripts gmake -f Makefile_MSIM.mk build_run TESTNAME=<test_case_name> DUMP=1\n
There are some optimizations in the Table below for convenience if you want to bypass some commands for both Synopsys VCS and Questasim:
Command (Synopsys VCS) Command (Questasim) Details gmake -f Makefile_VCS.mk build_all DUMP=1 gmake -f Makefile_MSIM.mk build_all DUMP=1 compile IP + compile RTL gmake -f Makefile_VCS.mk build_run TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk build_run TESTNAME= DUMP=1 compile RTL + run test gmake -f Makefile_VCS.mk do_it_all TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk do_it_all TESTNAME= DUMP=1 compile IP, RTL and run test gmake -f Makefile_VCS.mk rundb TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk rundb TESTNAME= DUMP=1 run test in sim dir + over-writes content"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#87-uvm-regression-test","title":"8.7 UVM Regression Test","text":"cd $VERDIR/scripts\n\nFor Regression in VCS with top/test package, execute the following command python uvm_regress.py -l -n 8 -s vcs -c\n\nResults are created in a sim directory ($VERDIR/sim) with individual testcase log dir\n\nFor Regression in MSIM with top/test package, execute the following command python uvm_regress.py -l -n 8 -s msim -c\n
Results are created in a sim directory ($VERDIR/sim_msim) with individual testcase log dir
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#88-uvm-waveform-and-transcript-analysis","title":"8.8 UVM Waveform and Transcript Analysis","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_5","title":"Synopsys VCS","text":"Running Synopsys VCS UVM tests will generate a ofs-d5005/verification/sim directory
- All build time logs are at ofs-d5005/verification/sim
- Each testcase will have separate directory inside sim ofs-d5005/verification/sim/
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Synopsys VCS. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 3
Figure 3 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 4
Figure 4 trans.log
The waveform generated is named as \"inter.vpd\". To open the waveform, go to simulation result directory and run
dve -full64 -vpd inter.vpd &\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_5","title":"Questasim","text":"Running Questasim UVM tests will generate a ofs-d5005/verification/sim_msim directory
- All build time logs are at ofs-d5005/verification/sim_msim
- Each testcase will have separate directory inside sim ofs-d5005/verification/sim_msim/
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Questasim. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 4
Figure 4 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 5
Figure 5 trans.log
The waveform generated is named as \"vsim.wlf\". To open the waveform, go to simulation result directory and run
vsim -view vsim.wlf &
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#9-modifying-uvm-testbench","title":"9 Modifying UVM Testbench","text":"The next section describe what needs to be considered when modifying the UVM, targeting a different device, adding a new interface to the testbench and creating a new UVM test for a customized OFS Accelerator platform.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#91-modifying-uvm-environment-when-targeting-different-device","title":"9.1 Modifying UVM environment when targeting different device","text":"A new device may have different design feature or flow. The base address must be allocated for the new device. The MMIO targeting the new device must be based on the base address. If it is a new PF or VF, PCIe HIP must be regenerated and enumeration sequence must be updated accordingly.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#92-modifying-uvm-environment-when-adding-a-new-interface","title":"9.2 Modifying UVM environment when adding a new interface","text":"Adding a new interface requires signal connections in the testbench. An additional BFM or verification IP is needed to drive the new interface. The main testbench file tb_top.sv is found at the following location
$OFS_ROOTDIR/verification/testbench\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#93-adding-a-new-uvm-test","title":"9.3 Adding a new UVM test","text":"In the following example we will modify an existing test \"he_lpbk\" and name it \"he_lpbk_new\", and rebuild the test to check it. Please follow the steps below
-
Create a new test sequence file under ofs-d5005/verification/tests/sequences
he_lpbk_seq_new.svh\n
-
Modify ifndef, define and endif statements in new test sequence case i.e he_lpbk_seq_new.svh file
`ifndef HE_LPBK_SEQ_NEW_SVH `define HE_LPBK_SEQ_NEW_SVH\n`endif // HE_LPBK_SEQ_NEW_SVH\n
also replace all occurences of he_lpbk_seq with he_lpbk_seq_new in the he_lpbk_seq_new.svh file
-
Append the new sequence name into ofs-d5005/verification/tests/sequences/seq_lib.svh file
`include \"he_lpbk_seq_new.svh\"\n
-
Create a new test under ofs-d5005/verification/tests
he_lpbk_test_new.svh\n
-
Modify ifndef, define and endif statements in new test case i.e he_lpbk_test_new.svh file
`ifndef HE_LPBK_TEST_NEW_SVH `define HE_LPBK_TEST_NEW_SVH\n`endif // HE_LPBK_TEST_NEW_SVH\n
also replace all occurences of he_lpbk_test with he_lpbk_test_new in the he_lpbk_test_new.svh file
-
Append the new test name into ofs-d5005/verification/tests/test_pkg.svh file
`include \"he_lpbk_test_new.svh\"\n
-
Rebuild UVM test suite for either Synopsys VCS or Questasim simulater
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk build_all\n
or
cd $VERDIR/scripts\ngmake -f Makefile_MSIM.mk build_all\n
-
Execute new test for either Synopsys VCS or Questasim simulater
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk run TESTNAME=he_lpbk_test_new\n
or
cd $VERDIR/scripts\ngmake -f Makefile_MSIM.mk run TESTNAME=he_lpbk_test_new\n
9) Check new test and log files cd ofs-d5005/verification/sim/he_lpbk_test_new
```sh\nopen runsim.log\n```\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/doc_modules/Glossary/","title":"Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/doc_modules/Notices_%26_Disclaimers/","title":"Notices & Disclaimers","text":""},{"location":"hw/doc_modules/Notices_%26_Disclaimers/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/doc_modules/afu_flr_disable/","title":"Afu flr disable","text":"The vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\n
Enable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\n
If you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\n
Sample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\n
"},{"location":"hw/doc_modules/agx7_pcie_attach_build_command_gen_tool/","title":"Agx7 pcie attach build command gen tool","text":"OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command."},{"location":"hw/doc_modules/agx7_pcie_attach_sec_base_ofss_file/","title":"Agx7 pcie attach sec base ofss file","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_build_work_directory/","title":"Agx7 pcie attach sec build work directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_fim_build_script/","title":"Agx7 pcie attach sec fim build script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_fim_simulation/","title":"Agx7 pcie attach sec fim simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_hssi_ofss_file/","title":"Agx7 pcie attach sec hssi ofss file","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_iopll_ofss_file/","title":"Agx7 pcie attach sec iopll ofss file","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_memory_ofss_file/","title":"Agx7 pcie attach sec memory ofss file","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_null_host_exercisers/","title":"Agx7 pcie attach sec null host exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_ofss_file_usage/","title":"Agx7 pcie attach sec ofss file usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_pcie_ofss_file/","title":"Agx7 pcie attach sec pcie ofss file","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_top_level_ofss_file/","title":"Agx7 pcie attach sec top level ofss file","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/doc_modules/contents_agx7_pcie_attach/","title":"Documentation Overview for Agilex 7 PCIe Attach Reference Designs","text":"Use this page as a guide to understand how to navigate the documentation depending on your role:
Document Description Board Developer Shell Developer Workload Developer Software Developer Board Installation Guide: OFS for Acceleration Development Platforms Describes how to install the N6000/N60001 platform in a server with required cabling and configure BIOS X X Board Installation Guide: OFS for Agilex\u00ae 7 PCIe Attach Development Kits Describes how to install evaluation kits in a server with required cabling and configure BIOS X X Software Installation Guide: OFS for PCIe Attach FPGAs Provides steps for installing Linux kernel packages and OPAE software development kit package X X X Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) Provides steps for loading a pre-packaged PCIe attach shell binary unto the I-Series Development Kit and using the OPAE user space tools X X Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) Provides steps for loading a pre-packaged PCIe attach shell binary unto the F-Series Development Kit and using the OPAE user space tools X X Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) Provides steps for loading a pre-packaged PCIe attach shell binary unto the Intel FPGA SmartNIC N6000/1-PL platform and using the OPAE user space tools X X Automated Evaluation User Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs Describes how to use provided automated script to step through setup, build flows, simulation and OneAPI tasks. Script can be leveraged for your own custom builds. X X PCIe Attach Shell Technical Reference Manual Describes features and functions of the Agilex 7 PCIe attach reference design for the Intel FPGA SmartNIC N6000-PL Platform X Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs Provides explanation of the shell reference designs targeting the I-Series Development Kit and steps for building and modifying the design X Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs Provides explanation of the shell reference design targeting the F-tile Development Kit and steps for building and modifying the design X Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs Provides explanation of the shell reference design targeting the Intel FPGA SmartNIC N6000-PL platform and steps for building and modifying the design X Hard Processor System Software Developer Guide: OFS for Agilex\u00ae FPGAs Provides steps for building or using pre-compiled ITB images for exploration and evaluation of the HPS bring-up flow on the Intel FPGA SmartNIC N6000/1-PL platforms. X X UVM Simulation User Guide: OFS for Agilex\u00ae 7 PCIe Attach Describes how to use the provided UVM environment files and test suite to simulate the default reference designs X Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs Describes how to build, test and debug an example workload for a PCIe attach shell. X PIM Based AFU Developer Guide Describes how to create a an AFU/workload using the Platform Interface Manager (PIM) shims X AFU Host Software Developer Guide Provides instructions on managing data transfer between the host and the AFU X X AFU Simulation Environment User Guide Provides steps for using the AFU Simulation Environment for hardware/software co-simulation of your workload X oneAPI Accelerator Support Package (ASP): Getting Started User Guide Provides steps to use and test with the default OneAPI Accelerator Support Package X oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack Provides details about the hardware and software components required for enabling OneAPI in OFS designs. X Software Reference Manual: Open FPGA Stack Provides details of OPAE SDK user-space software stack and the kernel-space linux-dfl drivers X KVM User Guide: Open FPGA Stack Describes how to setup and configure a virtual machine in an OFS-enabled design X X Docker User Guide: Open FPGA Stack Provides steps for implementing a docker container to evaluate and develop with OFS X X X Document Description Board Developer Shell Developer Workload Developer Software Developer"},{"location":"hw/doc_modules/contents_agx7_soc_attach/","title":"Documentation Overview for Agilex 7 SoC Attach Reference Designs","text":"Use this page as a guide to understand how to navigate the documentation depending on your role:
Document Description Board Developer Shell Developer Workload Developer Software Developer Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL Describes how to install the Intel FPGA IPU F2000X-PL Platform in a server with required cabling and configure BIOS X X Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Provides steps for installing Linux kernel packages and OPAE software development kit package X X X Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Provides steps for loading a pre-packaged SoC attach shell binary unto the Intel FPGA IPU F2000x-PL and using the OPAE user space tools X X Automated Evaluation User Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Describes how to use provided automated script to step through setup, build flows, simulation and OneAPI tasks. Script can be leveraged for your own custom builds. X X Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs Describes features and functions of the provided Agilex 7 SoC Attach Reference design X Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Provides explanation of the shell reference design targeting the Intel FPGA IPU F2000X-PL Platform and steps for building and modifying the design X UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach Describes how to use the provided UVM environment files and test suite to simulate the default reference designs X Workload Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Describes how to build, test and debug an example workload for a SoC attach shell. X PIM Based AFU Developer Guide Describes how to create a an AFU/workload using the Platform Interface Manager (PIM) shims X AFU Simulation Environment User Guide Provides steps for using the AFU Simulation Environment for hardware/software co-simulation of your workload X Software Reference Manual: Open FPGA Stack Provides details of OPAE SDK user-space software stack and the kernel-space linux-dfl drivers X KVM User Guide: Open FPGA Stack Describes how to setup and configure a virtual machine in an OFS-enabled design X X Docker User Guide: Open FPGA Stack Provides steps for implementing a docker container to evaluate and develop with OFS X X X Document Description Board Developer Shell Developer Workload Developer Software Developer"},{"location":"hw/doc_modules/contents_s10_pcie_attach/","title":"Documentation Overview for Stratix 10 PCIe Attach Reference Designs","text":"Use this page as a guide to understand how to navigate the documentation depending on your role:
Document Description Board Developer Shell Developer Workload Developer Software Developer Board Installation Guide: OFS for Acceleration Development Platforms Describes how to install the Intel FPGA PAC D5005 Platform in a server with required cabling and configure BIOS X X Software Installation Guide: OFS for PCIe Attach FPGAs Provides steps for installing Linux kernel packages and OPAE software development kit package X X X Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs Provides steps for loading a pre-packaged shell binary unto the Intel FPGA PAC D5005 platform and using the OPAE user space tools X X Automated Evaluation User Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs Describes how to use provided automated script to step through setup, build flows, simulation and OneAPI tasks. Script can be leveraged for your own custom builds. X X Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs Provides explanation of the SoC attach shell reference design targeting the Intel FPGA PAC D5005 Platform and steps for building and modifying the design X UVM Simulation User Guide: OFS for Stratix\u00ae 10 PCIe Attach Describes how to use the provided UVM environment files and test suite to simulate the default reference designs X Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs Describes how to build, test and debug an example workload for a PCIe attach shell. X PIM Based AFU Developer Guide Describes how to create a an AFU/workload using the Platform Interface Manager (PIM) shims X AFU Simulation Environment User Guide Provides steps for using the AFU Simulation Environment for hardware/software co-simulation of your workload X oneAPI Accelerator Support Package (ASP): Getting Started User Guide Provides steps to use and test with the default OneAPI Accelerator Support Package X oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack Provides details about the hardware and software components required for enabling OneAPI in OFS designs. X Software Reference Manual: Open FPGA Stack Provides details of OPAE SDK user-space software stack and the kernel-space linux-dfl drivers X KVM User Guide: Open FPGA Stack Describes how to setup and configure a virtual machine in an OFS-enabled design X X Docker User Guide: Open FPGA Stack Provides steps for implementing a docker container to evaluate and develop with OFS X X X Document Description Board Developer Shell Developer Workload Developer Software Developer"},{"location":"hw/doc_modules/links/","title":"OFS Collateral","text":""},{"location":"hw/doc_modules/links/#top-level-collateral-contents-pages","title":"Top Level Collateral Contents Pages","text":""},{"location":"hw/doc_modules/links/#automated-evaluation-guides","title":"Automated Evaluation Guides","text":""},{"location":"hw/doc_modules/links/#board-installation-guides","title":"Board Installation Guides","text":""},{"location":"hw/doc_modules/links/#software-installation-guides","title":"Software Installation Guides","text":""},{"location":"hw/doc_modules/links/#getting-started-guides","title":"Getting Started Guides","text":""},{"location":"hw/doc_modules/links/#shell-technical-reference-manuals","title":"Shell Technical Reference Manuals","text":""},{"location":"hw/doc_modules/links/#shell-developer-guides","title":"Shell Developer Guides","text":""},{"location":"hw/doc_modules/links/#workload-developer-guides","title":"Workload Developer Guides","text":""},{"location":"hw/doc_modules/links/#oneapi","title":"oneAPI","text":""},{"location":"hw/doc_modules/links/#uvm-simulation-user-guides","title":"UVM Simulation User Guides","text":""},{"location":"hw/doc_modules/links/#common","title":"Common","text":""},{"location":"hw/doc_modules/links/#ofs-repositories","title":"OFS Repositories","text":""},{"location":"hw/doc_modules/links/#oneapi-sites","title":"oneAPI Sites","text":""},{"location":"hw/doc_modules/links/#software","title":"Software","text":""},{"location":"hw/doc_modules/links/#afu-dev","title":"AFU Dev","text":""},{"location":"hw/doc_modules/links/#misc","title":"Misc","text":""},{"location":"hw/doc_modules/quartus_installation/","title":"Quartus installation","text":"Intel ${{ env.QUARTUS_PRO_VER_L }} is verified to work with the latest OFS release ${{ env.OFS_FIM_RELEASE }}. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use ${{ env.HOST_OS_L }} for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are ${{ env.QUARTUS_PATCHES }}.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }} then the new line is:
export PATH=/home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }}/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }}/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }}/quartus/bin/quartus\n
"},{"location":"hw/doc_modules/wt_clone_fim_repo/","title":"Wt clone fim repo","text":"Perform the following steps to clone the OFS ${{ env.DESIGN }} FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules ${{ env.OFS_FIM_REPO_URL }}.git\n
-
Check out the correct tag of the repository
cd ${{ env.OFS_FIM_REPO }}\ngit checkout --recurse-submodules tags/${{ env.OFS_FIM_TAG }}\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (${{ env.OFS_FIM_TAG }})\n
"},{"location":"hw/doc_modules/wt_compile_fim_in_prep_for_afu/","title":"Wt compile fim in prep for afu","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the [Set Up Development Environment] Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the [Clone FIM Repository] section for step-by-step instructions.
-
Set development environment variables. Refer to the [Set Development Environment Variables] section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/${{ env.MODEL }}.ofss ${{ env.MODEL }}:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_${{ env.MODEL }}\n
"},{"location":"hw/doc_modules/wt_install_git_lfs_rhel/","title":"Wt install git lfs rhel","text":"To install the Git Large File Storage (LFS) extension, execute the following commands:
- Obtain Git LFS package
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh\n
- Install Git LFS package
sudo dnf install git-lfs\n
- Install Git LFS
git lfs install\n
"},{"location":"hw/doc_modules/wt_run_individual_unit_level_sim/","title":"Wt run individual unit level sim","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the [Walkthrough: Set Up Development Environment] Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the [Walkthrough: Clone FIM Repository] section for step-by-step instructions.
-
Set development environment variables. Refer to the [Walkthrough: Set Development Environment Variables] section for step-by-step instructions.
-
Generate the simulation files for the ${{ env.MODEL }}
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\n./gen_sim_files.sh --ofss=$OFS_ROOTDIR/tools/ofss_config/${{ env.MODEL }}.ofss ${{ env.MODEL }}\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/${{ env.OFS_FIM_REPO }}/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/doc_modules/wt_set_fim_dev_env_vars/","title":"Wt set fim dev env vars","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ${{ env.OFS_FIM_REPO }}\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=${{ env.OPAE_BRANCH }}\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/doc_modules/wt_set_up_development_environment/","title":"Wt set up development environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that ${{ env.QUARTUS_PRO_VER_L }} for Linux with Intel Agilex FPGA device support is installed on your development machine. Refer to the [Walkthrough: Install Quartus Prime Pro Software] section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion ${{ env.QUARTUS_PRO_VER_S }} SC Pro Edition\nCopyright (C) 2024 Intel Corporation. All rights reserved.\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python ${{ env.PYTHON_VER }} or later
-
Verify version number
python --version\n
Example Output:
Python ${{ env.PYTHON_VER }}\n
-
GCC ${{ env.GCC_VER }} or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) ${{ env.GCC_VER }}\n
-
cmake ${{ env.CMAKE_VER }} or later
-
Verify version number
cmake --version\n
Example output:
cmake version ${{ env.CMAKE_VER }}\n
-
git ${{ env.GIT_VER }} or later.
-
Verify version number
git --version\n
Example output:
git version ${{ env.GIT_VER }}\n
-
Clone the ${{ env.OFS_FIM_REPO }} repository. Refer to the [Walkthrough: Clone FIM Repository] section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches ${{ env.QUARTUS_PATCHES }}. All required patches are provided in the Assets of the OFS FIM Release: ${{ env.OFS_FIM_RELEASE_PAGE_URL }}
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the [Walkthrough: Set Environment Variables] section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/f2000x/","title":"Index","text":"Location for SoC Attach Collateral for OFS.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/","title":"AFU Developer Guide: OFS for Agilex\u00ae 7 FPGA SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#1-introduction","title":"1. Introduction","text":"This document is a design guide for the creation of an Accelerator Functional Unit (AFU) using Open FPGA Stack (OFS) for Agilex\u00ae 7 FPGAs SoC Attach. The AFU concept consists of separating out the FPGA design development process into two parts, the construction of the foundational FPGA Interface Manager (FIM), and the development of the Acceleration Function Unit (AFU), as shown in the diagram below.
This diagram shows the separation of FPGA board interface development from the internal FPGA workload creation. This separation starts with the FPGA Interface Manager (FIM) which consists of the external interfaces and board management functions. The FIM is the base system layer and is typically provided by board vendors. The FIM interface is specific to a particular physical platform. The AFU makes use of the external interfaces with user defined logic to perform a specific application. By separating out the lengthy and complicated process of developing and integrating external interfaces for an FPGA into a board allows the AFU developer to focus on the needs of their workload. OFS for Agilex\u00ae 7 FPGAs SoC Attach provides the following tools for rapid AFU development:
- Scripts for both compilation and simulation setup
- Optional Platform Interface Manager (PIM) which is a set of SystemVerilog shims and scripts for flexible FIM to AFU interfacing
- Acceleration Simulation Environment (ASE) which is a hardware/software co-simulation environment scripts for compilation and Acceleration
- Integration with Open Programmable Acceleration Engine (OPAE) SDK for rapid software development for your AFU application
Please notice in the above block diagram that the AFU region consists of static and partial reconfiguration (PR) regions where the PR region can be dynamically reconfigured while the remaining FPGA design continues to function. Creating AFU logic for the static region is described in Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs. This guide covers logic in the AFU Main region.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#11-document-organization","title":"1.1. Document Organization","text":"This document is organized as follows:
- Description of design flow
- Interfaces and functionality provided in the Agilex\u00ae 7 FPGAs SoC Attach FIM
- Setup of the AFU Development environment
- Synthesize the AFU example
- Testing the AFU example on the IPU Platform F2000X-PL card
- Hardware/Software co-simulation using ASE
- Debugging an AFU with Remote Signal Tap
This guide provides theory followed by tutorial steps to solidify your AFU development knowledge.
NOTE:
This guide uses the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL as the platform for all tutorial steps. Additionally, this guide and the tutorial steps can be used with other Agilex\u00ae 7 FPGAs SoC Attach platforms..
Some of the document links in this guide are specific to the IPU Platform F2000X-PL . If using a different platform, please use the associated documentation for your platform as there could be differences in building the FIM and downloading FIM images.
If you have worked with previous Altera\u00ae Programmable Acceleration products, you will find out that OFS for Agilex\u00ae 7 FPGAs SoC Attach is similar. However, there are differences and you are advised to carefully read and follow the tutorial steps to fully understand the design tools and flow.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#12-prerequisite","title":"1.2. Prerequisite","text":"This guide assumes you have the following FPGA logic design-related knowledge and skills:
- FPGA compilation flows including the Quartus\u00ae Prime Pro Edition design flow
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition software, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL and coding practices to create synthesizable logic.
- Understanding of AXI and Avalon memory mapped and streaming interfaces.
- Simulation of complex RTL using industry standard simulators (Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae).
- Signal Tap Logic Analyzer tool in the Quartus\u00ae Prime Pro Edition software.
You are strongly encouraged to review the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#13-acceleration-functional-unit-afu-development-flow","title":"1.3. Acceleration Functional Unit (AFU) Development Flow","text":"The AFU development flow is shown below:
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#131-understanding-platform-capabilities","title":"1.3.1. Understanding Platform Capabilities","text":"The block diagram of the F2000x Board is shown below:
The F2000x FIM provided with this release is shown below:
This release F2000x FIM provides the following features:
- Host interface
- PCIe Gen4 x 16
- 2 - PFs
- MSI-X interrupts
- Logic to demonstrate simple PCIe loopback
- Network interface
- 2 - QSFP28/56 cages
- 8 X 25 GbE with exerciser logic demonstrating traffic generation/monitoring
- External Memory - DDR4 - 2400
- 4 Banks - 4 GB organized as 1 Gb x 32 with 1 Gb x 8 ECC
- Memory exerciser logic demonstrating external memory operation
- Board Management
- SPI interface
- FPGA management and configuration
- Example logic showing DFH operation
- Partial reconfiguration control logic
- SoC - Xeon Icelake-D subsystem with embedded Linux
- PCIe Gen4 x 16 interface to FPGA
- 1 - PF, 3 - VF, AXI-S TLP packets
- DDR Memory
- 2 Banks - 8 GB organized as 1 Gb x 64 with 1 Gb x 8 ECC
- NVMe SSD - 64 GB
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#132-high-level-data-flow","title":"1.3.2. High Level Data Flow","text":"The OFS high level data flow is shown below:
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#133-considerations-for-pim-usage","title":"1.3.3. Considerations for PIM Usage","text":"When creating an AFU, a designer needs to decide of what type of interfaces the AFU will provide to the platform (FIM). The AFU can use the native interfaces (i.e. PCIe TLP commands) provided by the FIM or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
The following resources are available to assist in creating an AFU:
PIM Core Concepts provides details on using the PIM and its capabilities.
PIM Based AFU Developer Guide provides details on interfacing your AFU to the FIM using the PIM.
The examples-afu repo provides several AFU examples:
Example Description PIM-based Hybrid Native clocks Example AFU using user configurable clocks. X copy_engine Example AFU moving data between host channel and a data engine. X dma Example AFU moving data between host channel and local memory with a DMA. X hello_world Example AFU sending \"Hello World!\" to host channel. X X X local_memory Example AFU reading and writing local memory. X X These examples can be run with the current OFS FIM package. There are three AFU types of examples provided (PIM based, hybrid and native). Each example provides the following:
- RTL, which includes the following interfaces:
- Host Channel:
- Host memory, providing a DMA interface.
- MMIO, providing a CSR interface.
- Local Memory
- Host software example interfacing to the CSR interface and host memory interface, using the OPAE C API.
- Accelerator Description File .json file
- Source file list
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#134-afu-interfaces-included-with-ipu-platform-f2000x-pl","title":"1.3.4. AFU Interfaces Included with IPU Platform F2000X-PL","text":"The figure below shows the interfaces available to the AFU in this architecture. It also shows the design hierarchy with module names from the fim (top.sv) to the PR region AFU (afu_main.sv). One of the main differences from the Stratix 10 PAC OFS architecture to this one is the presence of the static port gasket region (port_gasket.sv) that has components to facilitate the AFU and also consists of the PR region (afu_main.sv) via the PR slot. The Port Gasket contains all the PR specific modules and logic, e.g., PR slot reset/freeze control, user clock, remote STP etc. Architecturally, a Port Gasket can have multiple PR slots where user workload can be programmed into. However, only one PR slot is supported for OFS Release for Agilex. Everything in the Port Gasket until the PR slot should be provided by the FIM developer. The task of the AFU developer is to add their desired application in the afu_main.sv module by stripping out unwanted logic and instantiating the target accelerator. As shown in the figure below, here are the interfaces connected to the AFU (highlighted in green) via the SoC Attach FIM:
- AXI Streaming (AXI-S) interface to the Host via PCIe Gen4x16
- AXI Memory Mapped Channels (4) to the DDR4 EMIF interface
- AXI Streaming (AXI-S) interface to the HSSI 25 Gb Ethernet
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#2-set-up-afu-development-environment","title":"2. Set Up AFU Development Environment","text":"This section covers the setup of the AFU development environment.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#21-prepare-afu-development-environment","title":"2.1. Prepare AFU development environment","text":"A typical development and hardware test environment consists of a development server or workstation with FPGA development tools installed and a separate server with the target OFS compatible FPGA PCIe card installed. The typical usage and flow of data between these two servers is shown below:
Note: both development and hardware testing can be performed on the same server if desired.
This guide uses IPU Platform F2000X-PL as the target OFS compatible FPGA PCIe card for demonstration steps. The IPU Platform F2000X-PL must be fully installed following the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL. If using a different OFS FPGA PCIe card, contact your supplier for instructions on how to install and operate user developed AFUs.
The following is a summary of the steps to set up for AFU development:
- Install Quartus Prime Pro Version 23.4 for Linux with Agilex device support and required Quartus patches.
- Make sure support tools are installed and meet version requirements.
- Install OPAE SDK.
- Download the Basic Building Blocks repository.
- Build or download the relocatable AFU PR-able build tree based on your Agilex FPGA PCIe Attach FIM.
- Download FIM to the Agilex FPGA PCIe Attach platform.
Building AFUs with OFS for Agilex requires the build machine to have at least 64 GB of RAM.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#22-installation-of-quartus-and-required-patches","title":"2.2. Installation of Quartus and required patches","text":"Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use Ubuntu 22.04 LTS for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.17.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#23-installation-of-support-tools","title":"2.3. Installation of Support Tools","text":"Make sure support tools are installed and meet version requirements.
The OFS provided Quartus build scripts require the following tools. Verify these are installed in your development environment.
Item Version Python 3.6.8 GCC 8.5.0 cmake 3.15 git 1.8.3.1 perl 5.8.8"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#24-installation-of-opae-sdk","title":"2.4. Installation of OPAE SDK","text":"Working with the IPU Platform F2000X-PL card requires opae-2.12.0-5. Follow the instructions in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs to build and install the required OPAE SDK for the IPU Platform F2000X-PL. Make sure to check out the cloned repository to tag 2.12.0-5 and branch release/2.12.0.
$ git checkout tags/2.12.0-5 -b release/2.12.0\n
Note: The tutorial steps provided in the next sections assume the OPAE SDK is installed in default system locations, under the directory, /usr. In most system configurations, this will allow the OS and tools to automatically locate the OPAE binaries, scripts, libraries and include files required for the compilation and simulation of the FIM and AFUs.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#25-download-the-basic-building-blocks-repositories","title":"2.5. Download the Basic Building Blocks repositories","text":"The ofs-platform-afu-bbb repository contains the PIM files as well as example PIM-based AFUs that can be used for testing and demonstration purposes. This guide will use the host_chan_mmio AFU example in the ofs-platform-afu-bbb repository and the hello_world sample in the examples-afu repository to demonstrate how to synthesize, load, simulate, and test a PIM-based AFU using the IPU Platform F2000X-PL card with the SoC Attach FIM.
Execute the next commands to clone the BBB repositories.
# Create top level directory for AFU development\n$ mkdir OFS_BUILD_ROOT\n$ cd OFS_BUILD_ROOT\n$ export OFS_BUILD_ROOT=$PWD\n# Clone the ofs-platform-afu-bbb repository.\n$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/ofs-platform-afu-bbb.git\n$ cd ofs-platform-afu-bbb\n$ git checkout tags/ofs-2024.1-1\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n# Verify retrieval\n$ ls\nLICENSE plat_if_develop plat_if_release plat_if_tests README.md\n
The documentation in the ofs-platform-afu-bbb repository further addresses - The PIM concept. - The structure of the PIM-based AFU examples. - How to generate a release and configure the PIM. - How to connect an AFU to an FIM.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#26-build-or-download-the-relocatable-pr-build-tree","title":"2.6. Build or download the relocatable PR build tree","text":"A relocatable PR build tree is needed to build the AFU partial reconfiguration area for the intended FIM. The tree is relocatable and may be copied to a new location. It does not depend on files in the original FIM build.
You can use the IPU Platform F2000X-PL release package and download the PR build tree and FIM images, to develop your AFU. These are located at OFS-F2000X-PL release
Or you can build your own FIM and generate the PR build tree during the process.
To download and untar the pr_build_template:
$ cd $OFS_BUILD_ROOT\n$ wget https://github.com/OFS/ofs-f2000x-pl/releases/download/ofs-2024.1-1/f2000x-images_ofs-2024-2-1.tar.gz\n$ tar -zxvf f2000x-images_ofs-2024-2-1.tar.gz\n$ cd f2000x-images_ofs-2024-2-1/\n$ mkdir pr_build_template\n$ tar -zxvf pr_build_template.tar.gz -C ./pr_build_template\n$ cd pr_build_template\n$ export OPAE_PLATFORM_ROOT=$PWD\n
To build your own FIM and generate the PR build tree for the IPU Platform F2000X-PL platform, refer the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs and follow the Out-of-Tree PR FIM build flow. If you are using a different platform, refer to the Shell Developer Guide for your platform and follow the Out-of-Tree PR FIM build flow.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#27-download-fim-to-fpga","title":"2.7. Download FIM to FPGA","text":"The AFU requires that the FIM from which the AFU is derived be loaded onto the FPGA.
If you are using the IPU Platform F2000X-PL release package downloaded in the previous section, copy the associated FIM files to the SoC:
# On Development Host\n$ cd $OFS_BUILD_ROOT/f2000x-images_ofs-${{ env.COMMON_OFS_RELEASE_TAR }}/\n$ scp ofs_top_page1_unsigned_user1.bin <user>@<SoC IP address>:</remote/directory>\n$ scp ofs_top_page2_unsigned_user2.bin <user>@<SoC IP address>:</remote/directory>\n
If you are generating your own FIM, use the unsigned FPGA binary images from your FIM build and copy over to the SoC.
To downlaod the FIM to the IPU Platform F2000X-PL platform:
# On SoC\n$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin $ sudo fpgasupdate ofs_top_page2_unsigned_user2.bin $ sudo rsu fpga --page=user1
If you are using a different platform, refer to the documentation for your platform to download the FIM images onto your Agilex\u00ae SoC Attach Platform.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#28-set-up-required-environment-variables","title":"2.8. Set up required Environment Variables","text":"Set the required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks. You can create a simple script to set these variables and save time going forward.
# If not already done, export OFS_BUILD_ROOT to the top level directory for AFU development\n$ export OFS_BUILD_ROOT=<path to ofs build directory>\n\n# If not already done, export OPAE_PLATFORM_ROOT to the PR build tree directory\n$ export OPAE_PLATFORM_ROOT=<path to pr build tree>\n\n# If not already done, export OFS_PLATFORM_AFU_BBB to the clone of ofs-platform-afu-bbb repository which contains PIM files and AFU examples.\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu-bbb> # Quartus Tools\n# Note, QUARTUS_HOME is your Quartus installation directory, e.g. $QUARTUS_HOME/bin contains Quartus executable.\n$ export QUARTUS_HOME=<user_path>/intelFPGA_pro/23.4/quartus\n$ export QUARTUS_ROOTDIR=$QUARTUS_HOME\n$ export QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\n$ export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n$ export IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys\n$ export PATH=$QUARTUS_HOME/bin:$QSYS_ROOTDIR/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\n# OPAE SDK release\n$ export OPAE_SDK_REPO_BRANCH=release/2.12.0\n\n# OPAE and MPF libraries must either be on the default linker search paths or on both LIBRARY_PATH and LD_LIBRARY_PATH. \n$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#3-compiling-an-afu","title":"3. Compiling an AFU","text":"In this section, you will use the relocatable PR build tree created in the previous steps from the FIM to compile an example PIM-based AFU. This section will be developed around the host_chan_mmio and hello_world AFU examples to showcase the synthesis of a PIM-based AFU.
The build steps presented below demonstrate the ease in building and running an actual AFU on the board. To successfully execute the instructions in this section, you must have set up your development environment and have a relocateable PR Build tree as instructed in section 2 of this document.
The build steps presented below demonstrate the ease in building and running an actual AFU on the board. To successfully execute the instructions in this section, you must have set up your development environment and have a relocateable PR Build tree as instructed in section 2 of this document.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#31-creating-the-afu-synthesis-environment","title":"3.1. Creating the AFU Synthesis Environment","text":"The PIM flow provides the script afu_synth_setup to create the synthesis environment to build the AFU examples. See how to use it below.
usage: afu_synth_setup [-h] -s SOURCES [-p PLATFORM] [-l LIB] [-f] dst\n\nGenerate a Quartus build environment for an AFU. A build environment is\ninstantiated from a release and then configured for the specified AFU. AFU\nsource files are specified in a text file that is parsed by rtl_src_config,\nwhich is part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA platform name.\n -l LIB, --lib LIB FPGA platform release hw/lib directory. If not\n specified, the environment variables OPAE_FPGA_HW_LIB\n and then BBS_LIB_PATH are checked.\n -f, --force Overwrite target directory if it exists.\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#32-building-and-running-host_chan_mmio-example-afu","title":"3.2. Building and Running host_chan_mmio example AFU","text":"The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using Avalon and AXI interfaces. However, this guide will use the AXI version of the host_chan_mmio AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the actual AFU hardware.
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\n
Execute afu_synth_setup as follows to create the synthesis environment for a host_chan_mmio AFU that fits the SoC Attach FIM previously constructed.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_dev\n
Now, move into the synthesis environment afu_dev directory just created. From there, execute the afu_synth command. The successful completion of the command will produce the host_chan_mmio.gbs file under the synthesis environment directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev.
$ cd afu_dev\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating host_chan_mmio.gbs\n==================================\n...\n...\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\n
The previous output indicates the successful compilation of the AFU and the compliance with the timing requirements. Analyze the reports generated in case the design does not meet timing. The timing reports are stored in the directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev/build/syn/syn_top/output_files/timing_report.
Once the compilation finishes successfully, load the new host_chan_mmio.gbs bitstream file into the partial reconfiguration region of the target IPU Platform F2000X-PL board. Keep in mind, that the loaded image is dynamic - this image is not stored in flash and if the card is power cycled, then the PR region is re-loaded with the default AFU.
To load the image, perform the following steps:
# On Development Host\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev\n# Copy FIM files to SoC\n$ scp host_chan_mmio.gbs <user>@<SoC IP address>:</remote/directory>\n
# On SoC\n$ cd </remote/directory>\n$ fpgasupdate host_chan_mmio.gbs [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\n
Set up your board to work with the newly loaded AFU.
# On SoC\n# Verify PCIe b.d.f\n# For the following example, the F2000x SKU2 PCIe b:d.f is 15:00.0,\n# however this may be different in your system\n$ fpgainfo fme\nIntel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.2.4\nBoard Management Controller Build version: 1.2.4\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n...\n\n# Create the Virtual Functions (VFs) provided by the FIM, the default FIM has 3 VFs\n$ pci_device 15:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s 15:00\n15:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n15:00.1 Processing accelerators: Intel Corporation Device bccf\n15:00.2 Processing accelerators: Intel Corporation Device bccf\n15:00.3 Processing accelerators: Intel Corporation Device bccf\n\n# Bind VFs to VFIO driver. \n$ opae.io init -d 0000:15:00.1 opae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.1 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.1 is 52\n$ opae.io init -d 0000:15:00.2\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.2 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.2 is 53\n$ opae.io init -d 0000:15:00.3\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.3 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.3 is 54\n# Verify the new AFU is loaded. The host_chan_mmio AFU GUID is \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\".\n$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x6015000000000000\nPCIe s:b:d.f : 0000:15:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0x4015000000000000\nPCIe s:b:d.f : 0000:15:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0x2015000000000000\nPCIe s:b:d.f : 0000:15:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : 76d7ae9c-f66b-461f-816a-5428bcebdbc5\n
Now, navigate to the directory of the host_chan_mmio AFU containing the host application's source code, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw. Once there, compile the host_chan_mmio host application and execute it on the host server to excercise the functionality of the AFU.
Note: If OPAE SDK libraries were not installed in the default systems directories under /usr, you need to set the OPAE_LOC, LIBRARY_PATH, and LD_LIBRARY_PATH environment variables to the custom locations where the OPAE SDK libraries were installed.
# On Development Host, move to the sw directory of the the host_chan_mmio AFU. This directory holds the source for the host application.\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw\n$ make\n# Copy application to SoC\n$ scp host_chan_mmio <user>@<SoC IP address>:</remote/directory>\n
# On SoC, Run the application\n$ cd </remote/directory>\n$ ./host_chan_mmio\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#33-building-and-running-the-hello_world-example-afu","title":"3.3. Building and running the hello_world example AFU","text":"The platform-independent examples-afu repository also provides some interesting example AFUs. In this section, you will compile and execute the PIM based hello_world AFU. The RTL of the hello_world AFU receives from the host application an address via memory mapped I/O (MMIO) write and generates a DMA write to the memory line at that address. The content written to memory is the string \"Hello world!\". The host application spins, waiting for the memory line to be updated. Once available, the software prints out the string.
The hello_world example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using CCIP, Avalon, and AXI interfaces. However, this guide will use the AXI version of the AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the AFU hardware.
hello_world\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 ccip\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u2514\u2500\u2500 hello_world.json\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 hello_world\n \u251c\u2500\u2500 hello_world.c\n \u251c\u2500\u2500 Makefile\n \u2514\u2500\u2500 obj\n \u251c\u2500\u2500 afu_json_info.h\n \u2514\u2500\u2500 hello_world.o\n
The following instructions can be used to compile other AFU samples accompanying this repository.
If not done already, download and clone the examples-afu repository.
$ cd $OFS_BUILD_ROOT $ git clone https://github.com/OFS/examples-afu.git\n$ cd examples-afu\n$ git checkout tags/ofs-2024.1-1\n
Compile the hello_word sample AFU.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_synth_setup --source hw/rtl/axi/sources.txt afu_dev\n$ cd afu_dev\n$ ${OPAE_PLATFORM_ROOT}/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating hello_world.gbs\n==================================\n.\n.\n.\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'hello_world.gbs'\nDesign meets timing\n===========================================================================\n
To test the AFU in actual hardware, load the hello_world.gbs to the IPU Platform F2000X-PL card. For this step to be successful, the SoC Attach FIM must have already been loaded to the IPU Platform F2000X-PL card following the steps described in Section 2 of this document.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_dev/\n# Copy FIM files to SoC\n$ scp hello_world.gbs <user>@<SoC IP address>:</remote/directory>\n
# On SoC\n$ cd </remote/directory>\n$ fpgasupdate hello_world.gbs [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\n
Set up your IPU Platform F2000X-PL board to work with the newly loaded hello_world.gbs file.
# On SoC\n# Verify PCIe b.d.f\n# For the following example, the F2000x SKU2 PCIe b:d.f is 15:00.0,\n# however this may be different in your system\n$ fpgainfo fme\nIntel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.2.4\nBoard Management Controller Build version: 1.2.4\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n...\n\n# Create the Virtual Functions (VFs) provided by the FIM, the default FIM has 3 VFs\n$ pci_device 15:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s 15:00\n15:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n15:00.1 Processing accelerators: Intel Corporation Device bccf\n15:00.2 Processing accelerators: Intel Corporation Device bccf\n15:00.3 Processing accelerators: Intel Corporation Device bccf\n\n# Bind VFs to VFIO driver.\n$ opae.io init -d 0000:15:00.1\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.1 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.1 is 52\n$ opae.io init -d 0000:15:00.2\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.2 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.2 is 53\n$ opae.io init -d 0000:15:00.3\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.3 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.3 is 54\n# Verify the new AFU is loaded. The hello_world AFU GUID is \"c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\".\n$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x6015000000000000\nPCIe s:b:d.f : 0000:15:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0x4015000000000000\nPCIe s:b:d.f : 0000:15:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0x2015000000000000\nPCIe s:b:d.f : 0000:15:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n
Compile and execute the host application of the hello_world AFU. You should see the application outputs the \"Hello world!\" message in the terminal.
# On Development Host, move to the sw directory of the hello_world AFU and build application\n$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n$ make\n# Copy application to SoC\n$ scp hello_world <user>@<SoC IP address>:</remote/directory>\n
# On SoC, Run the application\n$ cd </remote/directory>\n$ ./hello_world\nHello world!\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#34-modify-the-afu-user-clocks-frequency","title":"3.4. Modify the AFU user clocks frequency","text":"AFU user clocks are currently not supported in F2000x base FIM configuration.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#4-simulating-an-afu-using-ase","title":"4. Simulating an AFU using ASE","text":"The Application Simulation Environment (ASE) is a hardware/software co-simulation environment for your AFU. See diagram below illustrating ASE operation:
ASE uses the simulator Direct Programming Interface (DPI) to provide HW/SW connectivity. The PCIe connection to the AFU under testing is emulated with a transactional model.
The following list describes ASE operation:
- Attempts to replicate the transactions that will be seen in real system.
- Provides a memory model to AFU, so illegal memory accesses can be identified early.
- Not a cache simulator.
- Does not guarantee synthesizability or timing closure.
- Does not model system latency.
- No administrator privileges are needed to run ASE. All code is user level.
The remainder of this section is a tutorial providing the steps on how to run ASE with either Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae using an example AFU and the AFU build tree previously created in this guide.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#41-set-up-steps-to-run-ase","title":"4.1. Set Up Steps to Run ASE","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#411-install-opae-sdk","title":"4.1.1. Install OPAE SDK","text":"The F2000x SKU2 card requires 2.12.0-5. Follow the instructions in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs to build and install the required OPAE SDK for the IPU Platform F2000X-PL. Make sure to check out the cloned repository to tag 2.12.0-5 and branch release/2.12.0.
$ git checkout tags/2.12.0-5 -b release/2.12.0\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#412-install-ase-tools","title":"4.1.2 Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK.
-
If not done already, set the environment variables as described in section, Set Up AFU Development Environment.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n$ git checkout tags/2.12.0-1 -b release/2.12.0
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#413-setup-required-ase-environment-variables","title":"4.1.3. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ cd /usr/bin\n$ export PATH=$PWD:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ cd /usr/lib\n$ export LIBRARY_PATH=$PWD\n$ cd /usr/lib64\n$ export LD_LIBRARY_PATH=$PWD\n$ cd $OFS_BUILD_ROOT/ofs-platform-afu-bbb\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to Modelsim installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#42-simulating-the-host_chan_mmio-afu","title":"4.2. Simulating the host_chan_mmio AFU","text":"The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files:
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\n
This example AFU contains examples using both Avalon and AXI interface buses. This guide will use the AXI version of the host_chan_mmio AFU.
ASE uses client-server application architecture to deliver hardware/software co-simulation. You require one shell for the hardware based simulation and another shell where the software application is running. The hardware is started first with a simulation compilation and simulator startup script, once the simulator has loaded the design, it will wait until the software process starts. Once the software process starts, the simulator proceeds. Transaction logging and waveform capture is performed.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#421-set-up-and-run-the-hw-simulation-process","title":"4.2.1 Set Up and Run the HW Simulation Process","text":"You will run the afu_sim_setup script to create the scripts for running the ASE environment. The afu_sim_setup script has the following usage:
usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n [-f] [--ase-mode ASE_MODE] [--ase-verbose]\n dst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\n Default simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
Run afu_sim_setup to create the ASE simulation environment for the host_chan_mmio example AFU. The '-t VCS' option indicates to prepare the ASE simulation environment for Synopsys\u00ae VCS\u00ae.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_sim_setup -s ./hw/rtl/test_mmio_axi1.txt -t VCS afu_sim\n\nCopying ASE from /opae-sdk/install-opae-sdk/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /ofs-f2000x-pl/work_pr/build_tree/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup creates the ASE scripts in the directory host_chan_mmio_sim where the afu_sim_setup script was run. Start the simulator as shown below:
$ cd afu_sim\n$ make\n$ make sim\n
This process launches the AFU hardware simulator. Before moving to the next section, pay attention to the simulator output highlighted in the image below.
The simulation artifacts are stored in host_chan_mmio/work and consist of:
log_ase_events.tsv\nlog_ofs_plat_host_chan.tsv \nlog_ofs_plat_local_mem.tsv \nlog_pf_vf_mux_A.tsv \nlog_pf_vf_mux_B.tsv \n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#422-set-up-and-run-the-sw-process","title":"4.2.2 Set Up and Run the SW Process","text":"Open an additional shell to build and run the host application that communicates with the actual AFU hardware. Set up the same environment variable you have set up in the shell you have been working on until this point.
Additionally, as indicated by the hardware simulator output that is currently executing in the \"simulator shell\", copy and paste the line \"export ASE_WORKDIR=...\", into the new \"software shell\". See the last image of the previous section.
$ export ASE_WORKDIR=$OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_sim/work\n
Then, go to the sw directory of the host_chan_mmio AFU example to compile the host application. $ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw $ make\n\nafu_json_mgr json-info --afu-json=../hw/rtl/host_chan_mmio.json --c-hdr=obj/afu_json_info.h\nWriting obj/afu_json_info.h\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c main.c -o obj/main.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c test_host_chan_mmio.c -o obj/test_host_chan_mmio.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/connect.c -o obj/connect.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/csr_mgr.c -o obj/csr_mgr.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/hash32.c -o obj/hash32.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/test_data.c -o obj/test_data.o\ncc -o host_chan_mmio obj/main.o obj/test_host_chan_mmio.o obj/connect.o obj/csr_mgr.o obj/hash32.o obj/test_data.o -z noexecstack -z relro -z now -pie -luuid -lopae-c-ase\n
Now, launch the host application to exercise the AFU hardware running on the simulator shell. The next image shows the AFU hardware simulation process on the left side shell. The right hand shell shows the host application's output of a successful simulation.
$ with_ase ./host_chan_mmio\n [APP] Initializing simulation session ...\nRunning in ASE mode\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n [APP] Deinitializing simulation session\n [APP] Took 1,003,771,568 nsec\n [APP] Session ended\n
Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
$ make wave\n
This brings up the Synopsys\u00ae VCS\u00ae simulator GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | afu , as shown below.
Right click on the afu (afu) entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#43-simulating-the-hello_world-afu","title":"4.3 Simulating the hello_world AFU","text":"In this section you will quickly simulate the PIM-based hello_world sample AFU accompanying the examples-afu repository.
-
Set the environment variables as described in section 4.1. Set Up Steps to Run ASE.
-
Prepare an RTL simulation environment for the AXI version of the hello_world AFU.
Simulation with ASE requires two software processes, one to simulate the AFU RTL and the other to run the host software that excercises the AFU. To construct an RTL simulation environment under the directory simulation, execute the following.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_sim_setup -s ./hw/rtl/axi/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/<user_area>/ofs-f2000x-pl/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup script constructs an ASE environment in the hello_world_sim subdirectory. If the command fails, confirm that the path to the afu_sim_setup is on your PATH environment variable (in the OPAE SDK bin directory) and that your Python version is at least 2.7.
- Build and execute the AFU RTL simulator.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_sim\n$ make\n$ make sim
The previous commands will build and run the Synopsys\u00ae VCS\u00ae RTL simulator, which prints a message saying it is ready for simulation. The simulation process also prints a message instructing you to set the ASE_WORKDIR environment variable in a second shell.
-
Open a second shell where you will build and execute the host software. In this new \"software shell\", set up the environment variables you have set up so far in the \"hardware simulation\" shell.
-
Also, set the ASE_WORKDIR environment variable following the instructions given in the \"hardware simulation\" shell.
$ export ASE_WORKDIR=$OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_sim/work\n
-
Then, move to the sw directory of the hello_world AFU sample to build the host software.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n$ make
-
Run the hello_world host application to resume the work of the RTL simulation. The host software process and the RTL simulation execute in lockstep. If successful, you should see the Hello world! output.
$ with_ase ./hello_world\n\n[APP] Initializing simulation session ...\nHello world!\n [APP] Deinitializing simulation session\n [APP] Took 43,978,424 nsec\n [APP] Session ended\n
The image below shows the simulation of the AFU hardware and the execution of the host application side-by-side.
- Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
make wave\n
This brings up the DVE GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | hello_afu, as shown below.
Right click on the hello_afu entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#5-adding-remote-signal-tap-logic-analyzer-to-debug-the-afu","title":"5. Adding Remote Signal Tap Logic Analyzer to debug the AFU","text":"Remote Signal Tap is currently not supported in F2000x base FIM configuration.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#6-disabling-the-flr-function-level-reset-during-afu-debugging","title":"6. Disabling the FLR (Function Level Reset) during AFU Debugging","text":"The vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\n
Enable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\n
If you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\n
Sample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#7-how-to-modify-the-pfvf-mux-configuration","title":"7. How to modify the PF/VF MUX configuration","text":"For information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/","title":"Shell Developer Guide: Agilex\u00ae 7 SoC Attach: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1-introduction","title":"1 Introduction","text":""},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#11-about-this-document","title":"1.1 About This Document","text":"This document serves as a guide for OFS Agilex\u00ae 7 SoC Attach developers targeting the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL. The following topics are covered in this guide:
- Compiling the OFS Agilex\u00ae 7 SoC Attach FIM design
- Simulating the OFS Agilex\u00ae 7 SoC Attach FIM design
- Customizing the OFS Agilex\u00ae 7 SoC Attach FIM design
- Configuring the FPGA with an OFS Agilex\u00ae 7 SoC Attach FIM design
This document uses the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL as the platform to illustrate key points and demonstrate how to extend the capabilities provided in OFS. The demonstration steps serve as a tutorial for the development of your OFS knowledge.
This document covers OFS architecture lightly. For more details on the OFS architecture, please see Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#111-knowledge-prerequisites","title":"1.1.1 Knowledge Prerequisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex\u00ae 7 SoC Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus Prime Pro Version 23.4, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Signal Tap Logic Analyzer tool software.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex\u00ae 7 SoC Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex\u00ae 7 SoC Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":"Agilex\u00ae 7 SoC Attach OFS supports the following features.
FIM BASE IPU Platform F2000X-PL f2000x PCIe Configuration Host: PCIe Gen4x16SoC: PCIe Gen4x16 SR-IOV support Host: 2 PFs, No VFsSoC: 1 PFs, 3 VFs AXI ST datapath 512b @ 470MHz Transceiver Subsystem Configuration 2x4x25G The FIM also integrates:
- SoC AFU and Host AFU
- Exercisers demonstrating PCIe, external memory, and Ethernet interfaces
- FME CSR
- Remote Signal Tap
- Partial Reconfiguration of the SoC AFU
The Host exercisers are provided for the quick evaluation of the FIM and can be leveraged for the verification of the platform's functionality and capabilities. The host exercisers can be removed by the designer to release FPGA real estate to accommodate new workload functions. To compile the FIM without host exercisers go to How to compile the FIM in preparation for designing your AFU.
OFS is extensible to meet the needs of a broad set of customer applications. The general use cases listed below are examples where the OFS base design is easily extended to build a custom FIM:
- Use OFS design example as-is
- Porting the code to another platform that is identical to OFS reference platform changing targeted FPGA device and pinout
- Change I/O assignments without changing design
- Update the configuration of peripheral IP in OFS design example, not affecting FIM architecture
- External memory settings
- Ethernet Subsystem analog settings
- Remove/update peripheral feature in OFS design example, not affecting FIM architecture
- External memory speed/width change
- Change number of VFs supported
- Add new features as an extension to OFS design example, not affecting the FIM architecture
- Add/remove external memory interface to the design
- Add/remove user clocks for the AFU
- Add/remove IP to the design with connection to the AFU
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1211-top-level-fpga","title":"1.2.1.1 Top Level FPGA","text":"OFS separates the FPGA design into two areas: FPGA Interface Manager (FIM) and workload (or Acceleration Function Unit) as shown in the figure below:
As can be seen in this diagram, the OFS FPGA structure has a natural separation into two distinct areas:
- FPGA Interface Manager (FIM or sometimes called the \"the shell\") containing:
- FPGA external interfaces and IP cores (e.g. Ethernet, DDR-4, PCIe, etc)
- PLLs/resets
- FPGA - Board management infrastructure
- Interface to Acceleration Function Unit (AFU)
- Acceleration Function Unit (\"the workload\")
- Uses the FIM interfaces to perform useful work inside the FPGA
- Contains logic supporting partial reconfiguration
- Remote Signal Tap core for remote debugging of SoC AFU PR region
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex\u00ae 7 SoC Attach design are listed below.
- Host interface
- PCIe Gen4 x 16
- SoC Interface
- PCIe Gen4 x 16
- Network interface
- 2 - QSFP28/56 cages
- Eight Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels of 25G Ethernet interfacing to an E-tile Ethernet Subsystem.
- External Memory - DDR4
- Four Fabric DDR4-2400 banks - 4 GB organized as 1Gb x 32 with 1 Gb x 8 ECC (ECC login not implemented in default FIM)
- Board Management
- SPI interface
- FPGA configuration
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex Agilex\u00ae 7 SoC Attach f2000x FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem Intel FPGA PCI Express Subsystem IP User Guide N/A Memory Subsystem Intel FPGA Memory Subsystem IP User Guide 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
The host control and data flow is shown in the diagram below:
The control and data paths are composed of the following:
- Host Interface Adapter (PCIe)
- SoC Interface Adapter (PCIe)
- Low Performance Peripherals
- Slow speed peripherals (JTAG, I2C, Smbus, etc)
- Management peripherals (FME)
- High Performance Peripherals
- Memory peripherals
- Acceleration Function peripherals (eg. AFUs)
- HPS Peripheral
- Fabrics
- Peripheral Fabric (multi drop)
- AFU Streaming fabric (point to point)
Peripherals are connected to one another using AXI, either:
- Via the peripheral fabric (AXI4-Lite, multi drop)
- Via the AFU streaming fabric (AXI-S, point to point)
Peripherals are presented to software as:
- OFS managed peripherals that implement DFH CSR structure.
- Native driver managed peripherals (i.e. Exposed via an independent PF, VF)
The peripherals connected to the peripheral fabric are primarily OPAE managed resources, whereas the peripherals connected to the AFU are \u201cprimarily\u201d managed by native OS drivers. The word \u201cprimarily\u201d is used since the AFU is not mandated to expose all its peripherals to OPAE.
OFS uses a defined set of CSRs to expose the functionality of the FPGA to the host software. These registers are described in Open FPGA Stack Reference Manual - MMIO Regions section.
If you make changes to the FIM that affect the software operation, then OFS provides a mechanism to communicate that information to the proper software driver that works with your new hardware. The Device Feature Header (DFH) structure is followed to provide compatibility with OPAE software. Please see FPGA Device Feature List (DFL) Framework Overview for a description of DFL operation from the driver perspective.
In the default design, the SoC and Host AFUs are isolated from each other. You must develop mechanisms for Host - SoC communication if desired.
Note: The default configuration of the Board Peripheral Fabric, there is a connection from the Host Interface to the PMCI-SS, however the PMCI-SS is not in the Host DFL, and is not discovered by Host SW by default. If you want to guarantee that the Host can not access the PMCI-SS, and by extension the Board BMC, you must implement a filtering mechanism, for example, in the Host ST2MM module to prevent access to the PMCI-SS address space.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workloads in the OFS Agilex\u00ae 7 SoC Attach f2000x FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI f2000x: Used to exercise and characterize HSSI interfaces. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex Agilex\u00ae 7 SoC Attach f2000x FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the SoC/Host software and AFU and board peripherals. The following tables describe the address mapping of the APF and BPF for both the SoC and the Host.
Table: SoC APF Address Map
Address Size (Bytes) Feature 0x00_0000 - 0x0F_FFFF 1024K Board Peripherals (See SoC BPF Address Map table) 0x10_0000 - 0x10_FFFF 64K ST2MM 0x11_0000 - 0x12_FFFF 128K Reserved 0x13_0000 - 0x13_FFFF 64K PR Gasket: 4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x14_0000 - 0x14_FFFF 64K AFU Error Reporting Table: Host APF Address Map
Address Size (Bytes) Feature 0x00_0000 - 0x0F_FFFF 1024K Board Peripherals (See Host BPF Address Map table) 0x10_0000 - 0x10_FFFF 64K ST2MM 0x11_0000 - 0x13_FFFF 192K Reserved 0x14_0000 - 0x14_FFFF 64K AFU Error Reporting Table: SoC BPF Address Map
Address Size (Bytes) Feature 0x0_0000 - 0x0_FFFF 64K FME 0x1_0000 - 0x1_0FFF 4K SoC PCIe IF 0x1_1000 - 0x1_1FFF 4K Reserved 0x1_2000 - 0x1_2FFF 4K QSFP0 0x1_3000 - 0x1_3FFF 4K QSFP1 0x1_4000 - 0x1_4FFF 4K Ethernet Sub-System 0x1_5000 - 0x1_5FFF 4K Memory Sub-System 0x8_0000 - 0xF_FFFF 512K PMCI Controller Table: Host BPF Address Map
Address Size (Bytes) Feature 0x0_0000 - 0x0_0FFF 4K Host PCIe IF 0x8_0000 - 0xF_FFFF 512K PMCI Controller"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customization Examples Table lists the general user flows for OFS Agilex\u00ae 7 SoC Attach f2000x FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customization Examples
Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Modify and run UVM tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Memory Sub-System Using IP Presets With OFSS Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Modify the Ethernet Sub-System Without HSSI OFSS Set up JTAG Program the FPGA via JTAG Program the FPGA via RSU"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Refer to the following guides for instructions on setting up an OFS deployment environment.
- Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL
- Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
The recommended Best Known Configuration (BKC) operating system for development of the OFS FIM is RedHatEnterprise Linux\u00ae (RHEL) 8.6, which is the assumed operating system for this developer guide.
The recommended development environment requires the following:
- Workstation or server with a Quartus Prime Pro Version 23.4 installed on a Quartus Prime Pro-supported Linux distribution. See Operating System Support. The Linux distribution known to work with this version of RedHatEnterprise Linux\u00ae (RHEL) 8.6 . Note, Windows is not supported.
- Compilation targeting Agilex\u00ae 7 FPGA devices requires a minimum of 64 GB of RAM.
- Simulation of lower level functionality (not chip level) is supported by Synopsys\u00ae VCS and Mentor Graphics\u00ae QuestaSim SystemVerilog simulators.
- Simulation of chip level requires Synopsys VCS and VIP
You may modify the build scripts and pin files to target different boards with Agilex\u00ae 7 FPGA devices.
Testing the Agilex\u00ae 7 SoC Attach on the IPU Platform F2000X-PL hardware requires a deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Development Operating System RedHatEnterprise Linux\u00ae (RHEL) 8.6 N/A Quartus Prime Software Quartus Prime Pro Version 23.4 for Linux + Patches 0.17 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A git 1.8.3.1 PERL 5.8.8 FIM Source Files ofs-2024.1-1 Section 1.3.2.1"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use Ubuntu 22.04 LTS for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.17.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"OFS provides a framework of FPGA synthesizable code, simulation environment, and synthesis/simulation scripts. FIM designers can use the provided code as-is, modify the provided code, or add new code to meet your specific product requirements. Instructions for compiling the existing design is given in the FIM Compilation section, while instructions for customizing the default design can be found in the FIM Customization section.
The source files for the OFS Agilex\u00ae 7 SoC Attach FIM are provided in the following repository: https://github.com/OFS/ofs-f2000x-pl/releases/tag/ofs-2024.1-1.
Some essential directories in the repository are described as follows:
| ipss // Contains files for IP Sub-Systems\n| | hssi // Contains source files for HSSI Sub-System\n| | mem // Contains source files for HSSI Sub-System\n| | pcie // Contains source files for PCIe Sub-System\n| | pmci // Contains source files for PMCI Sub-System\n| | qsfp // Contains source files for QSFP Sub-System\n| ofs-common // Contains files which are common across OFS platforms\n| | scripts // Contains common scripts\n| | src // Contains common source files, including host exercisers\n| | tools // Contains common tools files\n| | verification // Contains common UVM files\n| sim // Contains simulation files\n| | bfm\n| | common\n| | scripts\n| | unit_test // Contains files for all unit tests\n| src // Contains source files for Agilex Agilex\u00ae 7 SoC Attach FIM\n| | afu_top // Contains top-level source files for AFU\n| | includes // Contains source file header files\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | top // Contains top-level source files, including design top module\n| syn // Contains files related to design synthesis\n| | scripts\n| | setup // Contains setup files, including pin constraints and location constraints\n| | syn_top // Contains Quartus project files\n| tools\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| verification // Contains files for UVM testing\n| | scripts\n| | testbench\n| | tests\n| | unit_tb\n| | verifplan\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 SoC Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-f2000x-pl.git\n
-
Check out the correct tag of the repository
cd ofs-f2000x-pl\ngit checkout --recurse-submodules tags/ofs-2024.1-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.1-1)\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-f2000x-pl\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.F2000X_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.F2000X_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 23.4 for Linux with Agilex\u00ae 7 FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions. Quartus Prime Pro Version 23.4 is the currently verified version of Quartus Prime Pro used for building the FIM and AFU images. Porting to newer versions of Quartus Prime Pro may be performed by developers, however, you will need to verify operation.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 23.4 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
Clone the ofs-f2000x-pl repository if not already cloned. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $OFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.17. All required patches are provided in the Assets of the OFS FIM Release: https://github.com/OFS/ofs-f2000x-pl/releases/tag/ofs-2024.1-1
-
Extract and unzip the patch-agx7-2024-1.tar.gz file.
tar -xvzf patch-agx7-2024-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-23.4-0.17-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This section describes the theory behind FIM compilation.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config> Used to modify IP, such as the PCIe SS, using .ofss configuration files. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk | f2000x Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
If you wish to compile the f2000x FIM using the Quartus Prime Pro GUI, you must at least run the setup portion of the build_top.sh script before compiling with the GUI. For instructions on compiling the FIM using the Quartus GUI, refer to the Compiling the OFS FIM Using Quartus GUI section.
The build scripts included with OFS are verified to run in a bash shell. Other shells have not been tested. The full build script typically takes around 3 hours to complete.
The build script copies the ipss, ofs-common, sim, src,syn and tools directories to the specified work directory and then these copied files are used in the Quartus Prime Pro compilation process.
Some key output directories are described in the following table:
Directory Description $OFS_ROOTDIR/<WORK_DIR>/syn/syn_top Quartus Prime Pro project (ofs_top.qpf) and other Quartus Prime Pro specific files $OFS_ROOTDIR/<WORK_DIR>/syn/syn_top/output_files Directory with build reports and FPGA programming files The build script will run PACSign (if installed) and create an unsigned FPGA programming files for both user1 and user2 locations of the f2000x FPGA flash. Please note, if the f2000x has the root entry hash key loaded, then PACsign must be run to add the proper key to the FPGA binary file.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script can use OFSS files to easily customize design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building FIMs; it is not required if the source files are already configured as desired. The Provided OFSS Files table below describes the pre-made OFSS files for the f2000x that can be found in the $OFS_ROOTDIR/tools/ofss_config directory.
Table: Provided OFSS Files
OFSS File Name Location Type Description f2000x.ofss $OFS_ROOTDIR/tools/ofss_config Top Top level OFSS file which includes OFS, PCIe Host, PCIe SoC, IOPLL, and Memory OFSS files. f2000x_base.ofss $OFS_ROOTDIR/tools/ofss_config ofs Defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the f2000x Host pcie_soc.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the f2000x SoC iopll.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as f2000x hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE CAUI-4 hssi_4x100_caui2.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE CAUI-2 There can typically be three sections contained within an OFSS file.
-
[include]
- This section of an OFSS file contains elements separated by a newline, where each element is the path to an OFSS file that is to be included for configuration by the OFSS Configuration Tool. Ensure that any environment variables (e.g.
$OFS_ROOTDIR) is set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter
-
[ip]
- This section of an OFSS file contains a key value pair that allows the OFSS Config tool to determine which IP configuration is being passed in. The currently supported values of IP are
ofs, iopll, pcie, memory, and hssi.
-
[settings]
- This section of an OFSS file contains IP specific settings. Refer to an existing IP OFSS file to see what IP settings are set. For the IP type
ofss, the settings will be information of the OFS device (platform, family, fim, part #, device_id)
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2121-platform-ofss-file","title":"2.1.2.1 Platform OFSS File","text":"The <platform>.ofss file (e.g. $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss) is the platform level OFSS wrapper file. This is typically the OFSS file that is provided to the build script. It only contains an include section which lists all other OFSS files that are to be used when the <platform>.ofss file is passed to the build script.
The generic structure of a <platform>.ofss file is as follows:
[include]\n<PATH_TO_PLATFORM_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2122-ofs-ip-ofss-file","title":"2.1.2.2 OFS IP OFSS File","text":"An OFSS file with IP type ofs (e.g. $OFS_ROOTDIR/tools/ofss_config/f2000x_base.ofss) contains board specific information for the target board.
The default configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Defaults
Section Parameter f2000x Default Value [ip] type ofs [settings] platform f2000x family agilex fim base_x16 part AGFC023R25A2E2VR0 device_id 6100"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2123-pcie-ip-ofss-file","title":"2.1.2.3 PCIe IP OFSS File","text":"An OFSS file with IP type pcie (e.g. $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss) is used to configure the PCIe-SS in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, the Host must have at least 1 PF. The SoC must have at least 1 PF with 1 VF. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations tables below describe the supported number of PFs and VFs for the Host and SoC.
Table: Host PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 0 Max # of VFs 2000 distributed across all PFs Table: SoC PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 1 (on PF0) Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description [ip] type pcie Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss | soc_pcie_ss Specifies the output name of the PCIe-SS IP preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF bar0_address_width Integer bar4_address_width Integer vf_bar0_address_width Integer ats_cap_enable 0 | 1 vf_ats_cap_enable 0 | 1 prs_ext_cap_enable 0 | 1 pasid_cap_enable 0 | 1 pci_type0_vendor_id 32'h Value pci_type0_device_id 32'h Value revision_id 32'h Value class_code 32'h Value subsys_vendor_id 32'h Value subsys_dev_id 32'h Value sriov_vf_device_id 32'h Value exvf_subsysid 32'h Value The default values for all PCIe-SS parameters (that are not defined in the PCIe IP OFSS file) are defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2124-iopll-ip-ofss-file","title":"2.1.2.4 IOPLL IP OFSS File","text":"An OFSS file with IP type iopll (e.g. $OFS_ROOTDIR/tools/ofss_config/iopll/iopll.ofss) is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description [ip] type iopll Specifies that this OFSS file configures the IOPLL [settings] output_name sys_pll Specifies the output name of the IOPLL. instance_name iopll_0 Specifies the instance name of the IOPLL. [p_clk] freq Integer: 250 - 470 Specifies the IOPLL clock frequency in MHz. Note: The following frequencies have been tested on reference boards: 350MHz, 400MHz, 470MHz.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2125-memory-ip-ofss-file","title":"2.1.2.5 Memory IP OFSS File","text":"An OFSS file with IP type memory (e.g. $OFS_ROOTDIR/tools/ofss_config/memory/memory.ofss) is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description [ip] type memory Specifies that this OFSS file configures the Memory-SS [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. preset f2000x | String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi (e.g. $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_8x25.ofss) is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS num_channels Integer Specifies the number of channels. data_rate 10GbE | 25GbE | 100GCAUI-4 | 100GAUI-2 Specifies the data rate preset None | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. [1] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets should be stored in a $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is: $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/syn_top/output_files
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes some of the essential images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Description ofs_top.bin This is an intermediate, raw binary file. This intermediate raw binary file is produced by taking the Quartus Prime Pro generated *.sof file, and converting it to *.pof using quartus_pfg, then converting the *.pof to *.hexout using quartus_cpf, and finally converting the *.hexout to *.bin using objcopy. ofs_top.bin - Raw binary image of the FPGA. ofs_top_page0_unsigned_factory.bin This is the unsigned PACSign output generated for the Factory Image. Unsigned means that the image has been signed with an empty header. ofs_top_page1_user1.bin This is an input file to PACSign to generate ofs_top_page1_unsigned_user1.bin. This file is created by taking the ofs_top.bin file and assigning the User1 or appending factory block information. Unsigned means that the image has been signed with an empty header. ofs_top_page1_unsigned_user1.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User1 Image. This file is used to load the FPGA flash User1 Image using the fpgasupdate tool. Unsigned means that the image has been signed with an empty header. ofs_top_page2_user2.bin This is an input file to PACSign to generate ofs_top_page2_unsigned_user2.bin. This file is created by taking the ofs_top.bin file and assigning the User2 or appending factory block information. ofs_top_page2_unsigned_user2.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User2 Image. This file is used to load the FPGA flash User2 Image using the fpgasupdate tool. Unsigned means that the image has been signed with an empty header. ofs_top.sof This image is used to generate ofs_top.bin, and can also be used to program the Agilex\u00ae 7 FPGA device directly through JTAG Note: The build/output_files/timing_report directory contains clocks report, failing paths and passing margin reports.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <PATH_OF_RELOCATABLE_PR_TREE> <BOARD_TARGET> <WORK_DIRECTORY>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <PATH_OF_RELOCATABLE_PR_TREE> String Specifies the location of the relocatable PR directory tree to be created. <BOARD_TARGET> n6001 | n6000 | fseries-dk | iseries-dk Specifies the name of the board target. <WORK_DIRECTORY> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_page0_unsigned_factory.bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_page1_unsigned_user1.bin\n\u2514\u2500\u2500 \u2514\u2500\u2500 \u2514\u2500\u2500 ofs_top_page2_unsigned_user2.bin\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex\u00ae 7 SoC Attach FIM for the f2000x:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. Some examples are provided:
-
Out-of-Tree PR FIM using OFSS (Standard Flow)
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_oot_pr\n
Refer to the Create a Relocatable PR Directory Tree section for more information on out-of-tree PR builds.
-
Flat FIM using OFSS
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x_flat\n
-
In-Tree PR FIM using OFSS
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_in_tree_pr\n
-
Flat FIM using OFSS with Host Exercisers Removed
bash ./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat,null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_f2000x_flat_null_he
-
In-Tree PR FIM using Source
./ofs-common/scripts/common/syn/build_top.sh f2000x work_f2000x\n
The build takes ~3 hours to complete. A successful build will report the following:
Compile work directory: <OFS_ROOTDIR>/agx7-soc-attach/rel/default/ofs-f2000x-pl/work_f2000x/syn/syn_top\nCompile artifact directory: <OFS_ROOTDIR>/agx7-soc-attach/rel/default/ofs-f2000x-pl/work_f2000x/syn/syn_top/output_files\n\n***********************************\n***\n*** OFS_PROJECT: f2000x\n*** OFS_FIM: base\n*** OFS_BOARD: adp\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 1\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#226-creating-a-relocatable-pr-directory-tree","title":"2.2.6 Creating a Relocatable PR Directory Tree","text":"If you are developing a FIM to be used by another team developing AFU workload(s), scripts are provided that create a relocatable PR directory tree. ODM and board developers will make use of this capability to enable a broad set of AFUs to be loaded on a board using PR.
You can create this relocatable PR directory tree by either:
- Build FIM and AFU using ofs-common/scripts/common/syn/build_top.sh followed by running /ofs-common/scripts/common/syn/generate_pr_release.sh
- Build FIM and AFU using ofs-common/scripts/common/syn/build_top.sh with optional -p switch included
The generate_pr_release.sh has the following command structure:
ofs-common/scripts/common/syn/generate_pr_release.sh -t <work_directory>/build_tree <target_board> <work_directory>\n
Where: <work_directory>/build_tree is the location for your relocatable PR directory tree in the work directory<target_board> is the name of the board target/FIM e.g. f2000x <work_directory> is the work directory
Here is an example of running the generate_pr_release.sh script:
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/generate_pr_release.sh -t work_f2000x/build_tree f2000x work_f2000x
The resulting relocatable build tree has the following structure:
.\n\u251c\u2500\u2500 bin\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 afu_synth\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build_env_config\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 run.sh -> afu_synth\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 update_pim\n\u251c\u2500\u2500 hw\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 blue_bits\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ofs_top_page0_unsigned_factory.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ofs_top_page1_unsigned_user1.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ofs_top_page2_unsigned_user2.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 ofs_top.sof -> ../lib/build/syn/syn_top/output_files/ofs_top.sof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 lib\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-ifc-id.txt\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-platform-class.txt\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 platform\n\u2514\u2500\u2500 README\n
This build tree can be moved to a different location and used for AFU development of PR-able AFU to be used with this board."},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2261-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6.1 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM.
Note: This can be automatically done for you if you run the build script with the -p option.
Note: This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the f2000x OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_f2000x/pr_build_template f2000x work_f2000x\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
vim $OFS_ROOTDIR/syn/syn_top/ofs_top.qsf\n
set_global_assignment -name SEED 2\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#228-compiling-the-ofs-fim-using-quartus-gui","title":"2.2.8 Compiling the OFS FIM Using Quartus GUI","text":"Perform the following steps to compile the OFS FIM using the Quartus GUI:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Set the environment variables as described in the Setting Up Required Environment Variables section.
-
Run the setup portion of the build script. This takes a few seconds to complete.
./ofs-common/scripts/common/syn/build_top.sh --stage setup f2000x work_dir\n
-
Open the OFS FIM project using the Quartus Prime Pro GUI. The project is located at $OFS_ROOTDIR/work_dir/syn/syn_top/ofs_top.qpf.
-
Ensure the checkbox next to Assembler (Generate programming files) is marked.
-
Click the arrow next to Compile Design in the Compilation Flow window to start full compilation.
-
After compilation is complete, check the $OFS_ROOTDIR/work_dir/syn/syn_top/output_files directory. This directory will contain the output SOF image generated by Quartus, named ofs_top.sof.
-
Run the finish portion of the build script. This will generate the images that can be programmed to f2000x FPGA flash. Use the optional -p argument to create an out-of-tree PR build.
./ofs-common/scripts/common/syn/build_top.sh --stage finish f2000x work_dir\n
-
Check the $OFS_ROOTDIR/work_dir/syn/syn_top/output_files directory again to see that all output files have been generated.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#3-unit-level-simulation","title":"3 Unit Level Simulation","text":"Unit level simulation of key components in the FIM is provided. These simulations provide verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS-MX or Mentor Graphics Questasim simulators. Readme files are provided explaining how to run the simulation of each component.
The scripts to run the unit level simulations are located in $OFS_ROOTDIR/sim/unit_test, which contains subdirectories soc_tests and host_tests. The table below lists the tests that are provided.
Test Type Test Name SoC Tests csr_test dfh_walker flr he_lb_test he_mem_test hssi_kpi_test hssi_test mem_ss_csr_test mem_ss_rst_test mem_tg_test pf_vf_access_test qsfp_test regress_run.py Host Tests csr_test he_lb_test pcie_csr_test pf_vf_access_test pmci_csr_test pmci_mailbox_test pmci_rd_default_value_test pmci_ro_mailbox_test pmci_vdm_b2b_drop_err_scenario_test pmci_vdm_len_err_scenario_test pmci_vdm_mctp_mmio_b2b_test pmci_vdm_multipkt_error_scenario_test pmci_vdm_multipkt_tlp_err_test pmci_vdm_tlp_error_scenario_test pmci_vdm_tx_rx_all_random_lpbk_test pmci_vdm_tx_rx_all_toggle_test pmci_vdm_tx_rx_lpbk_test regress_run.py"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config> Used to modify IP, such as the PCIe SS, using .ofss configuration files. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. Platform Dependent[1] <build_target> n6001 | n6000 | fseries-dk | iseries-dk | f2000x Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional [1] Using OFSS is required for the N6000, F-Series Development Kit (2xF-Tile), and the I-Series Development Kit (2xR-Tile, 1xF-Tile).
Refer to the Run Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files; refer to the Run Regression Unit Level Simulation section for step-by-step instructions.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#321-walkthrough-run-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Run Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test on either the SoC or Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the target design.
./gen_sim_files.sh --ofss=$OFS_ROOTDIR/tools/ofss_config/f2000x.ofss f2000x\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
```bash sh run_sim.sh TEST=dfh_walker VCSMX=1
-
Once the test has completed, the test summary will be shown. The following is an example test summary after running the SoC DFH Walker Test:
Test status: OK\n\n********************\nTest summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/applications.fpga.ofs.fim-f2000x-pl/sim/unit\n_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 355393750\nV C S S i m u l a t i o n R e p o r t\nTime: 355393750 ps\nCPU Time: 59.870 seconds; Data structure size: 91.2Mb\nSun May 14 16:54:20 2023\n
-
The log for the test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_TYPE>/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the SoC DFH Walker test using VCS can be found in:
$OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker/sim_vcs/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review. You are encouraged to run the additional simulation examples to learn about each key area of the OFS shell.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"The regress_run.py script is provided to automatically run all unit tests for either the SoC or the Host. Note that running all tests will take around three hours for the SoC tests and around 2.5 hours for the Host tests to complete.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#331-walkthrough-run-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Run Regression Unit Level Simulation","text":"Perform the following steps to run comprehensive regression unit tests.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the test directory you wish to run from.
-
SoC Tests
cd $OFS_ROOTDIR/sim/unit_test/soc_tests\n
-
Host Tests
cd $OFS_ROOTDIR/sim/unit_test/host_tests\n
-
Run the tests with the regress_run.py. Use the -h argument to display the help menu. For example, to run all tests locally, generate the simulation files, using VCS, with 8 processors:
python regress_run.py -g -l -n 8 -k all -s vcs\n
-
Once all tests have completed, the comprehensive test summary will be shown. The following is an example test summary after running the SoC tests:
2023-05-14 19:10:55,404: Passing Unit Tests:12/12 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n2023-05-14 19:10:55,404: csr_test:......... PASS -- Time Elapsed:0:03:57.713154\n2023-05-14 19:10:55,404: dfh_walker:....... PASS -- Time Elapsed:0:02:46.025067\n2023-05-14 19:10:55,404: flr:.............. PASS -- Time Elapsed:0:03:26.289900\n2023-05-14 19:10:55,404: he_lb_test:....... PASS -- Time Elapsed:0:06:41.142643\n2023-05-14 19:10:55,404: he_mem_test:...... PASS -- Time Elapsed:1:39:01.824096\n2023-05-14 19:10:55,404: hssi_kpi_test:.... PASS -- Time Elapsed:2:21:33.007328\n2023-05-14 19:10:55,404: hssi_test:........ PASS -- Time Elapsed:2:16:36.821034\n2023-05-14 19:10:55,404: mem_ss_csr_test:.. PASS -- Time Elapsed:0:38:45.836540\n2023-05-14 19:10:55,404: mem_ss_rst_test:.. PASS -- Time Elapsed:0:40:51.065354\n2023-05-14 19:10:55,404: mem_tg_test:...... PASS -- Time Elapsed:0:54:00.210146\n2023-05-14 19:10:55,404: pf_vf_access_test: PASS -- Time Elapsed:0:03:25.561919\n2023-05-14 19:10:55,404: qsfp_test:........ PASS -- Time Elapsed:0:39:29.192304\n2023-05-14 19:10:55,404: Failing Unit Tests: 0/12 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n2023-05-14 19:10:55,404: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n2023-05-14 19:10:55,404: Number of Unit test results captured: 12\n2023-05-14 19:10:55,404: Number of Unit test results passing.: 12\n2023-05-14 19:10:55,404: Number of Unit test results failing.: 0\n2023-05-14 19:10:55,404: End Unit regression running at date/time................: 2023-05-14 19:10:55.404641\n2023-05-14 19:10:55,404: Elapsed time for Unit regression run....................: 2:22:39.310455\n
-
Output logs for each individual test are saved in the respective test's directory
$OFS_ROOTDIR/sim/unit_test/<TEST_TYPE>/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the SoC DFH Walker test using VCS can be found in:
$OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker/sim_vcs/transcript\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4-fim-customization","title":"4 FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Table: FIM Customization Walkthroughs
Walkthrough Name How to add a new module to the FIM How to debug the FIM with Signal Tap How to compile the FIM in preparation for designing your AFU How to resize the Partial Reconfiguration Region How to modify the Memory Subsystem How to compile the FIM with no HSSI How to change the PCIe Device ID and Vendor ID How to migrate to a different Agilex 7 device number How to change the Ethernet interface from 8x25 GbE to 8x10 GbE How to change the Ethernet interface from 8x25 GbE to 2x100 GbE How to add more transceiver channels to the FIM How to modify the PF/VF MUX configuration How to create a minimal FIM"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a walkthrough for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the SoC. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the f2000x card. The process for these are described in this section.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can only be mastered by the SoC. The BPF is an interconnect generated by Platform Designer. The figure below shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
You can create, modify, and/or generate the BPF manually in Platform Designer or more conveniently by executing a provided script.
We will add the Hello FIM module to an un-used address space in the SoC MMIO region. The table below shows the MMIO region for the SoC with the Hello FIM module added at base address 0x16000.
Offset Feature CSR set 0x00000 FME AFU 0x01000 Thermal Management 0x03000 Global Performance 0x04000 Global Error 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 HSSI Interface 0x15000 EMIF Interface 0x16000 Hello FIM 0x80000 PMCI Controller 0x100000 ST2MM (Streaming to Memory-Mapped) 0x130000 PR Control & Status (Port Gasket) 0x131000 Port CSRs (Port Gasket) 0x132000 User Clock (Port Gasket) 0x133000 Remote SignalTap (Port Gasket) 0x140000 AFU Errors (AFU Interface Handler) Refer to the FIM Technical Reference Manual: Interconnect Fabric for more information on the default MMIO region.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4113-files-to-edit-to-support-hello-fim","title":"4.1.1.3 Files to Edit to Support Hello FIM","text":"The table below shows all files in $OFS_ROOTDIR that will be modified or created in order to implement the Hello FIM module. The build_top.sh script copies files from $OFS_ROOTDIR into the target work directory and then the copied files are used in the Quartus build process. Details on editing these files is given in subsequent sections.
Category Status Path File Description Source Modify src/top top.sv Top RTL Modify src/pd_qsys bpf_top.sv BPF top RTL New src/hello_fim hello_fim_top.sv Hello FIM top RTL New src/hello_fim hello_fim_com.sv Hello FIM common RTL Design Files Modify syn/syn_top ofs_top.qsf Quartus setting file Modify syn/syn_top ofs_top_sources.tcl Define sources for top level design New syn/setup hello_fim_design_files.tcl Define RTLs of Hello FIM Modify syn/setup fabric_design_files.tcl Define IPs for fabric Platform Designer Modify src/pd_qsys/fabric bpf.txt Text definition of BPF interconnect Modify src/pd_qsys/fabric bpf.qsys BPF Qsys file Simulation Modify sim/unit_test/soc_tests/dfh_walker test_csr_defs.sv Define CSRs for simulation Verification Modify /ofs-common/verification/fpga_family/agilex/tests/sequences mmio_seq.svh MMIO testbench Modify /ofs-common/verification/fpga_family/agilex/tests/sequences dfh_walking_seq.svh DFH Walking testbench Modify ofs-common/verification/fpga_family/agilex/scripts Makefile_VCS.mk Makefile for VCS"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the SoC.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify syn/syn_top/ofs_top.qsf
-
Define the INCLUDE_HELLO_FIM Verilog macro to the Verilog Macros section. This will enable instantiation of the Hello FIM module. If this is not set, a dummy register will be instantiated instead.
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify syn/syn_top/ofs_top_sources.tcl
-
Add hello_fim_design_files.tcl to the list of subsystems in the Design Files section.
############################################\n# Design Files\n############################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE ../setup/hello_fim_design_files.tcl\n
-
Create syn/setup/hello_fim_design_files.tcl
-
Create hello_fim_design_files.tcl with the following contents:
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify /src/pd_qsys/fabric/fabric_design_files.tcl
-
Add bpf_hello_fim_slv.ip to the list of files in the BPF section.
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE ../ip_lib/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify src/top/top.sv
-
Add bpf_hello_fim_slv_if to AXI4-Lite Interfaces:
// AXI4-lite interfaces\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(12), .ARADDR_WIDTH(12)) bpf_hello_fim_slv_if();\n
-
Modify the value of NEXT_DFH_OFFSET of mem_ss_top from 24'h6B000 to 24'h1000
//*******************************\n// Memory Subsystem\n//*******************************\n`ifdef INCLUDE_DDR4\nmem_ss_top #(\n.FEAT_ID (12'h009),\n.FEAT_VER (4'h1),\n.NEXT_DFH_OFFSET (24'h1000),\n.END_OF_LIST (1'b0)\n) mem_ss_top (\n
-
Modify the value of NEXT_DFH_OFFSET of the Memory Subsystem dummy_csr from 24'h6B000 to 24'h1000
// Placeholder logic if no mem_ss\ndummy_csr #(\n.FEAT_ID (12'h009),\n.FEAT_VER (4'h1),\n.NEXT_DFH_OFFSET (24'h1000),\n.END_OF_LIST (1'b0)\n) emif_dummy_csr (\n
-
Add Hello FIM instance and dummy CSR after the Memory Subsystem. Set the NEXT_DFH_OFFSET to 24'h6A000 for both.
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (12),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (24'h6A000),\n.END_OF_LIST (1'b0)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if) );\n`else\ndummy_csr #( .FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (24'h6A000),\n.END_OF_LIST (1'b0)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif
-
Modify /src/pd_qsys/bpf_top.sv
-
Add bpf_hello_fim_slv_if to the interface descriptions
...\nmodule bpf_top (\n...\n//BPF funtions\n...\nofs_fim_axi_lite_if.master bpf_hello_fim_slv_if\n
-
Add bpf_hello_fim_slv_if to the module
module bpf_top (\n...\n);\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\nendmodule\n
-
Create src/hello_fim
-
Create src/hello_fim directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create src/hello_fim/hello_fim_top.sv
-
Create hello_fim_top.sv with the following contents:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : Nov 2021\n// Module Name : hello_fim_top.sv\n// Project : IOFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create src/hello_fim/hello_fim_com.sv
-
Create hello_fim_com.sv with the following contents:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Modify src/pd_qsys/fabric/bpf.txt
-
Add hello_fim as a slave in the BPF, and enable the SoC as a master for it.
#### - '#' means comment\n# NAME TYPE BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 21 fme,soc_pcie,hssi,qsfp0,qsfp1,emif,pmci,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute the helper script to re-generate the BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\nsh gen_fabrics.sh\n
-
After the shell script finishes, you can find the generated bpf_hello_fim_slv.ip file in $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/. This is the ip variant of the axi4lite shim that bridges the Hello FIM module with the BPF. The updated bpf.qsys file is located in $OFS_ROOTDIR/src/pd_qsys/fabric. You can view the updated bpf file in Platform designer as follows.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\nqsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/syn_top/ofs_top.qpf\n
The image below shows the BPF that integrates the bpf_hello_fim_slv axi4lite shim at address 0x16000, generated through the helper script gen_fabrics.sh.
-
Compile the Hello FIM design:
-
Return to the OFS root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_hello_fim\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker/test_csr_defs.sv
-
Add a HELLO_FIM_IDX entry to the t_dfh_idx enumeration:
typedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX,\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n
-
Add an entry for HELLO_FIM_IDX into the get_dfh_names() function:
function automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\ndfh_name[MAX_DFH_IDX-1:0] dfh_names;\ndfh_names[FME_DFH_IDX] = \"FME_DFH\";\ndfh_names[THERM_MNGM_DFH_IDX] = \"THERM_MNGM_DFH\";\ndfh_names[GLBL_PERF_DFH_IDX] = \"GLBL_PERF_DFH\";\ndfh_names[GLBL_ERROR_DFH_IDX] = \"GLBL_ERROR_DFH\";\ndfh_names[QSFP0_DFH_IDX] = \"QSFP0_DFH\";\ndfh_names[QSFP1_DFH_IDX] = \"QSFP1_DFH\";\ndfh_names[HSSI_DFH_IDX] = \"HSSI_DFH\";\ndfh_names[EMIF_DFH_IDX] = \"EMIF_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\";\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\ndfh_names[PG_PR_DFH_IDX] = \"PG_PR_DFH\";\ndfh_names[PG_PORT_DFH_IDX] = \"PG_PORT_DFH\";\ndfh_names[PG_USER_CLK_DFH_IDX] = \"PG_USER_CLK_DFH\";\ndfh_names[PG_REMOTE_STP_DFH_IDX] = \"PG_REMOTE_STP_DFH\";\ndfh_names[AFU_ERR_DFH_IDX] = \"AFU_ERR_DFH\";\nreturn dfh_names;\nendfunction\n
-
Modify the expected DFH value of the EMIF from 64'h3_00000_06B000_1009 to 64'h3_00000_001000_1009 and add the expected value for HELLO_FIM as 64'h3_00000_06A000_0100:
function automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\nlogic[MAX_DFH_IDX-1:0][63:0] dfh_values;\ndfh_values[FME_DFH_IDX] = 64'h4000_0000_1000_0000;\ndfh_values[THERM_MNGM_DFH_IDX] = 64'h3_00000_002000_0001;\ndfh_values[GLBL_PERF_DFH_IDX] = 64'h3_00000_001000_0007;\ndfh_values[GLBL_ERROR_DFH_IDX] = 64'h3_00000_00e000_1004;\ndfh_values[QSFP0_DFH_IDX] = 64'h3_00000_001000_0013;\ndfh_values[QSFP1_DFH_IDX] = 64'h3_00000_001000_0013;\ndfh_values[HSSI_DFH_IDX] = 64'h3_00000_001000_100f;\ndfh_values[EMIF_DFH_IDX] = 64'h3_00000_001000_1009;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_06A000_0100;\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_080000_1012;\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_030000_0014;\ndfh_values[PG_PR_DFH_IDX] = 64'h3_00000_001000_1005;\ndfh_values[PG_PORT_DFH_IDX] = 64'h4_00000_001000_1001;\ndfh_values[PG_USER_CLK_DFH_IDX] = 64'h3_00000_001000_1014;\ndfh_values[PG_REMOTE_STP_DFH_IDX] = 64'h3_00000_00d000_2013;\ndfh_values[AFU_ERR_DFH_IDX] = 64'h3_00001_000000_2010;\nreturn dfh_values;\nendfunction\n
-
Regenerate the simulation files
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\nsh gen_sim_files.sh f2000x
-
Run the DFH Walker test
cd $OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker\nsh run_sim.sh\n
-
Check the output for the presence of the HELLO_FiM module at address 0x16000:
********************************************\nRunning TEST(0) : test_dfh_walking\n********************************************\n\n...\n\nREAD64: address=0x00015000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000010001009\n\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nREAD64: address=0x00016000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x30000006a0000100\n\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000006a0000100)\nREAD64: address=0x00080000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000800001012\n\nPMCI_DFH\n Address (0x80000)\nDFH value (0x3000000800001012)\n...\n\nTest status: OK\n\n********************\nTest summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#414-walkthrough-modify-and-run-uvm-tests-for-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Modify and run UVM tests for a FIM that has a new module","text":"Perform the following steps to modify the UVM simulation files to support a design that has a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Modify $OFS_ROOTDIR/verification/tests/sequences/dfh_walking_seq.svh
-
Modify the dfh_offset_array to insert the Hello FIM.
dfh_offset_array = new[16];\ndfh_offset_array[ 0] = tb_cfg0.PF0_BAR0; // FME_DFH 0x8000_0000\ndfh_offset_array[ 1] = dfh_offset_array[ 0] + 64'h0_1000; // THERM_MNGM_DFH 0x8000_1000\ndfh_offset_array[ 2] = dfh_offset_array[ 1] + 64'h0_2000; // GLBL_PERF_DFH 0x8000_3000\ndfh_offset_array[ 3] = dfh_offset_array[ 2] + 64'h0_1000; // GLBL_ERROR_DFH 0x8000_4000\ndfh_offset_array[ 4] = dfh_offset_array[ 3] + 64'h0_E000; // QSFP0_DFH 0x8001_2000\ndfh_offset_array[ 5] = dfh_offset_array[ 4] + 64'h0_1000; // QSFP1_DFH 0x8001_3000\ndfh_offset_array[ 6] = dfh_offset_array[ 5] + 64'h0_1000; // HSSI_DFH 0x8001_4000\ndfh_offset_array[ 7] = dfh_offset_array[ 6] + 64'h0_1000; // EMIF_DFH 0x8001_5000\ndfh_offset_array[ 8] = dfh_offset_array[ 7] + 64'h0_1000; // HELLO_FIM_DFH 0x8001_6000\ndfh_offset_array[ 9] = dfh_offset_array[ 8] + 64'h6_a000; // PMCI_DFH 0x8008_0000\ndfh_offset_array[ 10] = dfh_offset_array[ 9] + 64'h8_0000; // ST2MM_DFH 0x8010_0000\ndfh_offset_array[ 11] = dfh_offset_array[10] + 64'h3_0000; // PG_PR_DFH_IDX 0x8013_0000\ndfh_offset_array[ 12] = dfh_offset_array[11] + 64'h0_1000; // PG_PORT_DFH_IDX 0x8013_1000\ndfh_offset_array[ 13] = dfh_offset_array[12] + 64'h0_1000; // PG_USER_CLK_DFH_IDX 0x8013_2000\ndfh_offset_array[ 14] = dfh_offset_array[13] + 64'h0_1000; // PG_REMOTE_STP_DFH_IDX 0x8013_3000\ndfh_offset_array[ 15] = dfh_offset_array[14] + 64'h0_D000; // PG_AFU_ERR_DFH_IDX 0x8014_0000\n
-
Modify $OFS_ROOTDIR/verification/tests/sequences/mmio_seq.svh
-
Add test code related to the Hello FIM. This code will verify the scratchpad register at 0x16030 and read only the register at 0x16038.
// HELLO_FIM_Scratchpad 64 bit access\n`uvm_info(get_name(), $psprintf(\"////Accessing PF0 HELLO_FIM_Scratchpad Register %0h+'h16030////\", tb_cfg0.PF0_BAR0), UVM_LOW)\nassert(std::randomize(wdata));\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h30;\nmmio_write64(.addr_(addr), .data_(wdata));\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h38;\nwdata = 64'h6626_0701_5000_0034;\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\n
Note: uvm_info and uvm_error statements will put a message into log file.
-
Modify $OFS_ROOTDIR/verification/scripts/Makefile_VCS.mk
-
Add INCLUDE_HELLO_FIM define option to enable Hello FIM on UVM
VLOG_OPT += +define+INCLUDE_HELLO_FIM\n
-
Re-generate the UVM files
-
Navigate to the verification scripts directory
cd $VERDIR/scripts\n
-
Clean the output of previous builds
gmake -f Makefile_VCS.mk clean\n
-
Compile the IP files
gmake -f Makefile_VCS.mk cmplib_adp\n
-
Build the RTL and Test Benches
gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1
Note: Using the DEBUG option will provide more detail in the log file for the simulation.
-
Run the UVM DFH Walker simulation
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk run TESTNAME=dfh_walking_test DUMP=1 DEBUG=1\n
Note: Using the DEBUG option will provide more detail in the log file for the simulation.
-
Verify the DFH Walker test results
-
The output logs are stored in the $VERDIR/sim/dfh_walking_test directory. The main files to note are described in the table below:
File Name Description runsim.log A log file of UVM trans.log A log file of transactions on PCIe bus inter.vpd A waveform for VCS -
Run the following command to quickly verify- that the Hello FIM module was successfully accessed. In the example below, the message DFH offset Match! Exp = 80016000 Act = 80016000 shows that the Hello FIM module was successfully accessed.
cd $VERDIR/sim/dfh_walking_test\ncat runsim.log | grep \"DFH offset\"\n
Expected output:
UVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 111950000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp = 80000000 Act = 80000000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 112586000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80001000 Act = 80001000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113222000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80003000 Act = 80003000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113858000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80004000 Act = 80004000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 114494000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80012000 Act = 80012000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115147000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80013000 Act = 80013000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115801000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80014000 Act = 80014000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 116628000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80015000 Act = 80015000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117283000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80016000 Act = 80016000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117928000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80080000 Act = 80080000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 118594000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80100000 Act = 80100000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119248000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80130000 Act = 80130000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119854000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80131000 Act = 80131000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 120460000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80132000 Act = 80132000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121065000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80133000 Act = 80133000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121672000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80140000 Act = 80140000\n
-
Run UVM MMIO Simulation
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1\n
-
Verify the MMIO test results. Run the following commands to show the result of the scratchpad register and Hello FIM ID register. You can see the \"Data match\" message indicating that the registers are successfuly verified.
cd $VERDIR/sim/mmio_test\ncat runsim.log | grep \"Data\" | grep 1603\n
Expected output:
UVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/mmio_seq.svh(68) @ 115466000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016030, data = 880312f9558c00e1\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/mmio_seq.svh(76) @ 116112000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016038, data = 6626070150000034\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#415-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.5 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
Start in your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Run the fpgainfo fme command to determine the PCIe B:D.F of your board. In this example, the PCIe B:D.F is 15:00.0.
Example Output:
Intel IPU Platform f2000x\nBoard Management Controller NIOS FW version: 1.2.4 Board Management Controller Build version: 1.2.4 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : fb25ff1d-c31a-55d8-81d8-cbcedcfcea17\nBoot Page : user1\nUser1 Image Info : 81d8cbcedcfcea17fb25ff1dc31a55d8\nUser2 Image Info : a566ceacaed810d43c60b0b8a7145591\nFactory Image Info : None\n
-
Check that the driver software on 0000:15:00.0 is dfl-pci.
opae.io ls\n
Example output:
[0000:15:00.0] (0x8086:0xbcce 0x8086:0x17d4) Intel IPU Platform f2000x (Driver: dfl-pci)\n
-
Initialize the opae.io tool
opae.io init -d 15:00.0\n
-
Confirm the driver software on 0000:15:00.0 has been updated to vfio-pci.
opae.io ls\n
Example Output:
[0000:15:00.0] (0x8086:0xbcce 0x8086:0x17d4) Intel IPU Platform f2000x (Driver: vfio-pci)\n
-
Run the DFH Walker test to verify there is a module at offset 0x16000
opae.io walk -d 15:00.0\n
Example output:
offset: 0x0000, value: 0x4000000010000000\n dfh: id = 0x0, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x1000, value: 0x3000000020000001\n dfh: id = 0x1, rev = 0x0, next = 0x2000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x3000, value: 0x3000000010000007\n dfh: id = 0x7, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x4000, value: 0x30000000e0001004\n dfh: id = 0x4, rev = 0x1, next = 0xe000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x12000, value: 0x3000000010000013\n dfh: id = 0x13, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x13000, value: 0x3000000010000013\n dfh: id = 0x13, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x14000, value: 0x3000000010002015\n dfh: id = 0x15, rev = 0x2, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000006a0000100\n dfh: id = 0x100, rev = 0x0, next = 0x6a000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x80000, value: 0x3000000800002012\n dfh: id = 0x12, rev = 0x2, next = 0x80000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x100000, value: 0x3000000300000014\n dfh: id = 0x14, rev = 0x0, next = 0x30000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x130000, value: 0x3000000010001005\n dfh: id = 0x5, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x131000, value: 0x4000000010001001\n dfh: id = 0x1, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x132000, value: 0x3000000010001014\n dfh: id = 0x14, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x133000, value: 0x30000000d0002013\n dfh: id = 0x13, rev = 0x2, next = 0xd000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x140000, value: 0x3000010000002010\n dfh: id = 0x10, rev = 0x2, next = 0x0, eol = 0x1, reserved = 0x0, feature_type = 0x3\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d 15:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d 15:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d 0000:15:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d 15:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d 15:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:15:00.0] (0x8086:0xbcce 0x8086:0x17d4) Intel IPU Platform f2000x (Driver: dfl-pci)\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#416-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.6 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
For more detailed information on Signal Tap please see refer to Quartus Prime Pro Edition User Guide: Debug Tools (RDC Document ID 683819).
Signal Tap uses the FPGA Download Cable II USB device to provide access. Please see Intel FPGA Download Cable II for more information. This device is widely available via distributors for purchase.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Steps:
-
The design must be synthesized before adding Signal Tap.
-
If you are using the previously built Hello FIM design, copy the work directory and rename it so that we have a work directory dedicated to the Hello FIM Signal Tap design.
cp -r $OFS_ROOTDIR/work_hello_fim $OFS_ROOTDIR/work_hello_fim_with_stp\n
-
If you are adding signal tap to a new design that has not yet been synthesized, perform the following steps to synthesize the design.
-
Set the environment variables as described in the Setting Up Required Environment Variables section.
-
Run the build script with the -e option to synthesize the design.
./ofs-common/scripts/common/syn/build_top.sh -e --ofss tools/ofss_config/f2000x.ofss f2000x work_hello_fim_with_stp\n
-
Open the Hello FIM Signal Tap project in the Quartus Prime Pro GUI. The Quartus Prime Pro project is named ofs_top.qpf and is located in the work directory $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top/ofs_top.qpf.
-
Select Tools > Signal Tap Logic Analyzer to open the Signal Tap GUI.
-
Accept the \"Default\" selection and click \"Create\".
-
This opens the Signal Tap Logic Analyzer window.
-
Set up the clock for the STP instance. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Prime Pro Project Navigator to find the block of interest and open the design instance for review. For example, see the image below using Project Navigator to open the top module where hello_fim_top_inst is instantiated.
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note, that the clock selected should be associated with the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. Ensure that all signals that are to be sampled are on the same clock domain as the clock you select here. In the middle right of the Signal Tap window, under Signal Configuration, Clock:, select \"\u2026\" as shown below:
-
In the Node Finder tool that popped up, input \"hello_fim_top_inst|clk\" into the \"Named:\" textbox and click \"Search\". Select \"clk\" in the Matching Nodes list and click the \">\" button to select this clock as shown below. Click \"OK\" to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM. Select the signals that appear from the search patterns hello_fim_top_inst|reset and hello_fim_top_inst|csr_lite_if*. Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, \"STP_For_Hello_FIM\".
-
Save the newly created Signal Tap file, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will aurtomatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE STP_For_Hello_FIM.stp\nset_global_assignment -name SIGNALTAP_FILE STP_For_Hello_FIM.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
ofs-common/scripts/common/syn/build_top.sh -p -k f2000x work_hello_fim_with_stp\n
Alternatively, you can copy the ofs_top.qsf and STP_For_Hello_FIM.stp files from the Hello FIM with STP work directory to replace the original files in the cloned OFS repository. In this scenario, all further FIM compilation projects will include the Signal Tap instance integrated into the design. Execute the following commands for this alternative flow:
Copy the modified file \"work_hello_fim_with_stp/syn/syn_top/ofs_top.qsf\" over to the source OFS repository, into \"syn/syn_top/\".
cd $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top\ncp ofs_top.qsf $OFS_ROOTDIR/syn/syn_top\ncp STP_For_Hello_FIM.stp $OFS_ROOTDIR/syn/syn_top\n
Compile the FIM using the files from the OFS repository to create a new work directory.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh -p f2000x work_hello_fim_with_stp_from_src_repo\n
-
Ensure that the compile completes successfully and meets timing:
***********************************\n***\n*** OFS_PROJECT: f2000x\n*** OFS_FIM: base\n*** OFS_BOARD: adp\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 0\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the f2000x. Refer to Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the f2000x via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top/output_files/ofs_top.sof image to the f2000x FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open Quartus Signal Tap GUI.
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap GUI, open your STP file. Your STP file settings will load. In this example we used STP_For_Hello_FIM.stp.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable USB-BlasterII. In the Device: selection box select the Agilex\u00ae 7 FPGA device.
-
If the Agilex\u00ae 7 FPGA is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
If not already set, you can create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'STP_For_Hello_FIM' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, go back to the terminal and walk the DFH list or peek/poke the Hello FIM registers as was done during the creation of the Hello FIM design example.
opae.io init -d 0000:15:00.0\nopae.io walk -d 0000:15:00.0\nopae.io release -d 0000:15:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
-
The PCIe AER feature is automatically re-enabled by rebooting the server.
This concludes the example on how to instrument an OFS FIM with the Quartus Prime Signal Tap Logic Analyzer.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" block during compile-time. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination, order or all can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#421-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
-
Navigate to the OFS root directory
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the null host exerciser options.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_f2000x\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to reduce the size of the PR region to fit the additional logic into the static region. Similiarly, if you reduce logic in the FIM region, then you can increase the size of the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions is reported in the Logic Lock Region Usage Summary sections of following two files:
$OFS_ROOTDIR/<YOUR_WORK_DIRECTORY>/syn/syn_top/output_files/ofs_top.fit.place.rpt-
$OFS_ROOTDIR/<YOUR_WORK_DIRECTORY>/syn/syn_top/output_files/ofs_top.fit.rpt
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to comfortably hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to customize the resources allocated to the AFU in the PR regions:
-
The $OFS_ROOTDIR/syn/setup/pr_assignments.tcl TCL file defines the Logic Lock Regions in the design, including the PR partition where the AFU is allocated.
The default design uses the the following Logic Lock Regions:
set TOP_MEM_REGION \"X115 Y310 X219 Y344\"\nset BOTTOM_MEM_REGION \"X0 Y0 X294 Y20\"\nset SUBSYSTEM_REGION \"X0 Y0 X60 Y279; X0 Y0 X300 Y39; X261 Y0 X300 Y129;\"\nset AFU_PLACE_REGION \"X61 Y40 X260 Y309; X220 Y130 X294 Y329; X12 Y280 X114 Y329;\"\nset AFU_ROUTE_REGION \"X0 Y0 X294 Y329\"\n
Each region is made up of rectangles defined by the origin (X0,Y0) and the top right corner (X1,Y1).
-
Use Quartus Chip Planner to identify the locations of the resources available within the Agilex\u00ae 7 FPGA for placement and routing your AFU. The image below shows the default floorplan for the f2000x Agilex\u00ae 7 FPGA.
-
Make changes to the $OFS_ROOTDIR/syn/setup/pr_assignments.tcl file based on your findings in Quartus Chip Planner. You can modify the size and location of existing Logic Lock Regions or add new ones and assign them to the AFU PR partition. You will need to modify the coordinates of other regions assigned to the FIM accordingly to prevent overlap.
-
Recompile your FIM and create the PR relocatable build tree using the following commands:
cd $OFS_ROOTDIR ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x <YOUR_WORK_DIRECTORY>\n
-
Analyze the resource utilization report per partition produced after recompiling the design.
-
Make further modifications to the PR regions until the results are satisfactory. Make sure timing constraints are met.
Refer to the following documentation for more information on how to optimize the floor plan of your Partial Reconfiguration design:
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3: Floorplan the Design
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe sub-system IP and PF/VF MUX can be modified either using the OFSS flow or the IP Presets flow. The OFSS flow supports a subset of all available PCIe Sub-system settings, while the IP Preset flow can make any available PCIe Sub-system settings change. With PCIe-SS modifcations related to the PFs and VFs, the PF/VF MUX logic is automatically configured based on the PCIe-SS configuration when using OFSS. The sections below describe each flow.
- PCIe Configuration Using OFSS
- PCIe Sub-System configuration Using IP Presets
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS Agilex\u00ae 7 SoC Attach FIM for the f2000x can support up to 8 PFs and 2000 VFs distributed accross all PFs on both the Host and the SoC.
For reference FIM configurations, you must have at least 1 PF on the Host, and at least 1 PF with 1 VF on the SoC. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: Host PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 0 Max # of VFs 2000 distributed across all PFs Table: SoC PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 1 (on PF0) Max # of VFs 2000 distributed across all PFs New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe configuration registers contains the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00001771 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00001771"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4431-walkthrough-modify-the-pcie-sub-system-and-pfvf-mux-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS","text":"Perform the following steps to modify the PF/VF MUX configuration.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Decide which PCIe PF/VFs require modification. If you are modifying host side PF/VF configuration, you must edit file pcie_host.ofss file found in $OFS_ROOTDIR/tools/pfvf_config_tool. If you want to modify SoC-side PF/VF configuration, edit the pcie_soc.ofss file found in the same location. The the following code shows the default Host OFSS file:
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nbar0_address_width = 21\n[pf1]\n
This default configuration is made up of two physical functions (PF), and neither of them has any virtual functions (VF).
-
In this example, we will modify the Host PCIe configuration. Create a new Host PCIe OFSS file from the existing pcie_host.ofss file.
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_pfvf_mod.ofss\n
-
Modify the new pcie_pfvf_mod.ofss OFSS file with the new PF/VF configuration. An example modification to the OFSS file is shown below. In this example we have changed the configuration to: 6 PFs in total, 4 VFs in PF0, 1 VF in PF2, and 2 VFs on PF3. You can add up to 8 PFs and could conceivably add up to the number of VFs supported by the PCIe IP. Note that more PFs/VFs will use more FPGA resources, which may cause fitter challenges.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nbar0_address_width = 21\nnum_vfs = 4\n[pf1]\n[pf2]\nnum_vfs = 1\n[pf3]\nnum_vfs = 2\n[pf4]\n[pf5]\n
-
Edit the top level OFSS file to use the new PCIe OFSS file pcie_host_pfvf_mod.ofss. In this example, we will edit the f2000x top level OFSS file $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host_pfvf_mod.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n
-
Compile the FIM.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_pfvf_mod\n
-
Copy the resulting .bin user 1 image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the .bin image to the f2000x FPGA. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
From the Host, verify the number of VFs on the PFs. In this example, we defined 4 VFs on PF0 in Step 5.
sudo lspci -vvv -s b1:00.0 | grep VF\n
Example output:
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 6, stride: 1, Device ID: bccf
-
Verify communication with the newly added PFs. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
In the following steps, we will verify the newly added PF5.
-
Initialize the driver on PF5
sudo opae.io init -d b1:00.5\n
-
Read the GUID for the PF5 CSR stub.
sudo opae.io -d b1:00.5 -r 0 peek 0x8\nsudo opae.io -d b1:00.5 -r 0 peek 0x10\n
Example output:
0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n
Note: The PCIe B:D.F associated with your board may be different. Use the fpgainfo fme command to see the PCIe B:D:F for your board.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#444-pcie-sub-system-configuration-using-ip-presets","title":"4.4.4 PCIe Sub-System configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings as a starting point.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4441-walkthrough-modify-pcie-sub-system-and-pfvf-mux-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets","text":"Perform the following steps to use an IP preset file to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID on PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
[OPTIONAL] Run the setup stage of the build script using your desired OFSS configration to create a working directory for the target board. In this example we will target the f2000x.
./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
Open the host PCIe-SS using Quartus Parameter Editor. If you performed Step 3, open the PCIe-SS IP from the work directory; otherwise, open the PCIe-SS IP from the source files.
```bash qsys-edit $OFS_ROOTDIR/work_f2000x/ipss/pcie/qip/pcie_ss.ip
-
Modify the settings as desired. In this example we will change the Revision ID to 0x2. In the IP Parameter Editor, scroll down and expand the PCIe Interfaces Ports Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab and make this change.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, f2000x-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 6.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = f2000x-rev2\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss file to call new OFSS file created in Step 10.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n
-
Compile the FIM.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_f2000x_pcie_mod/syn/syn_top/output_files/ofs_top.sof image to your deployment environmenment for JTAG programming, or copy a bin file (e.g. ofs_top_page1_unsigned_user1.bin) for RSU programming.
Note: OPAE FPGA management commands require recognition of the FPGA PCIe Device ID for control. If there is a problem between OPAE management recognition of FPGA PCIe values, then control of the card will be lost. For this reason, you are strongly encouraged to program the FPGA via JTAG to load the test FPGA image. If there is a problem with the SOF image working with your host software that is updated for the new PCIe settings, then you can load a known good SOF file to recover. Once you sure that both the software and FPGA work properly, you can load the FPGA into FPGA flash using the OPAE command fpgasupdate.
-
Program the image to the f2000x FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step JTAG programming instructions, or the Program the FPGA via RSU Section for step-by-step RSU programming instructions.
-
Use lspci to verify that the PCIe changes have been implemented.
lspci -nvmms b1:00.0\n
Example output:
Slot: b1:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 1771\nPhySlot: 1\nRev: 02\nNUMANode: 1\n
Note: Some changes to software may be required to work with certain new PCIe settings. These changes are described in Software Reference Manual: Open FPGA Stack
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#451-walkthrough-create-a-minimal-fim","title":"4.5.1 Walkthrough: Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM. A minimal FIM is one that has the host exercisers and ethernet subsystem removed. This frees up resources that can be used as desired.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions. To create this minimal FIM, perform the following steps:
-
Edit the Host PCIe OFSS file to use the minimal number of PFs (1).
-
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nbar0_address_width = 21\n
-
Edit the SoC PCIe OFSS file to use the minimal number of PFs (1) and VFs (1).
-
$OFS_ROOTDIR/tools/ofss_config/pcie/soc_host.ofss
[ip]\ntype = pcie\n\n[settings]\noutput_name = soc_pcie_ss\n\n[pf0]\nnum_vfs = 1\nbar0_address_width = 21\nvf_bar0_address_width = 21
-
Run the build script with exercisers and ethernet subsystem (HSSI) removed.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_f2000x_minimal_fim\n
-
The build will complete with reduced resources as compared to the base version. You may review the floorplan in Quartus Chip Planner and modify the Logic Lock regions to allocate more resources to the PR region if desired. Refer to the How to Resize the Partial Reconfiguration Region section for information regarding modifications to the floorplan.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#46-migrate-to-a-different-agilex-device-number","title":"4.6 Migrate to a Different Agilex Device Number","text":"The following instructions enable you to change the Agilex 7 FPGA device part number of the f2000x, for example, to migrate to a device with a larger density. Be aware that this release works with Agilex\u00ae 7 FPGAs that have P-tile for PCIe and E-tile for Ethernet.
The default device for the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL is AGFC023R25A2E2VR0
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"This walkthrough describes how to change the device to a larger density with the same package. In this example, we will change the device from part AGFC023R25A2E2VR0 to part AGFA027R25A2E2VR0.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the design repository. See the Clone the OFS Git Repo section.
-
Set the environment variables as described in the Setting Up Required Environment Variables section.
-
Navigate to the OFS Root Directory
cd $OFS_ROOTDIR\n
-
Use the following command to change the device part number throughout the OFS Root directory heirarchy, replacing <DEFAULT_OPN> and <NEW_OPN> with the part numbers specific to your update:
grep -rli '<DEFAULT_OPN>' * | xargs -i@ sed -i 's/<DEFAULT_OPN>/<NEW_OPN>/g' @\n
For example, use the following command to change from part AGFC023R25A2E2VR0 to part AGFA027R25A2E2VR0:
grep -rli 'AGFC023R25A2E2VR0'* | xargs -i@ sed -i 's/AGFC023R25A2E2VR0/AGFA027R25A2E2VR0/g' @\n
This changes all occurrences of the default device (AGFC023R25A2E2VR0) in the $OFS_ROOTDIR directory to the new device number (AGFA027R25A2E2VR0).
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/f2000x_base.ofss file to use the new part number; in this example AGFA027R25A2E2VR0. This is only necessary if you are using the OFSS flow.
[ip]\ntype = ofs\n\n[settings]\nplatform = f2000x\nfim = base_x16\nfamily = agilex\npart = AGFA027R25A2E2VR0\ndevice_id = 6100\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/syn_top/ofs_top.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFA027R25A2E2VR0\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/syn_top/ofs_pr_afu.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFA027R25A2E2VR0\n
-
Modify the DEVICE field in te $OFS_ROOTDIR/ipss/pmci/pmci_ss.qsf file.
set_global_assignment -name DEVICE AGFA027R25A2E2VR0\n
-
Compile the flat (non-PR) design to verify the compilation is successful with the new part. The flat design is compiled without any Logic Lock constraints.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat <YOUR_WORK_DIRECTORY>\n
-
To enable the PR region, use Quartus Chip Planner to analyze the compiled flat design and adjust the Logic Lock constraints defined in $OFS_ROOTDIR/syn/setup/pr_assignments.tcl for the new device layout. Refer to the How to Resize the Partial Reconfiguration Region section for instructions. Re-compile the design with the out-of-tree PR region enabled.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x <YOUR_WORK_DIRECTORY>\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM. This section provides an example walkthrough for modifiying the Memory-SS.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#471-walkthrough-modify-the-memory-sub-system-using-ip-presets-with-ofss","title":"4.7.1 Walkthrough: Modify the Memory Sub-System Using IP Presets With OFSS","text":"In this example we will modify the Memory Subsystem to enable ECC on all of the existing memory interfaces. You may make different modifications to meet your own design requirements. Perform the following steps to make this change.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Memory Subsystem IP file in Platform Designer to perform the required edits.
qsys-edit $OFS_ROOTDIR/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
The Memory Subsystem IP will open in IP Parameter Editor. Click Dive Into Packaged Subsystem.
-
The Platform Designer mem_ss view opens. All of the EMIFs are shown in the Filter window.
-
Click each EMIF 0 through 3 and perform the following actions.
-
In the Parameters window, click the Memory tab and change the DQ width to 40.
-
In the Parameters window, click the Controller tab. Scroll down and check the box for Enable Error Detection and Correction Logic with ECC.
-
Once Step 6 has been done for each EMIF 0-3, click File -> Save. Close the Platform Designer window.
-
In the IP Parameter Editor Presets window, click New to create an IP Presets file.
-
In the New Preset window, set the Name for the preset. In this case we will name it f2000x-ecc.
-
Click the ... button to select the location for the Preset file.
-
In the Save As window, change the save location to $OFS_ROOTDIR/ipss/mem/qip/presets and change the File Name to f2000x-ecc.qprs. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close the IP Parameter Editor. You do not need to generate or save the IP.
-
Edit the $OFS_ROOTDIR/syn/setup/emif_loc.tcl file to assign the pins required for ECC enabled interfaces.
-
Uncomment the DQS4 (ECC) pin assignments for all memory interfaces
#---------------------------------------------------------\n# EMIF CH0\n#---------------------------------------------------------\n...\n# # CH0 DQS4 (ECC)\nset_location_assignment PIN_A39 -to ddr4_mem[0].dq[32]\nset_location_assignment PIN_J35 -to ddr4_mem[0].dq[33]\nset_location_assignment PIN_C38 -to ddr4_mem[0].dq[34]\nset_location_assignment PIN_G34 -to ddr4_mem[0].dq[35]\nset_location_assignment PIN_G38 -to ddr4_mem[0].dq[36]\nset_location_assignment PIN_C34 -to ddr4_mem[0].dq[37]\nset_location_assignment PIN_J39 -to ddr4_mem[0].dq[38]\nset_location_assignment PIN_A35 -to ddr4_mem[0].dq[39]\nset_location_assignment PIN_C36 -to ddr4_mem[0].dqs[4]\nset_location_assignment PIN_A37 -to ddr4_mem[0].dqs_n[4]\nset_location_assignment PIN_G36 -to ddr4_mem[0].dbi_n[4]\n#---------------------------------------------------------\n# EMIF CH1\n#---------------------------------------------------------\n...\n# # CH1 DQS4 (ECC)\nset_location_assignment PIN_N7 -to ddr4_mem[1].dq[32]\nset_location_assignment PIN_L12 -to ddr4_mem[1].dq[33]\nset_location_assignment PIN_L6 -to ddr4_mem[1].dq[34]\nset_location_assignment PIN_U14 -to ddr4_mem[1].dq[35]\nset_location_assignment PIN_U7 -to ddr4_mem[1].dq[36]\nset_location_assignment PIN_W12 -to ddr4_mem[1].dq[37]\nset_location_assignment PIN_W6 -to ddr4_mem[1].dq[38]\nset_location_assignment PIN_N14 -to ddr4_mem[1].dq[39]\nset_location_assignment PIN_L9 -to ddr4_mem[1].dqs[4]\nset_location_assignment PIN_N10 -to ddr4_mem[1].dqs_n[4]\nset_location_assignment PIN_W9 -to ddr4_mem[1].dbi_n[4]\n#---------------------------------------------------------\n# EMIF CH2\n#---------------------------------------------------------\n...\n# CH2 DQS4 (ECC)\nset_location_assignment PIN_GC37 -to ddr4_mem[2].dq[32]\nset_location_assignment PIN_GC41 -to ddr4_mem[2].dq[33]\nset_location_assignment PIN_FY37 -to ddr4_mem[2].dq[34]\nset_location_assignment PIN_GE40 -to ddr4_mem[2].dq[35]\nset_location_assignment PIN_FV36 -to ddr4_mem[2].dq[36]\nset_location_assignment PIN_FY41 -to ddr4_mem[2].dq[37]\nset_location_assignment PIN_GE36 -to ddr4_mem[2].dq[38]\nset_location_assignment PIN_FV40 -to ddr4_mem[2].dq[39]\nset_location_assignment PIN_GE38 -to ddr4_mem[2].dqs[4]\nset_location_assignment PIN_GC39 -to ddr4_mem[2].dqs_n[4]\nset_location_assignment PIN_FV38 -to ddr4_mem[2].dbi_n[4]\n#---------------------------------------------------------\n# EMIF CH3\n#---------------------------------------------------------\n...\n# # CH3 DQS4 (ECC)\nset_location_assignment PIN_FP46 -to ddr4_mem[3].dq[32]\nset_location_assignment PIN_FT43 -to ddr4_mem[3].dq[33]\nset_location_assignment PIN_FH47 -to ddr4_mem[3].dq[34]\nset_location_assignment PIN_FP42 -to ddr4_mem[3].dq[35]\nset_location_assignment PIN_FT47 -to ddr4_mem[3].dq[36]\nset_location_assignment PIN_FH43 -to ddr4_mem[3].dq[37]\nset_location_assignment PIN_FK46 -to ddr4_mem[3].dq[38]\nset_location_assignment PIN_FK42 -to ddr4_mem[3].dq[39]\nset_location_assignment PIN_FP44 -to ddr4_mem[3].dqs[4]\nset_location_assignment PIN_FT45 -to ddr4_mem[3].dqs_n[4]\nset_location_assignment PIN_FK44 -to ddr4_mem[3].dbi_n[4]\n
-
Change the pin assignment for ddr4_mem[1].dbi_n[4] from PIN_G12 to PIN_W9
set_location_assignment PIN_W9 -to ddr4_mem[1].dbi_n[4]\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/memory/memory.ofss file to use the new presets file generated previously.
[ip]\ntype = memory\n\n[settings]\noutput_name = mem_ss_fm\npreset = f2000x-mem-ecc\n
-
Compile the design with the f2000x OFSS file which calls the Memory OFSS file edited in the previous step.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x <YOUR_WORK_DIRECTORY>\n
-
You may need to adjust the floorplan of the design in order to meet timing after a design change such as this. Refer to the How to Resize the Partial Reconfiguration Region section for information regarding modifications to the floorplan.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System. There are three flows you may use to make modifications.
- Modify the Ethernet Sub-System with OFSS supported changes only. These modifications are supported natively by the build script, and are made at run-time of the build script. This flow is useful for users who only need to leverage natively supported HSSI OFSS settings.
- Modify the Ethernet Sub-System with OFSS supported changes, then make additional custom modifications not covered by OFSS. These modifications will be captured in a presets file which can be used in future compiles. This flow is useful for users who whish to leverage pre-made HSSI OFSS settings, but make additional modifications not natively supported by HSSI OFSS.
- Modify the Ethernet Sub-System without HSSI OFSS. These modification will be made directly in the source files.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#481-walkthrough-modify-the-ethernet-sub-system-channels-with-pre-made-hssi-ofss","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS","text":"This walkthrough describes how to use OFSS to configure the Ethernet-SS. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This walkthrough is useful for users who only need to leverage the pre-made, natively supported HSSI OFSS settings.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the $OFS_ROTDIR/tools/ofss_config/f2000x.ofss file to use the desired Ethernet-SS OFSS configuration. The pre-provided OFSS configurations are as follows:
-
To select the 2x4x25GbE configuration, include the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x25.ofss\n
-
To select the 2x4x10GbE configuration, include the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x10.ofss\n
-
To select the 2x1x100GbE configuration, include the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_2x100.ofss\n
-
Compile the FIM using the f2000x OFSS file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
The resulting FIM will contain the Ethernet-SS configuration specified in Step 3. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#482-walkthrough-add-channels-to-the-ethernet-sub-system-channels-with-custom-hssi-ofss","title":"4.8.2 Walkthrough: Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS","text":"This walkthrough describes how to create an use a custom OFSS file to add channels to the Ethernet-SS and compile a design with a 3x4x25GbE Ethernet-SS configuration. This walkthrough is useful for users who wish to leverage the natively supported HSSI OFSS settings.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a new HSSI OFSS file $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_12x25.ofss with the following contents. In this example we are using 12 channels.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 12\ndata_rate = 25GbE\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss file to use the new HSSI OFSS file generated in Step 3.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_12x25.ofss\n
-
Identify the which channels will be added. You may use the E-Tile Channel Placement Tool to aid in your design. In this example we will add the 4 new 25GbE channels to Channels 8-11.
-
Based on your channel selection, identify which pins will be used. Refer to the [Pin-Out Files for Altera FPGAs] determine the required pins for your device. In this example we are targeting the AGFC023R25A2E2VR0 device. Set the pin assignments in the $OFS_ROOTDIR/syn/setup/eth_loc.tcl file.
set_location_assignment PIN_DL8 -to hssi_if[8].rx_p\nset_location_assignment PIN_DN13 -to hssi_if[9].rx_p\nset_location_assignment PIN_DY8 -to hssi_if[10].rx_p\nset_location_assignment PIN_EB13 -to hssi_if[11].rx_p\n\nset_location_assignment PIN_DL1 -to hssi_if[8].tx_p\nset_location_assignment PIN_DN4 -to hssi_if[9].tx_p\nset_location_assignment PIN_DY1 -to hssi_if[10].tx_p\nset_location_assignment PIN_EB4 -to hssi_if[11].tx_p\n
-
Change the number of QSFP ports from 2 to 3 in the $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv file.
localparam NUM_QSFP_PORTS_USED = 3; // Number of QSFP cages on board used by target hssi configuration\n
-
Edit $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/hssi_wrapper.sv so that the QSFP LED signals use NUM_QSFP_PORTS_USED defined in the previous step.
// Speed and activity LEDS\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_green, // Link up in Nx25G or 2x56G or 1x100G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_yellow, // Link up in Nx10G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_green, // Link up and activity seen\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_red // LOS, TX Fault etc\n
-
Compile the design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x_12x25g\n
-
You may need to adjust the floorplan in order to compile with a PR region that meets timing. Refer to the How to Resize the Partial Reconfiguration Region section for information regarding modifications to the floorplan.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#483-walkthrough-modify-the-ethernet-sub-system-with-pre-made-hssi-ofss-plus-additional-modifications","title":"4.8.3 Walkthrough: Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications","text":"This walkthrough describes how to use OFSS to first modify the Ethernet-SS, then make additional modifications on top. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This flow is useful for users who whish to leverage pre-made OFSS settings, but make additional modifications not natively supported by OFSS.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the $OFS_ROTDIR/tools/ofss_config/f2000x.ofss file to use the desired Ethernet-SS OFSS configuration starting point. Examples for using pre-provided HSSI OFSS files are given below.
-
To select 2x4x25GbE configuration, add the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x25.ofss\n
-
To select 2x4x10GbE configuration, add the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x10.ofss\n
-
To select 2x1x100GbE configuration, add the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_2x100.ofss\n
-
Run the setup stage of the build script with the OFSS file to create a work directory which contains the Ethernet-SS IP configuration specified in Step 3.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
Open the Ethernet-SS IP in Quartus Parameter Editor. The IP settings will match te configuration of the OFSS file defined in Step 3. Make any additional modifications in the Parameter Editor as desired.
qsys-edit $OFS_ROOTDIR/work_f2000x/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the New... button in the Presets pane of IP Parameter Editor.
-
In the New Preset window, create a unique Name. In this example the name is f2000x-hssi-presets.
-
Click the ... button to select where to save the preset file. Give it a name, and save it to $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets. Create the presets directory if necessary.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close out of all Quartus GUIs. You do not need to save or compile the IP.
-
Create a new HSSI OFSS file in the $OFS_ROOTDIR/tools/ofss_config/hssi directory named hssi_preset_f2000x.ofss with the following contents. Note that the num_channels and data_rate settings will be overwritten by the contents of the preset file. The preset setting must match the name you selected in Step 7.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 8\ndata_rate = 25GbE\npreset = f2000x-hssi-presets\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss file to use the new HSSI OFSS file created in Step 10.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_preset_f2000x.ofss\n
-
Compile the design using the f2000x OFSS file. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x_hssi_preset\n
-
The resulting FIM will contain the Ethernet-SS configuration specified by the presets file. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#484-walkthrough-modify-the-ethernet-sub-system-without-hssi-ofss","title":"4.8.4 Walkthrough: Modify the Ethernet Sub-System Without HSSI OFSS","text":"This walkthrough describes how to modify the Ethernet-SS wihout using OFSS. This flow will edit the Ethernet-SS IP source directly. This walkthrough is useful for users who wish to make all Ethernet-SS modifications manually, without leveraging HSSI OFSS.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Ethernet-SS IP in Quartus Parameter Editor. Make your modifications in the Parameter Editor.
qsys-edit $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the Generate HDL. Save the design if prompted.
-
Compile the design.
-
If you are not using any other OFSS files in your compilation flow, use the following command to compile. It is recommended to compile a flat design before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh f2000x:flat work_f2000x\n
- If you are using OFSS files for other IP in the design, ensure that the top level OFSS file (e.g.
$OFS_ROOTDIR/tools/ofss_config/f2000x.ofss) does not specify an HSSI OFSS file. Then use the following command to compile. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x\n
-
The resulting FIM will contain the Ethernet-SS configuration contained in the hssi_ss.ip source file.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the f2000x can be done by Remote System Update (RSU) using OPAE commands, or by programming a SOF image to the FPGA via JTAG using Quartus Programer.
Programming via RSU will program the flash device on the board for non-volatile image updates. Programming via JTAG will configure the FPGA for volatile image updates.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"The f2000x has a 10 pin JTAG header on the top side of the board. This JTAG header provides access to either the Agilex 7 FPGA or Cyclone\u00ae 10 BMC device. A JTAG connection with the FPGA Download Cable II can be used to configure the FPGA and to access the Signal Tap Instance. This walkthrough describes how to connect the FPGA Download Cable II and target the Agilex 7 device.
Pre-requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 23.4 tools installed, specifically the
jtagconfig tool.
- This walkthrough requires an Intel FPGA Download Cable II.
Steps:
-
Locate SW2 and SW3 on the f2000x board as shown in the following figure.
-
Set the switches described in the following table:
Switch Position SW2 ON SW3.3 ON -
Connect the FPGA Download Cable to the JTAG header of the f2000x as shown in the figure below.
-
Depending on your server, install the card in a slot that allows room for the JTAG cable. The figure below shows the f2000x installed in a Supermicro server slot.
-
There are two JTAG modes that exist. Short-chain mode is when only the Cyclone 10 device is on the JTAG chain. Long-chain mode is when both the Cyclone 10 and the Agilex\u00ae 7 FPGA are on the JTAG chain. Check which JTAG mode the f2000x board is in by running the following command.
$QUARTUS_ROOTDIR/bin/jtagconfig\n
-
Example output when in short-chain mode (only Cyclone 10 detected):
1) USB-BlasterII [3-4]\n020F60DD 10CL080(Y|Z)/EP3C80/EP4CE75\n
-
Example output when in long-chain mode (both Cyclone 10 and Agilex\u00ae 7 FPGA):
1) USB-BlasterII [3-4]\n020F60DD 10CL080(Y|Z)/EP3C80/EP4CE75\n234150DD AGFC023R25A(.|AE|R0)\n
If the Agilex\u00ae 7 FPGA does not appear on the chain, ensure that the switches described in Step 1 have been set correctly and power cycle the board. Also ensure that the JTAG Longchain bit is set to 0 in BMC Register 0x378. The BMC registers are accessed through SPI control registers at addresses 0x8040C and 0x80400 in the PMCI. Use the following commands to clear the JTAG Longchain bit in BMC register 0x378.
Note: These commands must be executed as root user from the SoC.
Note: You may find the PCIe BDF of your card by running fpgainfo fme.
opae.io init -d <BDF>\nopae.io -d <BDF> -r 0 poke 0x8040c 0x000000000\nopae.io -d <BDF> -r 0 poke 0x80400 0x37800000002\nopae.io release -d <BDF>\n
For example, for a board with PCIe BDF 15:00.0: opae.io init -d 15:00.0\nopae.io -d 15:00.0 -r 0 poke 0x8040c 0x000000000\nopae.io -d 15:00.0 -r 0 poke 0x80400 0x37800000002\nopae.io release -d 15:00.0\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"Every successful run of build_top.sh script creates the file $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/syn_top/output_files/ofs_top.sof which can be used with the FPGA Download Cable II to load the image into the FPGA using the f2000x JTAG access connector.
This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the f2000x. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
Temporarily disable the PCIe AER feature and remove the PCIe port for the board you are going to update. This is required because when you program the FPGA using JTAG, the f2000x PCIe link goes down for a moment causing a server surprise link down event. To prevent this server event, temporarily disable PCIe AER and remove the PCIe port using the following commands:
Note: enter the following commands as root.
-
Find the PCIe BDF and Device ID of your board from the SoC. You may use the OPAE command fpaginfo fme on the SoC to display this information. Run this command on the SoC.
fpgainfo fme\n
Example output:
Intel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.2.3\nBoard Management Controller Build version: 1.2.3\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50103024BF5B5B1\nBitstream Version : 5.0.1\nPr Interface Id : e7926956-9b1b-5ea1-b02c-307f1cb33446\nBoot Page : user1\nUser1 Image Info : b02c307f1cb33446e79269569b1b5ea1\nUser2 Image Info : b02c307f1cb33446e79269569b1b5ea1\nFactory Image Info : b02c307f1cb33446e79269569b1b5ea1\n
In this case, the PCIe BDF for the board on the SoC is 15:00.0, and the Device ID is 0xBCCE.
-
From the SoC, use the pci_device OPAE command to \"unplug\" the PCIe port for your board using the PCIe BDF found in Step 3.a. Run this command on the SoC.
pci_device <B:D.F> unplug\n
For example:
pci_device 15:00.0 unplug\n
-
Find the PCIe BDF of your board from the Host. Use lspci and grep for the device ID found in Step 3.a to get the PCIe BDF. Run this command on the Host.
For example:
lspci | grep bcce\n
Example output:
31:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n31:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
In this case, the board has PCIe BDF 31:00.0 from the Host.
-
From the Host, use the pci_device OPAE command to \"unplug\" the PCIe port for your board using the PCIe BDF found in Step 3.c. Run this command on the Host.
pci_device <B:D.F> unplug\n
For example:
pci_device 31:00.0 unplug\n
-
Launch \"Quartus Prime Programmer\" software from the device which the FPGA Programmer is connected.
$QUARTUS_ROOTDIR/bin/quartus_pgmw\n
Click on Hardware Setup, select USB-BlasterII in the Current Selected Hardware list, and ensure the JTAG Hardware Frequency is set to 16Mhz (The default is 24MHz).
Alternatively, use the following command from the command line to change the clock frequency:
jtagconfig \u2013setparam \u201cUSB-BlasterII\u201d JtagClock 16M\n
-
Click Auto Detect and make sure the Agilex\u00ae 7 FPGA is shown in the JTAG chain. Select the Cyclone 10 and Agilex\u00ae 7 FPGA if prompted.
-
Right-click on the cell in the File column for the Agilex\u00ae 7 FPGA and click on Change file
-
Select the .sof file that you wish to configure the Agilex\u00ae 7 FPGA with.
-
Tick the checkbox below \"Program/Configure\" column and click on Start to program this .sof file.
-
After successful programming, you can close the \"Quartus Prime Programmer\" software. You can answer 'No' if a dialog pops up asking to save the 'Chain.cdf' file. This completes the Agilex\u00ae 7 FPGA .sof programming.
-
Re-plug the PCIe ports on both the SoC and Host.
Note: enter the following commands as root.
-
From the SoC, use the pci_device OPAE command to \"plug\" the PCIe port for your board using the PCIe BDF found in Step 3.a. Run this command on the SoC.
pci_device <B:D.F> plug\n
For example:
pci_device 15:00.0 plug\n
-
From the Host, use the pci_device OPAE command to \"plug\" the PCIe port for your board using the PCIe BDF found in Step 3.c. Run this command on the Host.
pci_device <B:D.F> plug\n
For example:
pci_device 31:00.0 plug\n
-
Verify the f2000x is present by checking expected bitstream ID using fpaginfo fme on the SoC:
fpgainfo fme\n
Example output:
Intel IPU Platform f2000x-PL\nBoard Management Controller NIOS FW version: 1.2.4\nBoard Management Controller Build version: 1.2.4\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : cf00eed4-a82b-5f07-be25-0528baec3711\nBoot Page : user1\nUser1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nUser2 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nFactory Image Info : None\n
Note: The Image Info fields will not change, because these represent the images stored in flash. In this example, we are programming the Agilex\u00ae 7 FPGA directly, thus only the Bitstream ID should change.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#53-remote-system-update","title":"5.3 Remote System Update","text":"The OPAE fpgasupdate tool can be used to update the Cyclone 10 Board Management Controller (BMC) image and firmware (FW), root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate tool only accepts images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then you must also sign the image using the correct keys.
The RSU flow requires that an OPAE enabled design is already loaded on the FPGA. All OPAE commands are run from the SoC, and the new image will be updated from the SoC.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#531-walkthrough-program-the-fpga-via-rsu","title":"5.3.1 Walkthrough: Program the FPGA via RSU","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL with a BIN image via RSU.
Pre-Requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough requires a
BIN image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a BIN image. The image used for programming must be formatted with PACsign before programming. This is done automatically by the build script.
Steps:
-
Start in your deployment environment.
-
Use the fpgainfo fme command to obtain the PCIe s:b:d.f associated with your board. In this example, the PCIe s:b:d.f is 0000:15:00.0.
fpgainfo fme\n
Example output:
Intel IPU Platform f2000x\nBoard Management Controller NIOS FW version: 1.2.4 Board Management Controller Build version: 1.2.4 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : fb25ff1d-c31a-55d8-81d8-cbcedcfcea17\nBoot Page : user1\nUser1 Image Info : a566ceacaed810d43c60b0b8a7145591\nUser2 Image Info : a566ceacaed810d43c60b0b8a7145591\nFactory Image Info : a566ceacaed810d43c60b0b8a7145591\n
-
Use the OPAE fpgasupdate command to program a signed image to flash. The flash slot which will be programmed is determined by the signed header of the image.
sudo fpgasupdate <IMAGE> <PCIe B:D.F>\n
-
Example: update User Image 1 in flash
sudo fpgasupdate ofs_top_page1_unsigned_user1.bin 15:00.0\n
-
Example: update User Image 2 in flash
sudo fpgasupdate ofs_top_page2_unsigned_user2.bin 15:00.0\n
-
Example: update Factory Image in flash
sudo fpgasupdate ofs_top_page0_unsigned_factory.bin 15:00.0\n
Note: The label \"unsigned\" in the images produced by the build script mean that the image has been signed with a null header. These images can only be programmed to devices which have not had any keys provisioned.
-
Use the OPAE rsu command to reconfigure the FPGA with the new image. You may select which image to configure from (User 1, User 2, Factory).
sudo rsu fpga --page <PAGE> <PCIe B:D.F>\n
-
Example: configure FPGA with User 1 Image
sudo rsu fpga --page user1 15:00.0\n
-
Example: configure FPGA with User 2 Image
sudo rsu fpga --page user2 15:00.0\n
-
Example: configure FPGA with Factory Image
sudo rsu fpga --page factory 15:00.0\n
-
Run the fpgainfo fme command again to verify the User1 Image Info has been updated.
Example Output:
Intel IPU Platform f2000x\nBoard Management Controller NIOS FW version: 1.2.4 Board Management Controller Build version: 1.2.4 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : fb25ff1d-c31a-55d8-81d8-cbcedcfcea17\nBoot Page : user1\nUser1 Image Info : 81d8cbcedcfcea17fb25ff1dc31a55d8\nUser2 Image Info : a566ceacaed810d43c60b0b8a7145591\nFactory Image Info : None\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#6-single-event-upset-reporting","title":"6 Single Event Upset Reporting","text":"A Single Event Upset (SEU) is the change in state of a storage element inside a device or system. They are caused by ionizing radiation strikes that discharge the charge in storage elements, such as configuration memory cells, user memory and registers.
Error Detection CRC (EDCRC) circuitry in the Card BMC is used to detect SEU errors. The CRC function is enabled in Quartus Prime Pro to enable CRC status to be reported to the FM61 via the dedicated CRC_ERROR pin.
With the EDCRC there is no method to determine the severity of an SEU error i.e. there is not way to distinguish between non-critical and catastrophic errors. Hence once the SEU error is detected, the Host system must initiate the Card BMC reset procedure.
SEU errors can be read from either the Card BMC SEU Status Register or the PMCI Subsystem SEU Error Indication Register. The processes to read these registers are described in greater detail in the BMC User Guide. Contact your Altera representative for access to the BMC User Guide.
Additionally, refer to the Agilex 7 SEU Mitigation User Guide for more information on SEU detection and mitigation.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#appendix","title":"Appendix","text":""},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#appendix-a-fim-fpga-resource-usage","title":"Appendix A: FIM FPGA Resource Usage","text":"The provided design includes both required board management and control functions as well as optional interface exerciser logic that both creates transactions and validates operations. These exerciser modules include:
- HE_MEM - this module creates external memory transactions to the DDR4 memory and then verifies the responses.
- HE_MEM-TG -The memory traffic generator (TG) AFU provides a way for users to characterize local memory channel bandwidth with a variety of traffic configuration features including request burst size, read/write interleave count, address offset, address strobe, and data pattern.
- HE_HSSI - this module creates ethernet transactions to the HSSI Subsystem and then verifies the responses.
The FIM uses a small portion of the available FPGA resources. The table below shows resource usage for a base FIM built with 2 channels of external memory, a small AFU instantiated that has host CSR read/write, external memory test and Ethernet test functionality.
Note: The host exerciser modules allow you to evaluate the FIM in hardware and are removed when you begin development.
The AGFC023R25A2E2VR0 FPGA has the following resources available for base FIM design :
Resource needed / total on device (%) Logic utilization (ALMs) 229,622 / 782,400 ( 29 % ) M20K blocks 1,241 / 10,464 (12 %) Pins 518 / 742 ( 70 % ) IOPLLs 10 / 15 ( 67 % ) The resource usage for the FIM base:
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 229,646.10 29.35 1241 11.86 soc_afu 87,364.80 11.17 273 2.61 soc_pcie_wrapper 37,160.80 4.75 195 1.86 pcie_wrapper 36,233.40 4.63 187 1.79 host_afu 26,462.20 3.38 140 1.34 hssi_wrapper 20,066.30 2.56 173 1.65 pmci_wrapper 8,449.90 1.08 186 1.78 mem_ss_top 7,907.10 1.01 60 0.57 auto_fab_0 2,708.90 0.35 13 0.12 soc_bpf 1,210.20 0.15 0 0.00 qsfp_1 635.50 0.08 4 0.04 qsfp_0 628.70 0.08 4 0.04 fme_top 628.60 0.08 6 0.06 host_soc_rst_bridge 151.40 0.02 0 0.00 rst_ctrl 16.80 0.00 0 0.00 soc_rst_ctrl 16.50 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00 The following example without the he_lb,he_hssi,he_mem,he_mem_tg:
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 162,010.20 20.71 992 9.48 pcie_wrapper 36,771.70 4.70 195 1.86 soc_afu_top 34,851.30 4.45 85 0.81 pcie_wrapper 33,358.90 4.26 175 1.67 hssi_wrapper 20,109.90 2.57 173 1.65 afu_top 14,084.20 1.80 91 0.87 pmci_wrapper 8,447.90 1.08 186 1.78 mem_ss_top 8,379.70 1.07 60 0.57 alt_sld_fab_0 2,725.10 0.35 13 0.12 bpf_top 1,213.00 0.16 0 0.00 fme_top 638.30 0.08 6 0.06 qsfp_top 626.70 0.08 4 0.04 qsfp_top 619.20 0.08 4 0.04 axi_lite_rst_bridge 147.40 0.02 0 0.00 rst_ctrl 17.40 0.00 0 0.00 rst_ctrl 15.90 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00 The following example without the Ethernet Subsystem (no_hssi):
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 189,827.00 24.26 980 9.37 soc_afu_top 67,751.40 8.66 197 1.88 pcie_wrapper 36,909.30 4.72 195 1.86 pcie_wrapper 36,077.70 4.61 187 1.79 afu_top 26,549.40 3.39 140 1.34 pmci_wrapper 8,688.10 1.11 186 1.78 mem_ss_top 8,079.00 1.03 60 0.57 alt_sld_fab_0 1,751.90 0.22 9 0.09 bpf_top 1,186.00 0.15 0 0.00 dummy_csr 664.70 0.08 0 0.00 dummy_csr 662.80 0.08 0 0.00 dummy_csr 661.20 0.08 0 0.00 fme_top 649.40 0.08 6 0.06 axi_lite_rst_bridge 161.70 0.02 0 0.00 rst_ctrl 16.30 0.00 0 0.00 rst_ctrl 16.00 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00 The following example without the Ethernet Subsystem (no_hssi) + no host exercisers (he_lb, he_hssi, he_mem, he_mem_tg):
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 139,105.70 17.78 807 7.71 pcie_wrapper 36,518.80 4.67 195 1.86 pcie_wrapper 33,234.50 4.25 175 1.67 soc_afu_top 32,700.00 4.18 85 0.81 afu_top 14,178.20 1.81 91 0.87 pmci_wrapper 8,693.20 1.11 186 1.78 mem_ss_top 7,999.00 1.02 60 0.57 alt_sld_fab_0 1,758.40 0.22 9 0.09 bpf_top 1,183.50 0.15 0 0.00 dummy_csr 667.20 0.09 0 0.00 dummy_csr 666.30 0.09 0 0.00 dummy_csr 663.10 0.08 0 0.00 fme_top 652.80 0.08 6 0.06 axi_lite_rst_bridge 153.80 0.02 0 0.00 rst_ctrl 18.20 0.00 0 0.00 rst_ctrl 16.50 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/doc_modules/Glossary/","title":"Glossary","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/doc_modules/Notices_%26_Disclaimers/","title":"Notices & Disclaimers","text":""},{"location":"hw/f2000x/doc_modules/Notices_%26_Disclaimers/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for the safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/","title":"Shell Technical Reference Manual for Agilex 7 SoC Attach: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1-overview","title":"1 Overview","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#11-about-this-document","title":"1.1 About this Document","text":"This document describes the hardware architecture for the SoC Attach reference FIM of the Open FPGA Stack (OFS) targeting the Agilex 7 FPGA. After reviewing this document you should understand the features and functions of the components that comprise the FPGA Interface Manager (FIM), also known as the \"shell.\"
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#12-glossary","title":"1.2 Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#13-introduction-to-open-fpga-stack","title":"1.3 Introduction to Open FPGA Stack","text":"The Open FPGA Stack (OFS) is a modular infrastructure of hardware platform components, open source unstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS Provides a framework of FPGA synthesizable code, simulation environment and synthesis/simulation scripts.
The key components of OFS include:
- Target development platforms such as Intel-branded Acceleration Development Platforms (ADPs) and third-party platforms.
- Board Management Controller RTL and firmware that supports telemetry monitoring and capability for remote configuration updates.
- Source accessible, modular FPGA Interface manager (FIM) RTL with a UVM infrastructure unit tests that can be leveraged for your own custom FIM design. The FIM can be thought of as the FPGA shell that provides the I/O ring and timing closed management components for the FPGA.
- Basic building blocks for interconnect and PF/VF translation and arbitration; Platform Interface Manager (PIM) which provides Avalon\u00ae and Arm\u00ae AMBA\u00ae 4 AXI4 bus compliant interfaces.
- AFU examples both in the git repository and workload examples provided by 3rd party vendors.
- Unit level simulation test suite
- System level simulation through a unified verification methodology (UVM)
- OPAE software development kit (APIs, up-streamed Linux drivers and software tools)
- Support for other frameworks to be built on top of the OPAE such as DPDK
These components are available in a two GitHub locations:
- OFS hardware GitHub site
- OPAE software GitHub site
The OFS hardware repository supports hardware development and simulation. Repositories for OFS high-level design support and board management controller RTL and firmware source code are also provided. These repositories can be found in the Opensource Technology GitHub location, which requires entitlement access. To request access, please contact your local Altera sales representative.
Table 1-2 OFS Hardware Repositories
Repository Contains OFS f2000x FIM Github Branch Contains FIM or shell RTL, automated compilation scripts, and unit tests and UVM framework. The OPAE software GitHub site is fully opensource and contains resources for both software and workload developers.
Table 1-3 OFS Software Repositories
OPAE Git Repository Folder Contains OPAE SDK Repo Contains the files for building and installing OPAE SDK from source. Linux DFL Contains OFS Linux drivers that are being upstreamed to the Linux kernel. ofs-platform-afu-bbb Contains the files and scripts to build the platform interface manager. opae-sim Contains the files for an AFU developer to build the Accelerator Functional Unit Simulation Environment (ASE) for workload development. Providing the hardware and software source code and supporting test frameworks in a GitHub repository allows you to customize your designs with the latest versions easily.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14-ofs-features","title":"1.4 OFS Features","text":"The OFS architecture within the FPGA comprises two partitions:
- FPGA Interface Manager (FIM)
- Accelerator Functional Unit (AFU) - AFU SoC Dynamic Region and Static Region - AFU Host Static Region
The FIM or shell provides platform management functionality, clocks, resets, and interface access to the host and peripheral features of the acceleration platform.
The FIM architecture along with the supporting OPAE software supports features such as partial reconfiguration and virtualization.
The FIM provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 datapath interface. The FIM resides in the static region of the FPGA.
The AFU partition is provided for custom acceleration workloads and may contain both static and partial reconfiguration regions.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#141-fpga-interface-manager-fim","title":"1.4.1 FPGA Interface Manager (FIM)","text":"The updated OFS architecture for Agilex\u00ae 7 FPGA devices improves upon the modularity, configurability and scalability of the first release of the OFS architecture while maintaining compatibility with the original design. The primary components of the FPGA Interface Manager or shell of this reference design are:
- P-tile PCIe Subsystem
- E-Tile Ethernet Subsystem
- Memory Subsystem
- Reset Controller
- FPGA Management Engine (FME)
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The AFU Region provides design space for custom workloads and contains both static and partial reconfiguration regions. Partial reconfiguration allows you to update your specific logic blocks or entire workload while the rest of your static design is still in operation.
Note that as discussed previously, the BMC RTL and firmware, the OFS OPAE software stack and support for building your own customer board support package are also provided in separate OFS repositories.
Figure 1-1 OFS for OFS FIM Platform Block Diagram
The table provides an overview of the OFS features targeting the Agilex\u00ae 7 FPGA. This reference FIM (shell) is a starting point for your custom FPGA design. With this initial starting point, you can add or subtract interfaces or ports to different Agilex 7 devices.
Table 1-4 OFS FIM for Agilex\u00ae 7 FPGA Features
Key Feature Description PCIe P-tile PCIe* Gen4x16 to the HostP-tile PCIe* Gen4x16 to the SoC (IceLake-D) Virtualization Host: 2 physical functions SoC: 1 physical function and 3 Virtual functions Memory Four Fabric DDR4 banks, x40 (optional ECC, be configured as x32 and ECC x8 ), 1200 MHz, 4GB Ethernet Eight Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels of 25G Ethernet interfacing to an E-tile Ethernet Subsystem. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) * Platform Controller Management Interface (PMCI) Module contained within the Agilex 7 FPGA that interfaces through Avalon-Streaming x8 QuadSPI and SPI to a Board Management Controller Partial Reconfiguration Partial Reconfiguration region supported in hardware and software Software Support * Linux DFL drivers targeting OFS FIMs * OPAE Software Development Kit * OPAE Tools"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#subsystem-interfaces","title":"Subsystem Interfaces","text":"The PCIe, Memory and Ethernet interfaces in this design use a new flexible subsystem design that provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 interface. To access these FPGA IP Subsystem documents, please go here and search for the following ID numbers: * 690604: PCIe Subsystem IP User Guide (Note: you must login to myIntel and request entitled access) * 686148: Memory Subsystem IP User Guide (Note: you must login to myIntel and request entitled access) * 773413: [Ethernet Subsystem Intel FPGA IP] (public document)
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FIM contains only one FME, regardless of the number of host interfaces to the FIM. The FME provides management features for the platform and controls reset and loading of the AFU into the partial reconfiguration region of the FPGA.
Any feature, such as a memory interface or global error control that you want to control through FME, must expose its capability to host software drivers. New features are exposed to the FME by adding a device feature header (DFH) register at the beginning of the feature's control status register (CSR) space. The FME CSR maps to physical function 0 (PF0) Base address register 0 (BAR0) so that software can access it through a single PCIe link. For more information about DFHs, refer to the FPGA Device Feature List Framework Overview
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#streaming-datapath","title":"Streaming Datapath","text":"The FIM implements an Arm\u00ae AMBA\u00ae 4 AXI4-Stream bus protocol for data transfer in the FIM. Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels send data packets to and from the host channel IP without data abstraction. Memory-mapped I/O (MMIO) CSR accesses are routed to the ST2MM module, which converts the Arm\u00ae AMBA\u00ae 4 AXI4-Stream to an Arm\u00ae AMBA\u00ae 4 AXI4 memory-mapped protocol.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#virtualization","title":"Virtualization","text":"This design supports virtualization by making use of the virtualization functionality in the PCIe Hard IP and mapping packets to the appropriate physical or virtual function through a PF/VF multiplexer.
This reference FIM example supports 2 PFs for the host and 1PF, 3VFs for the SoC; however, you may extend your configuration to whatever the PCIe Hard IP can support or your application requires.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#142-afu","title":"1.4.2 AFU","text":"An AFU is an acceleration workload that interfaces with the FIM. The AFU boundary in this design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR-specific modules and logic required for partial reconfiguration. Only one partial reconfiguration region is supported in this design for SoC AFU.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways: * Your AFU (workload) for Host (Static Area) and SoC resides in the partial reconfiguration area. * Your AFU is part of the static region and is compiled from a flat design.
In this design, the AFU region is comprised of: * AFU Interface handler to verify transactions coming from AFU region. * PF/VF Mux to route transactions to and from corresponding AFU components: * Host: * ST2MM module. * PCIe loopback host exerciser (HE-LPBK) .
* SoC: * ST2MM module. * Ethernet Subsystem (previous HSSSI) host exerciser (HE-HSSI). * Memory Host Exerciser (HE-MEM). * Traffic Generator to memory (HE-MEM-TG). * Port Gasket (PRG).
- Arm\u00ae AMBA\u00ae 4 AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and Ethernet Interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
For this design the PF/VF Mux provides the following mappings (found in /src/afu_top/mux/top_cfg_pkg.sv and /src/afu_top/mux/soc_top_cfg_pkg.sv):
Table 1-5 PF/VF Mapping
SoC (IceLake-D)
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 2MB AFU peripherals Board peripherals BAR 0 256KB 768KB PF0 VF0 HE-MEM BAR 0 2MB VF1 HE-HSSI BAR 0 2MB VF2 HE-MEM TG BAR 0 2MB Xeon Host
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 1MB AFU peripherals Board peripherals 256KB 768KB PF1 HE-LPBK BAR 0 16KB Figure 1-2 AFU Diagram
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#143-platform-interface-manager","title":"1.4.3 Platform Interface Manager","text":"The PIM provides a way to abstract the Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface to the AFU by providing a library of shims that convert the host channel native packet into other protocols such as Arm\u00ae AMBA\u00ae 4 AXI4 memory-mapped, Avalon\u00ae streaming (Avalon-ST) or Avalon\u00ae memory-mapped (Avalon-MM).
The FPGA or AFU developer implements these interface abstractions in the AFU regions (afu_top and soc_afu_top) of the design.
For more information, refer to the AFU Developer's Guide.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#144-platform-feature-discovery","title":"1.4.4 Platform Feature Discovery","text":"This reference design comes with specific FPGA drivers that are upstreamed to linux-dfl. These drivers abstract the hardware and operating system specific details of the platform to the host.
The FIM implements a device feature header (DFH) at the beginning of each host-discoverable feature in a linked list format that allows an FPGA platform driver running on the host to discover FME, port, and AFU features.
You must implement a 64-bit DFH Device Feature Header register at the beginning (first 8B aligned address) of the feature CSR space for a new feature to be discovered or enumerated by a driver.
During host discovery, the driver traverses the DFH of the first feature from the first address on PF0 BAR0. Based on the information in the DFH, a driver can determine the CSR address range of the feature and other associated details. The end of the DFH contains a \"next DFH offset\" field that points the driver to the DFH of the next feature.
The software must continue traversing the linked list until it sees the EOL (End-Of-List) bit set to 1 in the \"next DFH offset\" field it inspects. A 1 indicates this is the last feature in the feature set. The figure below gives a simple illustration of the feature discovery by traversing the DFH registers. This model is similar to how PCIe enumeration occurs.
Figure 1-3 Device Feature Header Linked List Traversal
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#145-ofs-reference-design","title":"1.4.5 OFS Reference Design","text":"OFS provides FIM designs you can use as a starting point for your own custom design. These designs target a specific programmable acceleration card or development kit and exercise key FPGA device interfaces.
The Agilex\u00ae 7 code line for OFS targets the IPU Platform F2000X-PL. FIM designs are released to for evaluation and use.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#146-fim-simulation","title":"1.4.6 FIM Simulation","text":"OFS provides unit tests and a UVM environment for the FIM and a framework for new feature verification. UVM provides a modular, reusable, and scalable testbench structure by providing an API framework that can be deployed across multiple projects.
The FIM testbench is UVM compliant and integrates third-party verification IPs from Synopsys that require a license.
Verification components include:
- FIM monitor to detect correct design behavior
- FIM assertions for signal level integrity testing
- Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity
- FIM coverage to collect functional data
The verification infrastructure can be found here for evaluation and use. Please refer to the UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach for more information.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#2-ofs-high-level-architecture","title":"2 OFS High Level Architecture","text":"OFS provides distinct data paths that simplify the design and integration process for adding or removing interface modules:
- High Bandwidth data path for AFU-attached high-performance peripherals (Ethernet Subsystem, Memory, workload).
- Low Bandwidth data path for OFS management and slow peripheral components (JTAG, I2C, SMBus).
- AFU Peripheral Fabric (APF) to Board Peripheral Fabric (BPF) path to communicate with interface control and status registers (CSRs) and board components.
- Peer-to-peer datapath between AFU components.
- Peer-to-peer datapath between BPF components.
Depending on your design goals, you can present peripherals to software as:
- OFS managed peripherals with a device feature header that is part of a device feature list.
- Native driver managed peripherals that are exposed through an independent physical function or virtual function.
Figure 2-1 OFS Datapath Structure
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#3-pcie-subsystem","title":"3 PCIe Subsystem","text":"The FIM's PCIe Subsystem is a hierarchical design that targets the P-tile PCIe* hard IP and is configured to support Gen4 speeds and Arm\u00ae AMBA\u00ae 4 AXI4-Stream Data Mover functional mode. The IP supports SR-IOV and is configured to provide 2 PFs for the host and 1PF, 3VFs for the SoC. Native PCIe TLP packets are sent through the PCIe usingArm\u00ae AMBA\u00ae 4 AXI4 Stream Protocol. Before they reach the AFU, the packets go through an adapter in the subsystem that converts any headers to a data mover format. Tag allocation and management for packets sent from the application to the host are done by the PF/VF Mux which is part of the AFU region.
Figure 3-1 OFS FIM RX-TX Datapath
Some key features of the PCIe interface are:
Feature OFS for Agilex 7 FPGA SoC Attach Subsystem Configuration Mode Host: PCIe Gen4x16SoC: PCIe Gen4x16 Tile P-Tile Port Mode Native Endpoint SR-IOV Host: 2 PFs, No VFsSoC: 1 PFs, 3 VFs MSI-X Support Yes Functional Mode Data Mover Profile Basic TLP Bypass No Header Packing Scheme Simple Data Width 512-bit (64-byte) Arm\u00ae AMBA\u00ae 4 AXI4-ST Clock Frequency 500 MHz Tags Supported 128 Reordering Enabled with buffer 64 KB Maximum Payload Size 512 Bytes Memory Requests Supported 1CL, 2CL, 4CL MMIO transaction Size 4B, 8B Control Shadow Interface Enabled Completion Timeout Interface Enabled The PCIe PF, VF and Base Address Register (BAR) configuration can be modified in the PCIe Subsystem Platform Designer GUI interface. The current implementation for the OFS FIM for IPU Platform F2000X-PL is as follows:
Table 3-1 Function and BAR Table for OFS for IPU Platform F2000X-PL
SoC (IceLake-D)
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 2MB AFU peripherals Board peripherals BAR 0 256KB 768KB PF0 VF0 HE-MEM BAR 0 2MB VF1 HE-HSSI BAR 0 2MB VF2 HE-MEM TG BAR 0 2MB Xeon Host
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 1MB AFU peripherals Board peripherals 256KB 768KB PF1 HE-LPBK BAR 0 16KB"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#31-pcie-subsystem-header-format","title":"3.1 PCIe Subsystem Header Format","text":"The first 32 bytes of the TLP from the PCIe subsystem denotes the PCIe header. There are two types of header format supported \u2013 Power User Mode header and Data Mover mode header. The tuser_vendor[0] bit on the Arm\u00ae AMBA\u00ae 4 AXI4-Stream channel indicates the header format of the TLP; tuser_vendor[0] =0 indicates Power User Mode header and tuser_vendor[0] =1 indicates Data Mover Mode header.
The OFS FIM for Agilex 7 FPGA implements the Data Mover Functional Mode. With this implementation, the application has the flexibility to use either mode for PCIe transaction, as shown in the following table. For more detailed information about the PCIe Subsystem, see the PCIe Subsystem FPGA User Guide.
Table 3-2 PCIe Subsystem Header Format Support for OFS for Agilex 7 FPGA
Direction Type Power User Data Mover Host to Endpoint MWr, MRd Yes No CPL/CPLd Yes Yes Msg Yes No Endpoint to Host MWr, MRd Yes Yes Intr Yes (MWr) Yes CPL/CPLd Yes Yes Msg Yes Yes"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#311-power-user-header-format","title":"3.1.1 Power User Header Format","text":"The Power User Format provides user complete control over PCIe Hard IP. The user can implement functionality of interest with finer control over PCIe Transaction Layer Packet (TLP), credit handling and various mode provided by HIP.
The lower 16 bytes of the Power User Format are standard PCIe header as defined by PCIe specification, and the upper 16 bytes are specific to the PCIe Subsystem FPGA IP.
Table 3-3 Power User Header Format
The mapping of the PCIe defined header to the lower 16 bytes of the Arm\u00ae AMBA\u00ae 4 AXI4-Stream data channel is shown in the figure below. Each double word (DW) or 4 bytes in the PCIe header is mapped from big-endian to little-endian on Arm\u00ae AMBA\u00ae 4 AXI4-S data channel.
Figure 3-2 Power User Header Mapping to Arm\u00ae AMBA\u00ae 4 AXI4 Channel
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#312-data-mover-header-format","title":"3.1.2 Data Mover Header Format","text":"The data mover mode allows high bandwidth data transfers to and from Host memory. It hides the complexity of handling PCIe TLPs. This format provides a simple interface for reading and writing to Host Memory. The data mover checks link partner credits before transmitting packets. It also provides MSI-X interrupt generation capability.
In Data Mover Mode, the lower 16 bytes are data mover specific and do not follow a PCIe standard header.
Table 3-4 Data Mover Header Format
The mapping of the data mover header to the lower 16 bytes of the Arm\u00ae AMBA\u00ae 4 AXI4-S data channel is shown below. Each byte in the data mover header is mapped directly to the Arm\u00ae AMBA\u00ae 4 AXI4-S data channel.
Figure 3-3 Data Mover Header Mapping to Arm\u00ae AMBA\u00ae 4 AXI4 Channel
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#32-pcie-subsystem-interface-module","title":"3.2 PCIe Subsystem Interface Module","text":"The PCIe Subsystem Interface module (/ipss/pcie/rtl/pcie_ss_if.sv), provides the supporting interface between software and the PCIe subsystem.
The interface module provides the following:
- Device Feature Header Registers
- Control and Status Registers
- Indirect access to PCIe subsystem CSR registers through a CSR mailbox in the PCIe Subsystem Interface.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#33-data-mover-request-cycles","title":"3.3 Data Mover Request Cycles","text":"For Host read request cycles using the OFS FIM for Agilex 7 FPGA: * All requests in the RX direction will be MMIO. * Requester ID from the request does get sent to the AFU. It is the AFU's responsibility to send back a completion to the host with the correct completer ID. * Prefix is not supported. * Memory Mapped (MM) Mode is not supported. * Slot Number is 0. * Base address is not sent to the AFU. * Local Address field is not used.
For AFU/application request cycles using the OFS FIM for Agilex 7 FPGA: * All requests in the TX direction will be Memory Read/Write. * The tag must be generated by the AFU/application. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0 (non-0 only for switch) * VF Active, VF number and PF number are obtained from Data Mover Header Packet.
Figure 3-4 Data Mover Request Cycles
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#34-data-mover-completion-cycles","title":"3.4 Data Mover Completion Cycles","text":"For Host completion cycles using the OFS FIM for Agilex 7 FPGA: * All completions in the RX direction will be Data Completions. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0. * Data packet responses (for Memory Read requests from AFU) from the PCIe SS may come out of order when the size is >64B.
For AFU/application completion cycles using the OFS FIM for Agilex 7 FPGA: * All requests in the TX direction will be Memory Read/Write. * Requester ID is generated within the FIM. * That tag must be generated by the AFU/application. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0. * VF Active, VF Number and PF number are obtained from the Data Mover Header Packet.
Figure 3-5 Data Mover Completion Cycles
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#4-platform-interface-manager","title":"4 Platform Interface Manager","text":"The FIM interfaces to an application in the AFU region through Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels. This format allows the AFU to access the host channel's raw interface without any translation.
As a FIM developer, you have the option to provide the raw data format associated with the host interface channel to the workload or AFU developer or you can provide an intermediate protocol using Platform Interface Manager Components or your own custom interface.
If you expose the raw Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface of the FIM, workload developers also have the option to convert to a desired protocol using the PIM resources as well.
Refer to the AFU Developer Guide and the FPGA Interface Manager Developer Guide for more information on using the PIM in your design.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#5-interconnect-fabric","title":"5 Interconnect Fabric","text":"There are three types of interconnect fabric in the OFS FIM design: * Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux/demux fabric * AFU Peripheral Fabric (APF) * Board Peripheral Fabric (BPF)
Figure 5-1 Interconnect Fabric Diagram
TLP packets sent from upstream PCIe Subsystem on Arm\u00ae AMBA\u00ae 4 AXI4-Stream channel are demultiplexed in the Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux/demux fabric and routed to the respective PF/VF function based on the PF/VF information in the TLP header, such as vf_active or the PF/VF number. In the opposite direction, TLP packets from downstream PF/VF function are muxed in the fabric and sent to PCIe subsystem over Arm\u00ae AMBA\u00ae 4 AXI4-Stream channel.
All host MMIO requests targeting PF0 BAR0 are routed to the ST2MM module. The ST2MM converts MMIO TLP packets into Arm\u00ae AMBA\u00ae 4 AXI4-Lite memory requests and places the requests onto AFU Peripheral Fabric (APF). AFU peripherals, such as OFS managed AFU features and ST2MM, and Board Peripheral Fabric (BPF) are interconnected by APF. The BPF is the interconnect fabric one hierarchy below APF which connects all the board peripherals. Both APF and BPF allow multiple Arm\u00ae AMBA\u00ae 4 AXI4-Lite master and slave interconnect topology.
If you are modifying the APF or BPF connections, you must use Platform Designer to generate the fabrics directly. Please refer to the FPGA Interface Manager Developer Guide for directions on what files must be modified and how to generate the interconnect.
For modifying the PF/VF mux you must update parameters in these files: * src/includes/top_cfg_pkg.sv * src/common/pf_vf_mux.sv Then make the corresponding update to AFU top level instantiation and connections: * src/afu_top/soc_afu_top.sv(SoC_AFU) and src/afu_top/afu_top.sv (HOST_AFU)
For details on these modifications, please refer to the FIM Interface Manager Developer Guide.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#51-afu-peripheral-fabric-apf","title":"5.1 AFU Peripheral Fabric (APF)","text":"The AFU Peripheral Fabric (APF) is a 64-bit Arm\u00ae AMBA\u00ae 4 AXI4-lite compliant interconnect fabric that connects AFU peripheral modules to board peripheral modules through the Board Peripheral Fabric (BPF).
The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by the APF is listed below. All components are mapped to PF0 BAR0 and implement Arm\u00ae AMBA\u00ae 4 AXI4-lite slave interface. Note that none of the features in the APF mapping are designed to act as a master.
Table 5-1 APF Address Mapping
SoC PF0 BAR 0
Address Size (Byte) Feature 0x00000\u20130xFFFFF 1024K Board Peripherals (See BPF address mapping) AFU Peripherals 0x100000 \u2013 0x10FFFF 64K ST2MM 0x130000 \u2013 0x13FFFF 64K PR Gasket: 4K= PR Gasket DFH, control and status 4K= Port DFH 4K=User Clock 52K=Remote STP 0x140000 \u2013 0x14FFFF 64K AFU Error Reporting (Protocol checker) Host PF0 BAR 0
Address Size (Byte) Feature 0x00000\u20130xFFFFF 1024K Board Peripherals (See BPF address mapping) AFU Peripherals 0x100000 \u2013 0x10FFFF 64K ST2MM 0x140000 \u2013 0x14FFFF 64K AFU Error Reporting (Protocol checker)"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#52-board-peripheral-fabric-bpf","title":"5.2 Board Peripheral Fabric (BPF)","text":"The Board Peripheral Fabric is the 64-bit Arm\u00ae AMBA\u00ae 4 AXI4-Lite compliant interconnect fabric that connects board peripheral modules to APF. The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by BPF is listed below. All components are mapped to PF0 BAR0 and implement Arm\u00ae AMBA\u00ae 4 AXI4-lite slave interface. The Master column indicates if a component also implements Arm\u00ae AMBA\u00ae 4 AXI4-lite master interface which can send requests to the BPF.
Table 5-2 BPF Address Mapping
SoC PF0 BAR 0
Address Size (Byte) Feature Master 0x00000 \u2013 0x0FFFF 64K FME (FME, Error, etc) No 0x10000 \u2013 0x10FFF 4K SoC PCIe Interface No 0x11000 \u2013 0x11FFF 4K Reserved - 0x12000 \u2013 0x12FFF 4K QSFP Controller 0 No 0x13000 \u2013 0x13FFF 4K QSFP Controller 1 No 0x14000 \u2013 0x14FFF 4K Ethernet Subsystem - 0x15000 \u2013 0x15FFF 4K Memory Subsystem 0x80000 \u2013 0xFFFFF 512K PMCI Controller Yes Host PF0 BAR 0
Address Size (Byte) Feature Master 0x00000 \u2013 0x00FFF 4K Host PCIe Interface No 0x80000 \u2013 0xFFFFF 512K PMCI Controller Yes"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#53-arm-amba-4-axi4-stream-pfvf-muxdemux","title":"5.3 Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux routes the PCIe TLP packets from the PCIe subsystem Arm\u00ae AMBA\u00ae 4 AXI4-Stream RX channel to downstream PF/VF based on the pf_num and vf_num information in the PCIe TLP header.
The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux arbitrates PCIe TLP packets from downstream PF/VF to the PCIe SS Arm\u00ae AMBA\u00ae 4 AXI4-S TX channel. The PF/VF Mux/Demux is an M X N switch that allows any M port to target any N port, and any N port to target any M port, where M is the number of host/upstream ports, and N is the numbers functions/downstream ports.
The fpga top package file, /src/afu_top/mux/top_cfg_pkg.sv and soc_top_cfg_pkg.sv and, contains the PF/VF parameters and mappings.
Figure 5-2 PF/VF Mux
The PF/VF mux integration is part of afu_top (src/afu_top/afu_top.sv and /soc_afu_top). There are two independent TX PF/VF MUX trees, labeled \"A\" and \"B\".
Both an A and a B port are passed to each AFU component with a unique PF/VF. You can design your AFU components to send all requests to the primary A port or partition requests across both A and B ports. A typical high-performance AFU sends read requests to the B port and everything else to the A port, giving the arbiter freedom to keep both the host TX and RX channels busy.
In the reference FIM provided for Agilex 7 OFS, the A and B TX trees have been multiplexed down to a single channel for A and another for B. The A/B multiplexer merges them into a single TX stream that will be passed to the tag remapper.
The tag remapper provides unique tags as required by the PCIe specification. Tags are not provided by the PCIe Subsystem FPGA IP. When creating your own AFU you can leverage this module to generate unique tags.
Note that the primary PF/VF Mux A supports RX and TX ports. For the secondary PF/VF Mux B only TX ports are supported and the RX input to the Mux is tied off.
The default mapping is shown below:
Table 5-3 PF/VF Mapping
SoC (IceLake-D)
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 2MB AFU peripherals Board peripherals BAR 0 256KB 768KB PF0 VF0 HE-MEM BAR 0 2MB VF1 HE-HSSI BAR 0 2MB VF2 HE-MEM TG BAR 0 2MB Xeon Host
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 1MB AFU peripherals Board peripherals 256KB 768KB PF1 HE-LPBK BAR 0 16KB For information on how to modify the PF/VF mapping for your own design, refer to the OFS FIM Developer User Guide.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#54-afu-interface-handler","title":"5.4 AFU Interface Handler","text":"The AFU Interface Handler resides inline between the PCIe Arm\u00ae AMBA\u00ae 4 AXI4-Stream Adapter and the Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Demux/Mux logic. Its main function is to provide: * Unique PCIe tags \u2013 Each PCIe transaction shares the 512 tags across all VFs in the AFU region * AFU error logging for all VFs in the AFU region
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#541-unified-tag-remapping","title":"5.4.1 Unified Tag Remapping","text":"When a FPGA function sends out a read cycle, it allocates a unique tag which is subsequently used to identify the read completion. The tag is considered busy; it cannot be assigned to another read cycle until read completion. While a tag may be unique within a unit, two different units could unknowingly send out two read cycles of the same tag. The PCIe subsystem requires unique tags for all read cycles irrespective of their origins. Therefore, a mechanism is needed to uniquify tag globally across different units.
OFS contains a tag remapper (tag_remap.sv) that intercepts the read cycle, finds a globally unique tag, and replaces the original tag value. It also restores the original tag value when returning completion to the read requester. tag_remap is placed between the Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface of the PCIE subsystem and the PF/VF Mux/Demux.
The logic is described as follows:
- A sub-module (
ofs_fim_tag_pool) maintains a pool of available tags. - TX read requests are held until a tag is available from the pool by setting tvalid=0 to the host, and tready=0 to the PF/VF Mux/Demux.
- When a TX read is dispatched, the tag is marked busy in the pool.
- The original tag is stored in tag_reg, so it can be recovered when returning a completion to the unit/function.
- Because completion to a read request can split into multiple smaller transfer sizes, responses are monitored and the final completion is detected using PCIe TLP rules.
- Tags are released in the pool only when all requested data are transferred.
- When the completion returns, the original tag is restored from
tag_reg.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#542-afu-error-handling","title":"5.4.2 AFU Error Handling","text":"In this OFS design, the AFU Interface Handler handles error logging for all VFs in the AFU. Errors handled are as follows
Table 5-4 AFU Error Descriptions
Checker Field Description AFU protocol checker (PCIe TLP) Malformed TLP AFU PCIe TLP contains unsupported format type MaxPayloadError AFU memory write payload size exceeds max_payload_length limit MaxReadReqSizeError AFU memory read payload size exceeds max_read_request_size limit MaxTagError AFU memory read request tag value exceeds the maximum supported tag count UnalignedAddrErr The address field in AFU memory write/read request TLP is not DW-aligned. UnexpMMIOResp AFU is sending a MMIO read response with no matching MMIO read request. MMIOTimedOutAFU is not responding to a MMIO read request within the pre-defined response timeout period. MMIODataPayloadOverrunThe number of data payload sent by AFU for a MMIO response (cplD) is more than the data length specified in the response. MMIOInsufficientDataThe number of data payload sent by AFU for a MMIO response (cplD) is less than the data length specified in the response. TxMWrDataPayloadOverrun The number of data payload sent by AFU for a memory write request is more than the data length specified in the request. TxMWrInsufficientData The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU Protocol Checker (Arm\u00ae AMBA\u00ae 4 AXI4-Stream)TxValidViolationThree checkers are implemented in the FIM to catch errors and protocol violations."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#55-tlp-to-arm-amba-4-axi4-lite-memory-mapped-bridge-st2mm","title":"5.5 TLP to Arm\u00ae AMBA\u00ae 4 AXI4-Lite Memory Mapped Bridge (ST2MM)","text":"ST2MM implements the following key features: * Host MMIO bridge * Maps MMIO TLP packets received from the PCIe Subsystem over streaming interface to Arm\u00ae AMBA\u00ae 4 AXI4-Lite memory-mapped request. The memory-mapped request is sent to AFU or Board peripherals over APF and BPF. * Maps Arm\u00ae AMBA\u00ae 4 AXI4-lite MM response received from AFU or Board peripherals to TLP packets and send the packets over ST streaming channel to host HIA subsystem. * Sends MMIO response of all 0\u2019s for MMIO read to unused BAR region. * Interrupt * Sends interrupt packets to the PCIe subsystem when interrupt requests are received from the peripherals. Interrupts can be requested by a peripheral through a memory write to interrupt CSR registers in the ST2MM.
Figure 5-3 ST2MM Module
ST2MM implements both Arm\u00ae AMBA\u00ae 4 AXI4-lite master and slave interfaces that are connected to the designated slave and master port on APF. Host memory requests are sent on the ST2MM master interface to AFP where the requests are routed to the targeted peripherals.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#6-mmio-regions","title":"6 MMIO Regions","text":"The FIM and AFU expose their functionalities to the host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). An MMIO region is an address space within a base address register (BAR) region to which features are memory mapped.
For example, when a feature is mapped to an MMIO region, the CSR registers of that feature are located within the address range of that region. There can be multiple MMIO regions within a BAR region.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#61-feature-region","title":"6.1 Feature Region","text":"A group of related CSRs can be categorized as a feature region. For example, a DMA engine has queue management function and quality of service (QoS) function; these are two different features of the DMA engine. A feature region is contained within a single PCIe BAR and cannot span across two BAR region boundaries.
A Device Feature Header (DFH) register marks the start of the feature region and sub-feature region, and you must place it at the first address of the region. Each DFH starts at 4KB boundary. A DFH register contains information that OPAE software requires to enumerate the feature. It also has an offset field that points to the next DFH in a feature list. OPAE software traverses the linked list of DFHs in each BAR region to discover all the features implemented on the platform.
The EOL field in a DFH marks the end of a DFH list and is only set in the DFH of the last feature in the feature list. The feature type field in the DFH is used to differentiate between the different types of feature region. Basic building blocks (BBB) and private features are always a child of an AFU or FPGA Interface Unit (FIU) and must be contained within an AFU or FIU, respectively.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#611-device-feature-header-dfh-structure","title":"6.1.1 Device Feature Header (DFH) Structure","text":"All DFHs must follow the following structure to be compatible with OPAE software. Note that only features you want to be exposed to the OPAE software must have a DFH.
Table 6-1: DFH Register Structure
Bitfield Name Range Access Description FeatureType 63:60 RO 4\u2019b0000 \u2013 Reserved 4\u2019b0001 \u2013 AFU4\u2019b0010 \u2013 BBB4\u2019b0011 \u2013 Private Feature4'b0100 \u2013 FIM Reserved 59:41 Rsvd Reserved EOL 40 RO End of DFH List1'b0=No other feature header beyond this one1'b1=This is the last feature header NextDFHByteOffset 39:16 RO Next DFH byte offsetNext DFH Address= Current DFH address + Next DFH byte offset. You can also use this value as an indication of the maximum size of the MMIO region occupied by this feature. FeatureRev 15:12 RO For AFU Feature type= AFU major version number that is user defined.All other feature types= Feature revision number FeatureID 11:0 RO For AFU feature type= CoreFIM version numberFor BBB feature type= Intel defined ID for BBBFor private feature type= User-defined ID to identify within an AFU For FIU type=ID for FIU unit (ex. 0 for FME, 1 for Port) You must increment a feature revision number if a feature changes. This change requires a corresponding change in the software to detect the new version and report mismatches between the hardware and software revision number.
When indicating \"FeatureType\" in your DFH Structure, use the following guidance: * FIM= Any static region feature you want discovered through software. FIM features must implement mandatory FIM registers, including a FIM feature ID. * AFU= User Application region; may be PR region. AFU feature must implement mandatory AFU register including an AFU ID. * Private Feature = Linked list of features within the FIM or AFU which are enumerated and managed by separate software and do not require an ID. * Basic Building Blocks (BBB) = Special features within the AFU or FIM that are meant to be reusable basic building blocks. They are enumerated and called by separate software services.
Table 6-2: Mandatory FIM and AFU Registers
The following table details the registers required for a FIM feature or AFU to be discovered by OPAE software. The ID_H and ID_L fields allow you to create a unique feature or AFU identifier that remains unchanged while the FeatureRev in the DFH register increments.
Byte Address offset with respect to DFH Register Name 0x0000 DFH with corresponding FeatureType field implemented 0x0008 ID_L: Lower 64 bits of unique ID number (GUID) 0x0010 ID_H: Upper 64 bits of unique ID number (GUID) 0x0018 Next AFU Table 6-3: Next AFU Register Definition
The following table details the \"Next AFU Register Definition.\" This register is used if you have multiple AFUs that can be used with your FIM.
Bit Attribute Description 63:24 Reserved Reserved 23:0 RO Next AFU DFH Byte Offset Next AFU DFH address =current address + offset; where a value of 0 implies this is the last AFU in the list. Example: AFU0 at address 0x0: Next AFU offset= 0x100 AFU1@ address 0x100: Next AFU offset = 0x100 AFU2 at address 0x200: Next AFU offset = 0x0 (End of AFU List) 6.2 Control and Status Registers
All the Control and Status Registers (CSRs) in the FIM are 64-bit registers with the following MMIO write, and MMIO read support.
Table 6-4: CSR MMIO Read and Write Support
Request Memory Attribute Payload size Memory Ordering MMIO Write UC 4B or 8B Strongly ordered MMIO Read UC 4B or 8B Strongly ordered The FIM does not reorder the MMIO requests or responses. For MMIO writes, there is no reordering of requests in FIM, and UC ordering rules are followed. Similarly, for MMIO reads, there is no re-ordering of requests or responses in the FIM. An AFU may opt to re-order the MMIO read responses but the FIM does not enforce read response ordering.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#621-software-access-to-registers","title":"6.2.1 Software Access to Registers","text":" - Software accesses 64-bit registers as aligned quadwords. For example, to modify a field (bit or byte) in a 64-bit register, the entire quadword is read, the appropriate field(s) are modified, and the entire quadword is written back.
- When updating registers through multiple accesses (whether in software or due to hardware disassembly), certain registers may have specific requirements on how the accesses must be ordered for proper behavior. These are documented as part of the respective register descriptions.
- For compatibility with future extensions or enhancements, software must assign the last read value to all \u201cReserved and Preserved\u201d (RsvdP) fields when written. In other words, any updates to a register must be read so that the appropriate merge between the RsvdP and updated fields occurs. Also, software must assign a value of zero for \u201cReserved and Zero\u201d (RsvdZ) fields when written.
- PCIe locked operations to FPGA hardware registers are not supported. Software must not issue locked operations to access FPGA hardware registers.
In the following two cases, the FIM terminates MMIO Read requests by sending a completion with the data (CplD) specified below: * MMIO Timeout: This occurs when the AFU does not respond within a set timeout. The timeout value is currently configured to 512 pclks (clk_2x). In this case, the FIM returns all 1s.
- Illegal MMIO Accesses: This occurs when the read is accessing undefined registers in the FIM or if an AFU access violation. An example of an access violation is when a PF attempts to access the AFU when it is set to VF mode, or when a VF attempts to access the AFU when it is set to PF mode. In this case, the FME will returns all 0s.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#62-register-attribute-definition","title":"6.2 Register Attribute Definition","text":"Table 6-5: OFS Register Attribute Definitions
Attribute Expansion Description RW Read/Write This bit can be read or written by software. RO Read Only The bit is set by hardware only. Software can only read this bit. Writes do not have any effect. RW1C Read/ Write 1 to Clear Software can read or clear this bit. The software must write 1 to clear this bit. Writing zero to RW1C bit has no effect. Note that a multi-bit RW1C field may exist. In this case, all bits in the field are cleared if a 1 is written to any of the bits. RW1S Read/ Write 1 to Set Software can read this bit. Writing a 1 to the bit sets it to 1. Writing a 0 has no effect. It is not possible for software to set this bit to 0. The 1 to 0 transition can only be performed by HW. RW1CS Read/Write 1 to Clear Sticky Software can read and clear this bit. Writing a 1 to a bit clears it, while writing a 0 to a bit has no effect. This bit is only reinitialized to its default value by a power-on reset. RWD Read/Write Sticky across Hard Reset The bit can be read or written by SW. This bit is sticky or unchanged by any reset type, including Hard Reset. The bit gets cleared only with power on. *S Sticky across Soft Reset The bit will be sticky or unchanged by soft reset. These bits are only re-initialized to their default value by a power-on reset. *D Sticky across Hard Reset The bit is sticky or unchanged by or unchanged by any reset type, including hard reset. The bit gets cleared only with power on. Rsvd Reserved Reserved for future definitions. Currently don\u2019t care bits. RsvdP Reserved and Protected Reserved for future RW implementations. The software must preserve the value of this bit by read modify write. RsvdZ Reserved and Zero Reserved for future RW1C implementations. The software must write zero to this bit."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#623-csr-offset-in-bars","title":"6.2.3 CSR Offset in BARs","text":"The table below captures the FIM and AFU features in the supported BAR regions. The highlighted offset indicates the first DFH in the DFH list of a BAR region where device driver starts the DFH traversal.
Table 6-6: PF0 BAR0 Features SoC AFU
Offset Feature CSR set 0x0000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 Ethernet Interface 0x15000 Memory Interface 0x80000 PMCI 0x100000 St2MM 0x130000 PR Control & Status (Port Gasket) 0x131000 Port CSRs (Port Gasket) 0x132000 User Clock (Port Gasket) 0x133000 Remote SignalTap (Port Gasket) 0x140000 AFU Errors (Protocol Checker) Table 6-7: PF0 BAR0 Features Host AFU
Offset Feature CSR set 0x0000 PCIe Interface"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#7-clocks","title":"7 Clocks","text":"The following table provides external clock sources which correspond to pins on the FPGA that drive blocks of internal logic or are used as a reference clock for internal PLLs. The names in the table are used in the top.sv or are the logical clock names used by their respective functional blocks.
Table 7-1: External Clock Source
Clock Frequency Description SYS_REFCLK 100MHz Reference clock to system IOPLL (sys_pll) which provides FIM system clocks. PCIE_REFCLK0 100MHz PCIe Refclk 0 PCIE_REFCLK1 100MHz PCIe Refclk 1 SOC_PCIE_REFCLK0 100MHz PCIe Refclk 0 on the SOC side SOC_PCIE_REFCLK1 100MHz PCIe Refclk 1 on the SOC side qsfp_ref_clk 156.25MHz Ethernet Refclk ddr4[0].ref_clk 150MHz Refclk for EMIF channel 0 ddr4[1].ref_clk 150MHz Refclk for EMIF channel 1 ddr4[2].ref_clk 150MHz Refclk for EMIF channel 2 ddr4[3].ref_clk 150MHz Refclk for EMIF channel 3 sdm_config_clk 125MHz SDM Config Clock altera_reserved_tck 10MHz Default JTAG Clock Table 7-2: Internal Clocks
Clock Frequency Description clk_sys 400MHz System clock. Primary clock for PCIe datapath. clk_100m 100MHz 100MHz clock. For RemoteSTP and user clock, or any logic that requires a 100MHz clock. clk_csr 100MHz Arm\u00ae AMBA\u00ae 4 AXI4-lite interconnect fabric and CSR clock. Driven by clk_100m. clk_2x 400MHz 2x clock to AFU in PR slot. Driven by clk_sys. clk_1x 171.43MHz 1x clock to AFU in PR slot. uclk_usr 312.50 MHz Configurable \u201cH\u201d clk for AFU. Default 312.50 MHz uclk_usr_div2 156.25 MHz Configurable \u201cL\u201d clk for AFU. Default 156.25 MHz"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#8-reset","title":"8 Reset","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#81-reset-signals","title":"8.1 Reset Signals","text":"The FIM system reset signals are driven according to a their respective requirements derived from their use cases. The behavioral code defining the reset operation is located in the file rst_ctrl.sv. The FIM System Reset Drivers table below provides a list of the input and feedback signals that compose the various resets.
Table 8-1: FIM System Reset Drivers
Reset Description nPERST pin Active low PCIe reset pin from the PCIe card edge that may be set by the host machine for cold or warm resets. nCONFIG pin Active low input to the FPGA that causes the FPGA to lose its configuration data, enter a reset state, and tri-state all I/O pins. Host software must reload the FPGA FIM after nCONFIG is activated. ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. pcie_reset_status Active high reset status from PCIe hard IP. When driven high, this signal indicates that the PCIe IP core is not ready for usermode. locked Active high SYS IOPLL locked signal pcie_cold_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Cold Reset sequence has been completed. pcie_warm_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Warm Reset sequence has been completed. Upon power-on, the reset module in the FIM holds the FIM in reset until all the reset conditions are de-activated:
nPERST signal is asserted.- The
INIT_DONE pin is driven high to indicate core configuration is complete. - The SYS IOPLL is locked.
- The reset status from PCIe hard IP is de-asserted indicating the IP is ready for transfer.
The reset module places the FIM back into reset if any of these conditions becomes active again. The only way to invoke a system reset to the FIM after power-up is to deassert the nPERST pin either by performing a warm reboot or through PCIe driver intervention. There are soft reset signals set aside to allow software to reset the Port, AFU and partial reconfiguration IP.
The following table lists the reset outputs from the rst_ctrl.sv block.
\u200b Table 8-2: FIM System Reset Outputs
Reset Description rst_n_sys pin System general reset synchronous to clk_sys. This is a warm reset of the FPGA logic. Sticky bits in the FME and other CSR registers will not be reset by this signal. rst_n_100m pin System general reset synchronous to clk_100m. ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. pwr_good_n Hardware reset conditioned by ninit_done and the pll_locked signal. The signal is generally set when power has been cycled or a physical hardware fault has occurred, requiring a reset of the FPGA logic. This signal is synchronous to clk_sys. pcie_cold_rst_ack_n Hardware reset conditioned by the pcie_reset_status which is a summary reset driven by the PCIe Hard IP logic tile on the FPGA die. This signal is synchronous to clk_sys. pcie_warm_rst_ack_n Soft reset conditioned by nPERST when the pcie_reset_status is not asserted, meaning a warm reset is desired. Cold reset sticky bits in the PCIe subsystem will not be reset by this signal."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#9-interrupts","title":"9 Interrupts","text":"The OFS platform supports interrupts through MSI-X feature implemented in the PCIe SS. The MSI-X Interrupt feature handles FME and AFU interrupts.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#91-msi-x","title":"9.1 MSI-X","text":"FME interrupts are primarily used to notify the host of error events happened in the FIM. When any of the bits in the FME error status registers is set, an interrupt request is sent to the MSI-X module. There are FME error status registers for various FPGA features such as RAS Error, Temperature monitoring etc.\u202fIf any of those error registers log an error, we trigger an FME interrupt.
An AFU sends an interrupt to the MSI-X module in the PCIE SS on the AXI interrupt request channel.
The MSI-X table entries and PBA vectors are implemented in the PCIE SS. The PCIE SS supports upto 4096 vectors in Static MSI-X mode
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#92-msi-x-entry-table","title":"9.2 MSI-X Entry Table","text":"Each entry in the MSI-X table stores the message address, message data, and vector control for one interrupt vector. The address, data, and vector control information are populated by software (usually done by the PCIe SRIOV driver) after reading the PCIe endpoint configuration space.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#93-msi-x-pba-table","title":"9.3 MSI-X PBA Table","text":"The PBA table contains a corresponding bit for each MSI-X interrupt vector. This PBA bit is set if that interrupt is triggered but is masked. When the interrupt is unmasked, the PBA bit is unset, and an interrupt is sent to the Host. When the Application generates an interrupt from an IRQ source, it sets the corresponding PBA bit.
When the interrupt is triggered by an IRQ Source, the IRQ Processor checks the masking of that interrupt to determine if the PBA bit should be set. If unmasked, the IRQ Processor looks up MSI-X address and data for the interrupt vector and generates the interrupt request over the PCIe EP.
The FIM implements the pending bit array as per the MSI-X specification. When interrupts are enabled the FIM immediately generates an interrupt in response to a suitable event. When interrupt vectors are masked, the pending bit array entry for the corresponding interrupt vector is set without issuing an interrupt. When interrupt vectors are re-enabled, any pending interrupt entries are issued to the PCIe EP
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#94-interrupts-supported","title":"9.4 Interrupts Supported","text":"Table 9-1: OFS for Agilex 7 FPGA Interrupts Supported
PF/VF Vector Assignment PF0 0-5 Reserved 6 FME Error Interrupt PF0.VF00-3User AFU Interrupt"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#95-msi-x-table-locations","title":"9.5 MSI-X Table Locations","text":"The MSI-X vector tables are at BAR4, address 0x2000. The MSI-X PBA tables are at BAR4, address 0x3000.
Table 9-2: PF0 BAR4 Features
Offset Feature CSR set 0x02000 MSI-X Vector Tables 0x03000 MSI-X PBA Tables"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#10-external-memory-interface-emif","title":"10 External Memory Interface (EMIF)","text":"There are 4 EMIF channels (4 DDR4 banks) on the f2000x platform which is targeted OFS FIM for Agilex 7 FPGA. The HE-MEM exerciser module in AFU. ECC is not implemented in this design. Both memory subsystem and HE-MEM implement Arm\u00ae AMBA\u00ae 4 AXI4-MM interface.
**Table 10-1 Memory Subsystem Configuration on the IPU Platform F2000X-PL platform **
EMIF Channel # Width ECC Size (GB) Speed (MT/s) DDR4 Device FPGA Bank 0 x32 x8 4 2400 Three 1Gx16 3D 1 x32 x8 4 2400 Three 1Gx16 3A 2 x32 x8 4 2400 Three 1Gx16 2F 3 x32 x8 4 2400 Three 1Gx16 2E Figure 10-1: EMIF Interfaces
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#101-emif-csr","title":"10.1 EMIF CSR","text":"TheMIF is implemented as an external FME feature in OFS EA and the CSR for the EMIF feature is memory mapped to the FME BAR region. The following table captures the EMIF CSR registers.
Table 10-2: EMIF CSR Registers
EMIF_DFH 0x000 0x300000001000100F EMIF Management DFH FIELD NAME RANGE ACCESS DEFAULT DESCRIPTION FeatureType [63:60] RO 0x3 Feature Type = Private Feature Reserved40 [59:40] RsvdZ 0x0 Reserved NextDfhByteOffset [39:16] RO 0x1000 Next DFH Byte offset FeatureRev [15:12] RO 0x1 Feature Revision FeatureID [11:0] RO 0x00F Feature Id \u200b
EMIF_STAT 0x008 0x0000000000000000 EMIF Calibration Status Register FIELD NAME RANGE ACCESS DEFAULT DESCRIPTION Reserved [63:8] RsvdZ 0x0 Reserved CalFaliure [7:4] RO 0x0 EMIF PHY Calibration Failure (1 bit per interface) CalSuccess [3:0] RO 0x0 EMIF PHY Calibration Successful (1 bit per interface) EMIF_CAPABILI TY 0x010 0x000000000000000F EMIF Capabliity Register FIELD NAME RANGE ACCESS DEFAULT DESCRIPTION Reserved [63:4] RsvdZ 0x0 Reserved EMIFCap [3:0] RO 0xF Attached Memory Capability (1 bit per interface)"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#11-ethernet-interface-subsystem","title":"11 Ethernet Interface Subsystem","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#111-ethernet-interface-overview","title":"11.1 Ethernet Interface Overview","text":"The Ethernet Subsystem provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack. This reference FIM implements an E-tile Ethernet Subsystem IP in a 2x4x25GbE configuration and provides a Linux driver that can be leveraged for customization. Each group of 4x25GbE routes to a QSFP.
For more information about how to reconfigure the Ethernet Subsystem please refer to the [Ethernet Subsystem Intel FPGA IP].
Table 11-1: Ethernet Configurations for example OFS FIMs for Agilex 7 FPGAs
Configuration/Mode Base Fim IP file name hssi_ss_8x25g Number of ports enabled 8 Enabled ports Port 0-7 Port{x} profile 25GbE Port{x} subprofile MAC+PCS Port{x} RSFEC True Port{x} PTP False Enable AN/LT False (for simulation efficiency) Enable NPDME True Enable JTAG to Avalon master bridge False Enable Tx packet classifier NA PTP accuracy mode NA Enable iCAL and pCAL recipe at power True TX/RX maximum frame size 1518 Enforce maximum frame size False Link fault generation mode Bidirectional Stop TX traffic when link partner sends PAUSE Yes Bytes to remove from RX frames Remove CRC bytes Forward RX PAUSE requests False Use source address insertion False Enable TX/RX VLAN detection True Enable asynchronous adapter clocks False Enable preamble Passthrough False Enable asynchronous adapter clocks False Enable strict preamble check False Enable strict SFD check False Average IPG 12 Enable adaptation load soft IP True Additional IPG removed as per AM period 0 PMA adaptation select NRZ_28Gbps_VSR To determine which Transceiver Subsystem port maps to the QSFP A and B lanes, please refer to the following table:
Table 11-2: Transceiver Subsystem Port Mapping
Port Number base 2x4x25G 0 QSFP-A Lane-0 1 QSFP-A Lane-1 2 QSFP-A Lane-2 3 QSFP-A Lane-3 4 QSFP-B Lane-0 5 QSFP-B Lane-1 6 QSFP-B Lane-2 7 QSFP-B Lane-3 8 NA 9 NA 10 NA 11 NA 12 NA 13 NA 14 NA 15 NA Figure 11-1: Transceiver Subsystem Block Diagram
A host exerciser, named he-hssi, is provided in the pr_slot of the AFU partition. The Transceiver susbystem interface to the AFU has an Arm\u00ae AMBA\u00ae 4 AXI4-Stream data and sideband interface.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#112-ofs-ethernet-subsystem-interface-module","title":"11.2 OFS Ethernet Subsystem Interface Module","text":"A wrapper around the Ethernet Subsystem integrates the following features:
- DFH registers
- Control & status registers
- Indirect access to Transceiver SS CSR registers via CSR mailbox in the Ethernet Subsystem interface
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1121-ethernet-subsystem-control-and-status-register-csr-map","title":"11.2.1 Ethernet Subsystem Control and Status Register (CSR) Map","text":"The Ethernet Subsystem connects to the APF which is memory mapped to PF0 BAR0. The Ethernet Subsystem CSR space in the FIM consists of two parts:
- Ethernet Subsystem CSRs assigned from offset 0x000 to 0x7FF
- Additional CSRs are added to FIM at offset 0x800 to 0xFFF
The PCIe subsystem uses Arm\u00ae AMBA\u00ae 4 AXI4 Memory Mapped accesses to read and write Ethernet Subsystem Control and Status Registers in the FIM. The Ethernet Subsystem CSR Map structure is designed to scale according to IP capabilities.
The Ethernet Subsystem CSR Map can be found ipss/hssi/HSSI_SS_CSR.xls.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#114-ethernet-subsytem-software","title":"11.4 Ethernet Subsytem Software","text":"There are two pieces of software related to running the Ethernet Subsystem: The Linux* dfl network driver and the user space OPAE Tools.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1141-ethernet-subsystem-linux-driver","title":"11.4.1 Ethernet Subsystem Linux Driver","text":"The Ethernet subystem is exposed as a feature in the PCIe PF BAR0 region. It has a Device Feature Header (DFH) specifying the interface.
The primary functionality of the driver is to interact with the ethernet MAC and PHY through an indirect register access mailbox implemented by the HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers described above. To aid in RTL bringup, the driver provides a debugfs interface directly to the indirect register access mailbox. For each HSSI interface in the system there would be a directory with the following form containing two files, regaddr and regval: /sys/kernel/debug/dfl-fme.X.Y
To read a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then read the value as string out of regval file. To write a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then write the value as a C hex string to regval file.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1142-ethernet-subsystem-opae-user-space-tool","title":"11.4.2 Ethernet Subsystem OPAE User Space Tool","text":"User space OPAE Tools that are part of OPAE SDK provide support for the Ethernet Subsystem. You can use the --help option to print more information about each of these commands:
- hssi: Provides a means of interacting with the 10G and 100G HSSI AFUs through the host exerciser.
- hssiloopback: Enables and disables Ethernet loopback.
- hssistats: Provides MAC statistics.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#12-partial-reconfiguration","title":"12 Partial Reconfiguration","text":"Partial Reconfiguration (PR) is an Altera FPGA technology that allows users to reconfigure parts of the FPGA device dynamically, while the remainder of the device continues to operate. In a non-partial reconfiguration flow, any change to the design requires full reprogramming of the entire configuration RAM (CRAM) arrays in the device. With partial reconfiguration, you can dynamically reprogram one or more CRAM frames. A partial reconfiguration design has a static region, and a PR regions, which can be modified to implement new logic. The portion of the CRAM on the chip to be reconfigured is contained within a PR region. For the PR flow, your design must be partitioned into static region and reconfigurable region. The static region is the area of your FPGA that is not reconfigured without reprogramming the entire FPGA. An area of the chip that you plan to partially reconfigure is a PR region.
The Port Gasket contains all the PR specific modules and logic, such as PR slot reset/freeze control, user clock, remote STP etc. For this reference example only one PR slot is supported.
The Figure below shows the fundamental concepts required to support PR in OFS platform. There are 4 OFS management DFH \u2013 PR, Port, User Clock and Remote STP in Port Gasket that is exposed to OFS software. These platform capabilities are generally used in conjunction to Partial Reconfiguration. The Figure below also demonstrates the concepts of adding a new interface to the PR region.
Figure 12-1 Partial Reconfiguration Gasket
The isolation logic is provided on the output signals of the PR region to ensure they don\u2019t glitch and affect the interfacing logic in the Static Region (SR). The isolation logic is controlled by the PR Freeze logic during PR operation.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#121-user-clock-support","title":"12.1 User Clock Support","text":"The reference platform provides two user configurable clock (uclk_usr, uclk_usr_div2) for use by the AFU. These clocks are generated by a reconfigurable IOPLL. The control and status of these clocks is expose through two pairs of command and status registers (USR_CLK_FREQ_CMD0 / USR_CLK_FREQ_STS0 & USR_CLK_FREQ_CMD1 / USR_CLK_FREQ_STS1). The first pair controls the fPLL reconfiguration functionality. The second pair controls a clock frequency measurement block.
The following are the default values of the userclk.
uclk_usr: 312.5 MHz
uclk_usr_div2: 156.2 MHz
Table 12-1 usr_clk_* Acceptable Programming Range
Fabric Frequency Range uclk_usr (H) uclk_usr_div2 (L) Details 0-400 MHz 0-800 MHz 0-400 MHz Clocks set on 2x relationship for L<400 MHz 400-800 MHz 400-800 MHz 400-800 MHz Clks are equal for L>400 MHz PLL Reconfiguration
The blue bit stream hardware exposes the low level IOPLL reconfiguration interfaces directly to software control. Through the USR_CLK_FREQ_CMD0 register software can select the reference clock, assert the PLL power down pin and issue reads and writes on the IOPLL Avalon-mm reconfiguration bus. Through the USR_CLK_FREQ_STS0 register software can query the IOPLL active reference clock, locked status and readdata returned from the IOPLL AVMM interface for read requests.
Clock Frequency Counter
The user clocks, generated by the reconfigurable IOPLL, are connected to a frequency measurement circuit. Software selects which of the two clocks to measure by setting the clock select bit in the USER_CLK_FREQ_CMD1 register. After 10ms software can read the frequency, in 10 KHz units, from the USER_CLK_FREQ_STS1 register. Reading this register before 10ms has passed will return undefined data but otherwise has no effect.
Configuring User Clocks
To program the user clock to the desired frequency, user will set the clock-frequency-low and clock-frequency-high fields in the PR AFU GBS .json file to the desired frequency value. During PR, SW will try to provide the closest possible frequency to the value specified in the .json file.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#13-host-exercisers","title":"13 Host Exercisers","text":"The Host Exerciser (HE) modules are responsible for generating traffic with the intent of exercising the specific interface they are designed to connect to. They are intended to test the interface in simulation and hardware, enable measurement of bandwidth and other performance KPIs and, in some cases, provide an example of data movement between interfaces (PCIe to DDR for e.g.) for adoption into a customer application.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#131-he-lb-and-he-mem-host-exerciser","title":"13.1 HE-LB and HE-MEM Host Exerciser","text":"The Host Exerciser Loopback module is a traffic generator that can be attached both to host memory, over PCIe (HE-LB), and to local memory, such as board-attached DDR (HE-MEM). The Host Exerciser (HE) is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth as well as demonstrating data path correctness. The FIM picks up the HE-LB module behind the PIM (Platform Interface Manager). The PIM converts the Arm\u00ae AMBA\u00ae 4 AXI4 with TLP out of the PCIe SS to standard Arm\u00ae AMBA\u00ae 4 AXI4 (MM for completions and Lite for CSR) interfaces. The figure below shows how the HE-LB is instantiated in the FIM.
Figure 13-1 HE-LB Dataflow Diagram
The exerciser has two primary modes: loopback, in which read data is fed back as writes, and read/write mode, in which reads and writes are generated independently. Furthermore, the host_exerciser software provided with OPAE that drives the RTL has two major modes: \"lpbk\" and \"mem\". These modes only select the UUID of the HE AFU, with lpbk selecting a version configured with no local memory (56e203e9-864f-49a7-b94b-12284c31e02b) and mem seeking a version with local memory (8568ab4e-6ba5-4616-bb65-2a578330a8eb). The behavior of each engine with and without local memory is described below.
Figure 13-2 HE Engine Heirarchy
flowchart TB\n classDef gr fill:green,color:white;\n classDef ol fill:olive,color:white;\n classDef rd fill:maroon,color:white;\n\n he_lb(\"he_lb_top\"):::gr --- he_lb_main\n he_mem(\"he_mem_top\"):::gr --- he_lb_main\n he_lb_main:::gr ---- he_lb_engines\n he_lb_engines:::gr ---- mode_rdwr:::rd\n he_lb_engines:::gr ---- mode_lpbk:::rd\n he_lb_engines:::gr ---- mode_atomic:::rd\n he_lb_main --- he_lb_csr:::ol
For more details of HE-LB and HE-MEM IP, please refer to ofs-fim-common/src/common/he_lb/README.md
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#132-memory-traffic-generator-he-mem-tg","title":"13.2 Memory Traffic Generator (HE-MEM-TG)","text":"The memory traffic generator (TG) AFU provides a way for users to characterize local memory channel bandwidth with a variety of traffic configuration features including request burst size, read/write interleave count, address offset, address strobe, and data pattern.
The AFU consists of a series of configurable Arm\u00ae AMBA\u00ae 4 AXI4-MM traffic generators connected to the available local memory channels not attached to HE-MEM. It has a separate CSR block for AFU feature information and high-level status, including a TG_CAPABILITY register that describes the available traffic generators with a 1 bit active high mask and a TG_STAT register that provides the 4-bit, one-hot status of each TG in adjacent fields. The status is decoded left to right as follows: pass, fail, timeout, active. For additional detail, refer to the MEM_TG CSR table ofs-fim-common/src/common/mem_tg/MEM_TG_CSR.xls
Each traffic generator is configured through a separate Avalon-MM interface at incremental offsets of 0x1000 from the AFU DFH. For details on the TG2 configuration space, refer to the MEM_TG_CSR.xls. The default configuration for each TG performs a single-word write followed by a read at address 0. Triggering the start of the test on a TG will initiate a counter to measure the duration of the test which is recorded in the AFU CSR block and used to report memory channel bandwidth.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#132-hssi-host-exerciser-he-hssi","title":"13.2 HSSI Host Exerciser (HE-HSSI)","text":"HE-HSSI is an Ethernet AFU that handles client side ethernet traffic. The reference HE-HSSI has following features:
- HE-HSSI provides an E-tile compatible interface with the Transceiver Subsystem.
- Includes a traffic generator and checker or monitor
- Provides pause signals to the Transceiver Subsystem for XON and XOFF generation
- Generates traffic or incoming traffic that can be looped back into transmit path by enabling loopback mode, which will bypass traffic generator
- At the HE-HSSI interface boundary the Ethernet data interface is Arm\u00ae AMBA\u00ae 4 AXI4-Stream with 64-bit data at eth_clk clock
- The Traffic generator and checker modules have a 64-bit data interface at eth_clk clock.
- The traffic generator supports the following modes:
- Fixed length or Random Length
- Incremental pattern or Random pattern
- Traffic checker does a 32-bit crc check in 10/25G. In the 100G configuration, there is no data integrity check, only a packet counter.
- The CSR of this AFU is accessible through Arm\u00ae AMBA\u00ae 4 AXI4-Stream PCIe TLP interface
- The PCIe TLP to CSR Interface Conversion module converts PCIe TLPs into simple CSR interface
- The CSR space of the traffic generator and checker modules are accessed in an indirect way using mailbox registers
- If used for more than one channel, each channel has a separate traffic generator and traffic checker with separate CSR space.
- Reads and Writes to individual traffic controller CSR spaces can be done by selecting that particular channel using channel select register.
The HE-HSSI Ethernet block diagram is below.
Figure 13-3: HE-HSSI Block Diagram Block Diagram
The diagram below shows the path followed depending on if you enable loopback mode in HE-HSSI or not. In loopback mode, traffic is looped back into the transmit path which will bypass traffic. Alternatively, the traffic generates traffic.
Figure 13-4: HE-HSSI Operation Mode Traffic Patterns
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1322-he-hssi-csr-map","title":"13.2.2 HE-HSSI CSR Map","text":"The reference HSSI AFU contains the following registers and a similar arrangement of register space can be implemented for other use case-specific HSSI AFUs.
- AFU DFH Register: Device feature header for the AFU (AFU_DFH)
- AFU ID Registers: 128-bit UUID for the AFU which occupies two 64-bit registers (AFU_ID_L, AFU_ID_H)
- Mailbox Registers: Command and Data register for traffic controller register access. It follows the standard access method defined for OFS. Access method and implementation is same as Reconfiguration Interface defined for the HSSI FIM. (TRAFFIC_CTRL_CMD, TRAFFIC_CTRL_DATA)
- Channel Select Register: Channel select register for traffic controller mailbox access. It is used in cases where more than one channel is in the AFU, else it defaults to zero, meaning channel-0 is selected. (TRAFFIC_CTRL_PORT_SEL)
- Scratchpad Register: Scratchpad register for CSR access checking. (AFU_SCRATCHPAD)
The CSR excel for HE-HSSI module can be found at ofs-fim-common/src/common/he_hssi/HE_HSSI_CSR.xls.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#133-he-null-overview","title":"13.3 HE-Null Overview","text":"This module is a simple stub that is used to replace various HE and other blocks in the FIM whenever they are bypassed using the qsf compiler directive such as null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. To find out more about these compiler directives, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
Table 13-1 HE-Null DFH
REGISTER NAME ADDRESS ACCESS DEFAULT DESCRIPTION DFH 0x0000 RO 0x0000_0000_1000_0000 DFH register GUID_L 0x0008 RO 0xaa31f54a3e403501 Lower 64b of GUID GUID_H 0x0010 RO 0x3e7b60a0df2d4850 Upper 64b of GUID SCRATCHPAD 0x0018 RW 0x0 Scratchpad Figure 13-5: HE-Null Block Diagram
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14-reliability-accessibility-serviceability-ras-and-error-handling","title":"14 Reliability, Accessibility, Serviceability (RAS) and Error Handling","text":" - Downstream AFU checker: Identifies AFU violations. For example, these checker flags violations of the interface specification.
- Upstream software or PCIe link checker: Identifies invalid traffic from PCIe that violates FIM or PCIe specifications. For example, this checker flags an application sending traffic if it violates the FIM specification or creates a PCIe link issue by causing completion timeout or malformed TLP.
- FIM - Checks for bugs in the FIM fabric.
Errors reported by the checker are logged in either the FME error registers port error registers, or both, as shown in the table below. For more details on each of the registers, please refer to FME_CSR.xls
Table 14-1: Error Registers
MMIO Region Area Register Description FME CoreFIM FME_ERROR FME Error Status Register 0. Registers parity errors, underflow or overflow errors and access mismatches. FME CoreFIM FME_ERROR0_MASK FME Error Mask Register 0. Write a 0 to mask errors in the FME Error Status Register 0. FME External PCIE0_ERROR PCIe0 Error Status Register. FME External PCIE0_ERROR_MASK PCIe0 Error Mask Register 0. Write a 0 to mask errors in the PCIe0 Error Status Register 0. FME CoreFIM FME_FIRST_ERROR First FME Error Register. FME CoreFIM FME_NEXT_ERROR FME Next Error Register. FME CoreFIM RAS_NOFAT_ERR_STAT Reliability/Accessibility/Serviceability (RAS) Non-Fatal Error Status Register. FME CoreFIM RAS_NOFAT_ERR_MASK RAS Non-Fatal Error Mask Register. Write 0 to mask error fields in RAS_NOFAT_ERR_STAT Register. FME CoreFIM RAS_CATFAT_ERR_STAT RAS Catastrophic and Fatal Errors Status Register. FME CoreFIM RAS_CATFAT_ERR_MASK RAS Catastrophic and Fatal Errors Mask Register. Write 0 to mask error fields in the RAS_CATFAT_ERR_STAT Register. FME CoreFIM RAS_ERROR_INJ RAS error Injection Register. PORT CoreFIM PORT_ERROR Port Error Status Register. PORT CoreFIM PORT_FIRST_ERROR Port First Error Register . PORT CoreFIM PORT_MALFORMED_REQ0 Port Malformed Request Register 0. Provides malformed request header LSBs. PORT CoreFIM PORT_MALFORMED_REQ1 Port Malformed Request Register 1. Provides malformed request header MSBs."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#141-fme-errors","title":"14.1 FME Errors","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1411-fme_error0","title":"14.1.1 FME_ERROR0","text":"The FME_ERROR0 register flags CoreFIM FME errors in the Global Error (GLBL_ERROR) private feature. The error bits in this register are sticky bits. You can only clear these bits through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in FME_ERROR0_MASK register masks the error.
Table 14-2: FME Error Types
Error Type Description Fabric errors FIFO underflow/overflow condition in CoreFIM. These errors only occur if you have introduced bugs into the FIM or during very rare single event upset (SEU) or SEU-like events. Invalid port access A port can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the Port. If it finds a PF is trying to access a port that is mapped to a VF or vice-versa, an error will be reported. Invalid AFU access An AFU can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the AFU associated with the Port. If it finds a PF is trying to access an AFU that is mapped to a VF or vice-versa, an error is reported and a fake response is sent back to the requester to avoid a completion timeout on the host."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1412-pcie0_error","title":"14.1.2 PCIE0_ERROR","text":"The PCIe Avalon-ST to Arm\u00ae AMBA\u00ae 4 AXI4-Stream bridge monitors the PCIe link for errors and logs any such errors in the PCIE0_ERROR register (in PCIE0 feature region) and PCIE0_ERROR register in the GLBL_ERR private feature. The error bits in the PCIE0_ERROR register are sticky bits that you can only clear through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in PCIE0_ERROR0_MASK masks the error.
If you have other external FME features, you can add similar _ERROR registers to this space. Please refer to the following spreadsheet: https://github.com/OFS/ofs-f2000x-pl/ipss/pcie/rtl/PCIE_CSR.xls or the SystemVerilog file: https://github.com/OFS/ofs-f2000x-pl/ipss/pcie/rtl/pcie_csr.sv for more details on this register.
NOTE
The PCIE0_ERROR register is located in both the Global Error external feature memory space and a separate PCIe external feature memory space. OPAE software supports the PCIe external feature memory space beginning at offset 0x40000 for OFS EA and going forward. PCIe registers beginning at 0x4000 in the Global Error external feature memory space are there for backward compatibility to the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1413-fme_first_error-fme_next_error","title":"14.1.3 FME_FIRST_ERROR, FME_NEXT_ERROR","text":"The FME_FIRST_ERROR register flags which of the FME error reporting registers, such as FME_ERROR0, PCIE0_ERROR0, has reported the first error occurrence. The error fields of the first error register are then continuously logged into the FME_FIRST_ERROR register until a system reset or software clears all the errors in that first error register. Likewise, the FME_NEXT_ERROR indicates which of the FME error reporting registers (except the first error register) has reported the next occurrence of error after the first error register. The error fields of the next error register are continuously logged into the FME_NEXT_ERROR register until a system reset or software clears all the errors in the second error register.
Please refer to fme_csr.sv for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1414-reliability-accessibility-serviceability-ras-error-status","title":"14.1.4 Reliability, Accessibility, Serviceability (RAS) Error Status","text":"The RAS feature in CoreFIM labels errors as non-fatal, fatal or catastrophic based on their impact to the system. * A non-fatal error usually originates from software or an AFU. With a non-fatal error, the user application may still be able to recover from the error by performing a soft reset on the AFU, fixing the user application software if necessary, and clearing the error. On the other hand, a fatal or catastrophic error is non-recoverable and requires the platform to be reset. * Non-fatal errors are logged in the RAS_NOFAT_ERR_STAT register and fatal or catastrophic errors are logged in the RAS_CATFAT_ERR_STAT register.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14141-non-fatal-errors","title":"14.1.4.1 Non-Fatal Errors","text":"The RAS_NOFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It logs the high-level status of non-fatal errors in the hardware. Unlike the error bits in the PCIE0_ERROR and FME_ERROR0 registers which are RW1C (software can write a 1 to clear the error), the error bits in this register are read-only and can only be cleared by system reset. Software has an option to mask the error using RAS_NOFAT_ERR_MASK.
Please refer to FME_CSR.xls for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14142-catastrophic-fatal-errors","title":"14.1.4.2 Catastrophic & Fatal Errors","text":"The RAS_CATFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It captures the high-level status of errors that can only be recovered with a system reset. Therefore, the error bits in the RAS_CATFAT_ERR_STAT register are read-only and can only be cleared by system reset or masked through RAS_CATFAT_ERR_MASK. Please refer to FME_CSR.xls for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1415-ras-error-injection","title":"14.1.5 RAS Error Injection","text":"For software testing purposes, you can inject non-fatal, fatal, and catastrophic errors into the platform through the RAS_ERROR_INJ register. These errors are reported in the RAS_CATFAT_ERR_STAT and RAS_NOFAT_ERR_STAT registers. Please refer to FME_CSR.xls for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1416-fme-error-interrupts","title":"14.1.6 FME Error Interrupts","text":"In an event of an FME error, the MSI-X module in the FIM generates an interrupt so the host can decide on the next course of action. The FIM does not stall upstream and downstream traffic after the FME error. However, a port error triggered by invalid request from a user AFU stalls all the traffic going from AFU to PCIe. The interrupt capability is discoverable by querying the NumbSuppInterrupt field of the PORT_CAPABILITY register in the Port private feature. The MSI-X vector number is recorded in the InterruptVectorNumber field of the GLBL_ERROR_CAPABILITY register of the Global Error external feature.
An FME error interrupt is generated in response to the FME_ERROR0, PCIE0_ERROR0, RAS_NOFAT_ERR_STAT, or RAS_CATFAT_ERR_STAT registers recording a new, unmasked, error per the rules defined by CoreFIM interrupt feature.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14161-msi-x-masking-pending-bit-array-pba-clearing","title":"14.1.6.1 MSI-X Masking & Pending Bit Array (PBA) Clearing","text":"If the MSI-X vector corresponding to the FME error interrupt is masked, events are recorded in the PBA array. Clearing the FME error status registers clears the corresponding MSI-X PBA entries. If only some events are cleared, the normal interrupt triggering rules apply and a new pending interrupt is registered.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1417-fme-error-handling","title":"14.1.7 FME Error Handling","text":"When the host receives an FME error interrupt, it must take the recommended actions described below to bring the system back to its normal operation.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14171-catastrophicfatal-error","title":"14.1.7.1 Catastrophic/Fatal Error","text":"A system reset is mandatory for any catastrophic or fatal error.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14172-non-fatal-error","title":"14.1.7.2 Non-Fatal Error","text":"When software receives a non-fatal error interrupt which does not require a system reset, it can take the following steps to clear the error after software handles the error: 1. Set the *_ERROR_MASK register to all 1\u2019s to mask all errors 2. Clear the *_FIRST_ERROR register 3. Clear the *_ERROR register 4. Set *_ERROR_MASK register to all 0\u2019s to enable all errors
- Result: The *_ERROR & *_FIRST_ERROR registers begin capturing new errors.
NOTE
A system reset can only clear RAS Error status registers.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#143-mmio-requests","title":"14.3 MMIO Requests","text":"The FIM is designed to gracefully handle MMIO request scenarios.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1431-unsupported-functions-and-bar","title":"14.3.1 Unsupported Functions and BAR","text":"The PCIe hard IP in the FIM guarantees that only TLP packets for the functions and BARs supported by the FIM (as configured in PCIe HIP IP instantiation) are sent to the FIM.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1432-mmio-request-decoding","title":"14.3.2 MMIO Request Decoding","text":"The packet router and memory decoder in the FIM ensure that only legal MMIO requests are forwarded to the targeted MMIO region. Full address and BAR decoding is done both in the packet router and the memory decoder to ensure the requests are forwarded to the designated CSR region as defined in the MMIO Regions chapter. Any unsolicited/illegal MMIO request is dropped, and an error is reported back to host through the FME error register.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1433-unused-fmeport-csr-regions","title":"14.3.3 Unused FME/Port CSR Regions","text":"All the CSR slaves in FIM which are mapped to the FME or Port CSR regions must always respond to MMIO read requests targeting its associated CSR region. A CSR slave must return all 0s for MMIO reads to its unused CSR region such as a reserved space or a region where no CSR register is implemented for the address.
The FIM ensures MMIO reads to FME or Port CSR regions that are not mapped to any CSR slave always gets a response of all 0s. The memory decoder module and fake responder module in the FIM provide this guaranteed response.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1434-unsupported-mmio-request","title":"14.3.4 Unsupported MMIO Request","text":"Any MMIO request targeting FME or Port CSR regions with a length or address alignment that are not supported by the FIM is dropped, and an error is logged in PCIE0_ERROR register. The MMIO checker module in the FIM guarantees this response. When an unsupported MMIO read request to the FIM CSR region is detected, the FIM sends back a CPL (completion without data) with error status (UR) back to host.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1435-afu-access-violation","title":"14.3.5 AFU Access Violation","text":"AFU access violations refer to the scenarios where a PF is attempting to access the MMIO region of an AFU bound to a VF (virtualization enabled), or when a VF is trying to access the MMIO region of an AFU bound to a PF (virtualization disabled). When such a violation is detected, the FIM drops the request and logs the error in the FME_ERROR0 register. If the request is an MMIO read request, the FIM returns a fake response to the host.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1436-afu-mmio-response-timeout","title":"14.3.6 AFU MMIO Response Timeout","text":"An AFU MMIO Response timeout functions in the same manner described in the MMIO Response Timeout section.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#15-ofs-design-hierarchy","title":"15 OFS Design Hierarchy","text":"Files for design, build and unit test simulation are found at https://github.com/OFS/ofs-f2000x-pl, release branch ofs-2024.1-1.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#151-design-guidance","title":"15.1 Design Guidance","text":"The OFS FIM is designed with configurability and scalability in mind. At a high level, these are the necessary steps for a user to customize the design. Please refer to the [FPGA Interface Manager Developer's Guide]
Table 15-2 Features
Step Description Comments 1 Re-configure PCIe HIP for additional VFs (if necessary) * PF0 is mandatory for running OPAE software* Only modification of the VFs (added or subtracted) by the user is recommended. * Default configuration supports 1 PF and 3 VFs. 2 Update Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF MUX-DEMUX configuration (if necessary) * The PF/VF MUX-DEMUX is parameterized for flexibility. You can change, add or delete PF or VF functions by updating the top_cfg_pkg.sv file. * You also have the option of keeping the default configuration and tying off the unused VFs if needed. 3 Update top level and AFU level as necessary * If you integrating additional external interfaces, make the edits at the top level (ofs_top.sv) and propagate the interface down to the AFU level (afu_top.sv and soc_afu_top.sv) 4 Add user implemented function(s) in AFU * All of your implemented functions must have the required Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface for both the data path and the MMIO control path to CSRs. * All CSRs in the user-implemented function must have the required DFH layout. * See host exerciser CSRs for reference. 5 Update UVM testbench * The OFS full chip UVM environment is coded specifically for verifying the default configuration containing the host exercisers for the PCIe, memory, and Ethernet. * User must update the UVM testbench to reflect new RTL behavior for any customization changes. See The UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach For more information on modifying the FIM, refer to the [FPGA Interface Manager Developer's Guide]
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/","title":"Platform Evaluation Script: Open FPGA Stack for Agilex 7 SoC Attach FPGAs","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#1-overview","title":"1 Overview","text":""},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the checkout and evaluation of an IPU Platform F2000X-PL development platform using Open FPGA Stack (OFS). After reviewing the document, you will be able to:
- Set-up and modify the script to the your environment
- Compile and simulate an OFS reference design
- Run hardware and software tests to evaluate the complete OFS flow
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#table-1-2-software-version-summary","title":"Table 1-2: Software Version Summary","text":"Component Version Description FPGA Platform Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL Agilex platform you can use for your custom board development OFS FIM Source Code Branch: https://github.com/OFS/ofs-f2000x-pl, Tag: ofs-2024.1-1 OFS Shell RTL for Agilex 7 FPGA (targeting https://github.com/OFS/ofs-f2000x-pl) OFS FIM Common Branch: https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.1-1, Tag: https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.1-1 Common RTL across all OFS-based platforms AFU Examples Branch: examples-afu , Tag:ofs-2024.1-1 Tutorials and simple examples for the Accelerator Functional Unit region (workload region) OPAE SDK Branch: 2.12.0-5, Tag: 2.12.0-5 Open Programmable Acceleration Engine Software Development Kit Kernel Drivers Branch: ofs-2024.1-6.1-2, Tag: ofs-2024.1-6.1-2 OFS specific kernel drivers OPAE Simulation Branch: opae-sim, Tag: 2.12.0-5 Accelerator Simulation Environment for hardware/software co-simulation of your AFU (workload) Quartus Prime Pro Edition Design Software 23.4 Quartus\u00ae Prime Pro Edition Linux Software tool for Altera FPGA Development Operating System Ubuntu 22.04 Operating system on which this script has been tested A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL can be found on the OFS ofs-2024.1-1 official release drop on GitHub.
Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#2-introduction-to-ofs-evaluation-script","title":"2 Introduction to OFS Evaluation Script","text":"By following the setup steps and using the OFS evaluation script you can quickly evaluate many features that the OFS framework provides and also leverage this script for your own development.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#21-pre-requisites","title":"2.1 Pre-Requisites","text":"This script uses the following set of software tools which should be installed using the directory structure below. Tool versions can vary.
- Quartus\u00ae Prime Pro Software : The software can be downloaded here. For details on installation of required patches and quartus installation, refer to section 4.2 of the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
- Synopsys\u00ae VCS Simulator : The simulator can be downloaded in the Synopsys page here
- Siemens\u00ae Questa\u00ae Simulator : The simulator can be downloaded in the Siemens page here
Figure 2-1 Folder Hierarchy for Software Tools
- Download tar(scripts_ofs-2024.1-1_external.tar.gz) from the assets tab of the release page
-
Untar to folder using the following command
tar -xvf scripts_ofs-2024.1-1_external.tar.gz\n````\n\n3. The folder structure containing the clone script (ofs-clone_repositories.sh) and evaluation script (ofs_f2000x_eval.sh) is as shown in the figure 2-2\n\n**Figure 2-2 Directory Structure for the clone script**\n\n``` sh\n## ofs_scripts\n## -> host_chan_mmio.stp\n## -> ofs_f2000x_eval.sh\n## -> README_ofs_f2000x_eval.txt\n## ofs-clone_repositories.sh\n
-
Open the clone script ofs-clone_repositories.sh in any text editor
- Please enter the location of your proxy server to allow access to external internet to build software packages. (lines 6-8)
Note: Failing to add proxy server will prevent cloning of repositories and the user will be unable to build the OFS framework.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
-
Please enter the license file locations (lines 11-13) for the following tool variables
export LM_LICENSE_FILE=<user_license>\n export DW_LICENSE_FILE=<user_license>\n export SNPSLMD_LICENSE_FILE=<user_license>\n
-
Add the name of the directory where you want the platform repositories to be cloned (line 19)
export OFS_RELEASE=ofs-2024.1-1\n
-
After setting the required variables, source the clone script based on which platform you want to test
source ofs-clone_repositories.sh\n
- The script ofs-clone_repositories.sh has different platforms to choose from the menu to clone as shown in the figure 2-3.
Figure 2-3 Directory Structure for OFS Project
-
Once the repositories are cloned, navigate to the directory where you cloned
cd ofs-2024.1-1\n
-
After cloning, the OFS repositories cloned will look as per Figure 2-4.
Figure 2-4 Directory Structure for OFS Project
## ofs-2024.1-1\n## f2000x(OFS platform)\n## -> examples-afu\n## -> ofs-f2000x-pl\n## -> oneapi-asp\n## -> oneAPI-samples\n## -> opae-sim\n## -> opae-sdk\n## -> linux-dfl\n## -> ofs_f2000x_eval.sh\n## -> README_ofs_f2000x_eval.txt\n## -> host_chan_mmio.stp\n
- Once the repositories are cloned, in the platform specific evaluation script, for e.g., in ofs_f2000x_eval.sh, please follow the instructions below which explains which line numbers to change to adapt this script to the user environment.
a) Set-Up Proxy Server (lines 67-69)
Please enter the location of your proxy server to allow access to external internet to build software packages.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
b) Tools Location (line 87, 88, 89, 90)
Set Location of Quartus, Synopsys, Questasim and oneAPI Tools
export QUARTUS_TOOLS_LOCATION=/home\nexport SYNOPSYS_TOOLS_LOCATION=/home\nexport QUESTASIM_TOOLS_LOCATION=/home\nexport ONEAPI_TOOLS_LOCATION=/opt\n
In the example above /home is used as the base location of Quartus, Synopsys and Questasim tools, /opt is used for the oneAPI tools
c) PCIe (Bus Number)
The Bus number must be entered by the user after installing the hardware in the chosen server, in the example below \"b1\" is the Bus Number for a single card as defined in the evaluation script.
export OFS_CARD0_BUS_NUMBER=b1\n
The evaluation script uses the bus number as an identifier to interrogate the card. The command below will identify the accelerator card plugged into a server.
lspci | grep acc\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\nb1:00.1 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.2 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\nb1:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
The result identifies the card as being assigned \"b1\" as the bus number so the entry in the script changes to
export OFS_CARD0_BUS_NUMBER=b1\n
The user can also run the following command on the ofs_f2000x_eval.sh script to automatically change the bus number to b1 in the ofs_f2000x_eval.sh script.
grep -rli 'b1' * | xargs -i@ sed -i 'b1' @
if the bus number is 85 for example
85:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\n85:00.1 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.2 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\n85:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
the command to change to 85 in the evaluation script would be
grep -rli 'b1' * | xargs -i@ sed -i '85' @
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#ofs-platform-example-n6000-n6001-fseries-dk-iseries-dk-custom_board-choice-line-173","title":"OFS Platform (Example:= n6000, n6001, fseries-dk, iseries-dk, custom_board) choice (line 173)","text":"The script is designed to accommodate many OFS platform choices eg, n6000, n6001, fseries-dk, iseries-dk or a custom_board of your choice. By default the script defaults to n6001 as shown below and is set by a variable on line 173 of the ofs_f2000x_eval.sh script.
export OFS_PLATFORM=n6001\n
but the user can switch platform by changing the OFS_PLATFORM variable, so for example if the user wants to switch to the i-series development kit the command would be
export OFS_PLATFORM=iseries-dk\n
The ofs_f2000x_eval.sh script has now been modified to the server set-up and the user can proceed to build, compile and simulate the OFS stack
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#3-f2000x-evaluation-script","title":"3 f2000x Evaluation Script","text":""},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#31-overview","title":"3.1 Overview","text":"The figure below shows a snapshot of the full evaluation script menu showing all 62 options and each one one of 11 sub-menus which focus on different areas of evaluation. Each of these menu options are described in the next section.
Figure 3-1 ofs_f2000x_eval.sh Evaluation Menu
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#311-ofs-tools-menu","title":"3.1.1 OFS TOOLS MENU","text":"By selecting \"List of Documentation for OFS f2000x Project,\" a list of links to the latest OFS documentation appears. Note that these links will take you to documentation for the most recent release which may not correspond to the release version you are evaluating. To find the documentation specific to your release, ensure you clone the OFS tag that corresponds to your OFS version.
By selecting \"Check Versions of Operating System and Quartus Premier Design Suite\", the tool verifies correct Operating System, Quartus version, kernel parameters, license files and paths to installed software tools.
Menu Option Example Output 1 - List of Documentation for OFS f2000x Project Open FPGA Stack Overview Guides you through the setup and build steps to evaluate the OFS solution https://ofs.github.io 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS) Checking Linux release Linux version 5.15.92-dfl-66b0076c2c-lts (oe-user@oe-host) (x86_64-ese-linux-gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Wed Feb 8 21:21:27 UTC 2023 Checking RedHat release cat: /etc/redhat-release: No such file or directory Checking Ubuntu release cat: /etc/lsb-release: No such file or directory Checking Kernel parameters initrd=\\microcode.cpio LABEL=Boot root=PARTUUID=2beb0add-07bf-4736-bc31-9b60e7b78375 rootfstype=ext4 console=ttyS5,115200 iomem=relaxed intel_iommu=on pci=realloc default_hugepagesz=1G hugepagesz=1G hugepages=4 rootwait console=ttyS0,115200 console=tty0 Checking Licenses LM_LICENSE_FILE is set to port@socket number:port@socket number DW_LICENSE_FILE is set to port@socket number:port@socket number SNPSLMD_LICENSE_FILE is set to port@socket number:port@socket number Checking Tool versions QUARTUS_HOME is set to /home/intelFPGA_pro/23.4/quartus QUARTUS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus IMPORT_IP_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../ip QSYS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../qsys/bin Checking QPDS Patches Quartus Prime Shell Version 23.4 Build XXX XX/XX/XXXX Patches X.XX SC Pro Edition Copyright (C) XXXX Intel Corporation. All rights reserved."},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#312-ofs-hardware-menu","title":"3.1.2 OFS HARDWARE MENU","text":"Identifies card by PCIe number, checks power, temperature and current firmware configuration.
Menu Option Example Output 3 - Identify Acceleration Development Platform (OFS) f2000x Hardware via PCIe PCIe card detected as 15:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) Host Server is connected to SINGLE card configuration 4 - Identify the Board Management Controller (BMC) Version and check BMC sensors Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** BMC SENSORS ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e 5 - Identify the FPGA Management Engine (FME) Version Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** FME ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e Boot Page : user1 User1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899 User2 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899 Factory Image Info : None 6 - Check Board Power and Temperature Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** POWER ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e ( 1) QSFP (Primary) Supply Rail Voltage : N/A etc ...................... Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** TEMP ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e ( 1) FPGA FABRIC Remote Digital Temperature#3 : 33.00 Celsius etc ...................... 7 - Check Accelerator Port status //****** PORT ******// Object Id : 0xEF00000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 8 - Check MAC and PHY status Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** MAC ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e Number of MACs : 255 mac info is not supported Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** PHY ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e QSFP0 : Connected QSFP1 : Connected //****** HSSI information ******// HSSI version : 1.0 Number of ports : 8 Port0 :25GbE DOWN Port1 :25GbE DOWN Port2 :25GbE DOWN Port3 :25GbE DOWN Port4 :25GbE DOWN Port5 :25GbE DOWN Port6 :25GbE DOWN Port7 :25GbE DOWN"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#313-ofs-pfvf-mux-and-ofss-menu","title":"3.1.3 OFS PF/VF MUX and OFSS MENU","text":"This menu reports the number of PF/VF functions in the reference example and also allows you to reduce the number to 1PF and 1VF to reduce resource utilisation and create larger area for your workload development. This selection is optional and if the user wants to implement the default number of PF's and VF's then option 9, 10 and 11 should not be used. Additionally the code used to make the PF/VF modification can be leveraged to increase or modify the number of PF/VFs in the existing design within the limits that the PCIe Subsystem supports (8PF/2K VFs)
Menu Option Description 9 - Check PF/VF Mux and OFSS Configuration for OFS f2000x Project This menu selection displays the current configuration of all f2000x ofss files located in the following directory $OFS_ROOTDIR/tools/ofss_config Check f2000x base config OFSS set up [ip] type = ofs [settings] platform = f2000x fim = base_x16 family = agilex part = AGFC023R25A2E2VR0 device_id = 6100 Check f2000x hssi_2x100 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GCAUI-4 Check f2000x hssi_2x100_caui2 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GAUI-2 Check f2000x hssi_8x10 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 10GbE Check f2000x hssi_8x25 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 25GbE Check f2000x IOPLL OFSS set up [ip] type = iopll [settings] output_name = sys_pll instance_name = iopll_0 [p_clk] freq = 470 Check f2000x Memory OFSS set up [ip] type = memory [settings] output_name = mem_ss_fm preset = f2000x Check f2000x PCIe Host OFSS set up [ip] type = pcie [settings] output_name = pcie_ss [pf0] bar0_address_width = 21 [pf1] Check f2000x PCIe SoC OFSS set up [ip] type = pcie [settings] output_name = soc_pcie_ss [pf0] num_vfs = 3 bar0_address_width = 21 vf_bar0_address_width = 21 10 - Modify PF/VF Mux and OFSS Configuration for OFS f2000x Project As an example this menu selection modifies the pcie_host.ofss and pcie_soc.ofss file to 1 PF located in the following directory $OFS_ROOTDIR/tools/ofss_config/pcie This option also displays the the modified pcie_host.ofss and pcie_soc.ofss files 11 - Build PF/VF Mux and OFSS Configuration for OFS f2000x Project If option 10 is not used then the default number of PF's and VF's is used to build the FIM, if option 10 is selected then only 1 VF is built to reduce logic utilisation"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#314-ofs-fimpr-build-menu","title":"3.1.4 OFS FIM/PR BUILD MENU","text":"Builds FIM, Partial Reconfiguration Region and Remote Signal Tap
Menu Option Description 12 - Check OFS software versions for OFS f2000x Project OFS_ROOTDIR is set to /home/user_area/ofs-X.X.X/ofs-f2000x-pl OPAE_SDK_REPO_BRANCH is set to release/X.X.X OPAE_SDK_ROOT is set to /home/user_area/ofs-X.X.X/ofs-f2000x-pl/../opae-sdk LD_LIBRARY_PATH is set to /home/user_area/ofs-X.X.X/ofs-f2000x-pl/../opae-sdk/lib64: 13 - Build FIM for f2000x Hardware This option builds the FIM based on the setting for the $OFS_PLATFORM, $FIM_SHELL environment variable. Check this variable in the following file ofs_f2000x_eval.sh 14 - Check FIM Identification of FIM for f2000x Hardware The FIM is identified by the following file fme-ifc-id.txt located at $OFS_ROOTDIR/$FIM_WORKDIR/syn/syn_top/ 15 - Build Partial Reconfiguration Tree for f2000x Hardware This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development 16 - Build Base FIM Identification(ID) into PR Build Tree template This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU workloads 17 - Build Partial Reconfiguration Tree for f2000x Hardware with Remote Signal Tap This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development for the Remote Signal Tap flow 18 - Build Base FIM Identification(ID) into PR Build Tree template with Remote Signal Tap This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree for Remote Signal Tap to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU workloads"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#315-ofs-hardware-programmingdiagnostic-menu","title":"3.1.5 OFS HARDWARE PROGRAMMING/DIAGNOSTIC MENU","text":"The following submenu allows you to: * Program and check flash * Perform a remote system update (RSU) of the FPGA image into the FPGA * Bind virtual functions to VFIO PCIe driver * Run host exerciser (HE) commands such as loopback to test interfaces VFIO PCI driver binding * Read the control and status registers (CSRs) for bound modules that are part of the OFS reference design.
Menu Option Description 19 - Program BMC Image into f2000x Hardware The user must place a new BMC flash file in the following directory $OFS_ROOTDIR/bmc_flash_files. Once the user executes this option a new BMC image will be programmed. A remote system upgrade command is initiated to store the new BMC image 20 - Check Boot Area Flash Image from f2000x Hardware This option checks which location area in FLASH the image will boot from, the default is user1 Boot Page : user1 21 - Program FIM Image into user1 area for f2000x Hardware This option programs the FIM image \"ofs_top_page1_unsigned_user1.bin\" into user1 area in flash 22 - Initiate Remote System Upgrade (RSU) from user1 Flash Image into f2000x Hardware This option initiates a Remote System Upgrade and soft reboots the server and re-scans the PCIe bus for the new image to be loaded 2022-11-10 11:26:24,307 - [[pci_address(0000:b1:00.0), pci_id(0x8086, 0xbcce)]] performing RSU operation 2022-11-10 11:26:24,310 - [[pci_address(0000:b0:02.0), pci_id(0x8086, 0x347a)]] removing device from PCIe bus 2022-11-10 11:26:24,357 - waiting 10 seconds for boot 2022-11-10 11:26:34,368 - rescanning PCIe bus: /sys/devices/pci0000:b0/pci_bus/0000:b0 2022-11-10 11:26:35,965 - RSU operation complete 23 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 24 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 25 - Create Virtual Functions (VF) and bind driver to vfio-pci f2000x Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 23 to check that the new drivers are bound 26 - Verify FME Interrupts with hello_events The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors 27 - Run HE-LB Test This option runs 5 tests 1) checks and generates traffic with the intention of exercising the path from the AFU to the Host at full bandwidth 2) run a loopback throughput test using one cacheline per request 3) run a loopback read test using four cachelines per request 4) run a loopback write test using four cachelines per request 5) run a loopback throughput test using four cachelines per request 28 - Run HE-MEM Test This option runs 2 tests 1) Checking and generating traffic with the intention of exercising the path from FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host 2) run a loopback throughput test using one cacheline per request 29 - Run HE-HSSI Test This option runs 1 test HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic 1) Send traffic through the 10G AFU 30 - Run Traffic Generator AFU Test This option runs 3 tests TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank 1) Run the preconfigured write/read traffic test on channel 0 2) Target channel 1 with a 1MB single-word write only test for 1000 iterations 3) Target channel 2 with 4MB write/read test of max burst length for 10 iterations 31 - Read from CSR (Command and Status Registers) for f2000x Hardware This option reads from the following CSR's HE-LB Command and Status Register Default Definitions HE-MEM Command and Status Register Default Definitions HE-HSSI Command and Status Register Default Definitions"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#316-ofs-hardware-afu-testing-menu","title":"3.1.6 OFS HARDWARE AFU TESTING MENU","text":"This submenu tests partial reconfiguration by building and loading an memory-mapped I/O example AFU/workload, executes software from host, and tests remote signal tap.
Menu Option Description 32 - Build and Compile host_chan_mmio example This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) ready for hardware programming 33 - Execute host_chan_mmio example This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test 34 - Modify host_chan_mmio example to insert Remote Signal Tap This option inserts a pre-defined host_chan_mmio.stp Signal Tap file into the OFS code to allow a user to debug the host_chan_mmio AFU example 35 - Build and Compile host_chan_mmio example with Remote Signal Tap This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS(Green Bit Stream) ready for hardware programming with Remote Signal tap enabled 36 - Execute host_chan_mmio example with Remote Signal Tap This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test. The user must open the Signal Tap window when running the host code to see the transactions in the Signal tap window"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#317-ofs-hardware-afu-bbb-testing-menu","title":"3.1.7 OFS HARDWARE AFU BBB TESTING MENU","text":"This submenu tests partial reconfiguration using a hello_world example AFU/workload, executes sw from host
Menu Option Description 37 - Build and Compile hello_world example This option builds the hello_ world example from the following repo $FPGA_BBB_CCI_SRC/samples/tutorial/afu_types/01_pim_ifc/$AFU_BBB_TEST_NAME, where AFU_BBB_NAME=hello_world. This produces a GBS(Green Bit Stream) ready for hardware programming 38 - Execute hello_world example This option builds the host code for hello_world example and programs the GBS file and then executes the test"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#319-ofs-unit-test-project-menu","title":"3.1.9 OFS UNIT TEST PROJECT MENU","text":"Builds, compiles and runs standalone simulation block tests. More unit test examples are found at the following location ofs-f2000x-pl/sim/unit_test
Menu Option Result 39 - Generate Simulation files for Unit Test This option builds the simulation file set for running a unit test simulation 40 - Simulate Unit Test dfh_walker and log waveform This option runs the dfh_walker based on the environment variable \"UNIT_TEST_NAME=dfh_walker\" in the evaluation script. A user can change the test being run by modifying this variable"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#3110-ofs-uvm-project-menu","title":"3.1.10 OFS UVM PROJECT MENU","text":"Builds, compiles and runs full chip simulation tests. The user should execute the options sequentially ie 41,42, 43 and 44
Menu Option Description 41 - Check UVM software versions for f2000x Project DESIGNWARE_HOME is set to /home/synopsys/vip_common/vip_Q-2020.03A UVM_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm VCS_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel VERDIR is set to /home/user_area/ofs-X.X.X/fim-f2000x-pl/verification VIPDIR is set to /home/user_area/ofs-X.X.X/fim-f2000x-pl/verification 42 - Compile UVM IP This option compiles the UVM IP 43 - Compile UVM RTL and Testbench This option compiles the UVM RTL and Testbench 44 - Simulate UVM dfh_walking_test and log waveform This option runs the dfh_walking test based on the environment variable \"UVM_TEST_NAME=dfh_walking_test\" in the evaluation script. A user can change the test being run by modifying this variable 45 - Simulate all UVM test cases (Regression Mode) This option runs the f2000x regression mode, cycling through all UVM tests defined in verification/tests/test_pkg.svh file"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#3111-ofs-build-all-project-menu","title":"3.1.11 OFS BUILD ALL PROJECT MENU","text":"Builds the complete OFS flow, good for regression testing and overnight builds
For this menu a user can run a sequence of tests (compilation, build and simulation) and executes them sequentially. After the script is successfully executed, a set of binary files is produced which a you can use to evaluate your hardware. Log files are also produced which checks whether the tests passed.
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 46 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 41, 42, 43 and 44. These 19 menu options are chosen to build the complete OFS flow covering build, compile and simulation.
Menu Option Result 46 - Build and Simulate Complete f2000x Project Generating Log File with date and timestamp Log file written to /home/guest/ofs-2.3.1/log_files/f2000x_log_2022_11_10-093649/ofs_f2000x_eval.log"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#definition-of-multi-test-set-up","title":"Definition of Multi-Test Set-up","text":"Menu Option 46 above in the evaluation script can be refined to tailor it to the users need and is principally defined by the variable below
MULTI_TEST[A,B]=C
where
A= Total Number of menu options in script B= Can be changed to a number to select the test order C= Menu Option in Script
Example 1 MULTI_TEST[46,0]=2
A= 46 is the total number of options in the script B= 0 indicates that this is the first test to be run in the script C= Menu option in Script ie 2- List of Documentation for OFS f2000x Project
Example 2 MULTI_TEST[46,0]=2 MULTI_TEST[46,1]=9
In the example above two tests are run in order ie 0, and 1 and the following menu options are executed ie 2- List of Documentation for OFS f2000x Project and 9 - Check OFS software versions for OFS f2000x Project
The user can also modify the build time by de-selecting options they do not wish to use, see below for a couple of use-case scenarios.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#default-user-case","title":"Default User Case","text":"A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 46 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 41, 42, 43 and 44. All other tests with an \"X\" indicates do not run that test
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#user-case-for-ofs-fimpr-build-menu","title":"User Case for OFS FIM/PR BUILD MENU","text":"In the example below when the user selects option 46 from the main menu the script will only run options from the OFS FIM/PR BUILD MENU (7 options, main menu options 12, 13, 14, 15, 16, 17 and 18). All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#4-f2000x-common-test-scenarios","title":"4 f2000x Common Test Scenarios","text":"This section will describe the most common compile build scenarios if a user wanted to evaluate an acceleration card on their server. The Pre-requisite column indcates the menu comamnds that must be run befere executing the test eg To run Test 5 then a user needs to have run option 13, 15 and 16 before running options 23, 24, 25, 32 and 33.
Test Test Scenario Pre-Requisite Menu Option Menu Option Test 1 FIM Build - 13 Test 2 Partial Reconfiguration Build 13 15, 16 Test 3 Program FIM and perform Remote System Upgrade 13 21, 22 Test 4 Bind PF and VF to vfio-pci drivers - 23, 24, 25 Test 5 Build, compile and test AFU on hardware 13, 15, 16 23, 24, 25, 32, 33 Test 6 Build, compile and test AFU Basic Building Blocks on hardware 13, 15, 16 23, 24, 25, 37, 38 Test 7 Build and Simulate Unit Tests - 39, 40 Test 8 Build and Simulate UVM Tests - 42, 43, 44, 45"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/","title":"OFS Getting Started Guide for Intel Agilex 7 SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users get started in evaluating the 2024.1-1 version of the SoC Attach release targeting the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL. After reviewing this document, a user shall be able to:
- Set up their server environment according to the Best Known Configuration (BKC)
- Build a Yocto image with the OPAE SDK and Linux DFL Drivers included
- Load and verify firmware targeting the SR and PR regions of the board, the BMC, and the ICX-D SoC NVMe and BIOS
- Verify full stack functionality offered by the SoC Attach OFS solution
- Know where to find additional information on other SoC Attach ingredients
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the IPU Platform F2000X-PL. The card is an acceleration development platform (ADP) intended to be used as a starting point for evaluation and development. This document will cover key topics related to initial bring up and development of the IPU Platform F2000X-PL, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#13-references-and-versions","title":"1.3 References and Versions","text":""},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-2-software-and-firmware-component-version-summary-for-soc-attach","title":"Table 2: Software and Firmware Component Version Summary for SoC Attach","text":"Component Version Download link (where applicable) Available FIM Version(s) PR Interface ID: c2dac77b-757c-5e27-a566-aad3ffba2f4e ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell Host Operating System Ubuntu 22.04 LTS Official Release Page Host OPAE SDK 2.12.0-5 https://github.com/OFS/opae-sdk/releases/tag/2.12.0-5 SoC OS meta-intel-ese Reference Distro 1.0-ESE (kirkstone) ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell SoC Kernel Version 6.1.78-dfl ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell SoC OPAE SDK 2.12.0-5 https://github.com/OFS/opae-sdk/releases/tag/2.12.0-5 SoC Linux DFL ofs-2024.1-6.1-2 https://github.com/OFS/linux-dfl/releases/tag/ofs-2024.1-6.1-2 SoC BMC and RTL 1.2.4 ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell SoC BIOS 0ACRH608_REL ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell Not all components shown in Table 2 will have an update available upon release. The OPAE SDK and Linux DFL software stacks are incorporated into a Yocto image and do not need to be downloaded separately. Updates required for the ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell are shown under Table 9 in section 2.0 Updating the IPU Platform F2000X-PL.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-3-related-repositories","title":"Table 3: Related Repositories","text":"Name Location Intel-FPGA-BBB https://github.com/OPAE/intel-fpga-bbb.git OFS-PLATFORM-AFU-BBB https://github.com/OFS/ofs-platform-afu-bbb.git, tag: ofs-2024.1-1 AFU-EXAMPLES https://github.com/OFS/examples-afu.git, tag: ofs-2024.1-1 OPAE-SDK https://github.com/OFS/opae-sdk, tag: 2.12.0-5 LINUX-DFL https://github.com/OFS/linux-dfl, tag: ofs-2024.1-6.1-2 META-OFS https://github.com/OFS/meta-ofs, tag: ofs-2024.1-2"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#14-board-installation-and-server-requirements","title":"1.4 Board Installation and Server Requirements","text":"The F2000X-PL device physical setup procedure and required server settings are detailed in this device's Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL. Please review this document and follow proper procedure when installing your device. The rest of this document will assume you have completed basic platform bring-up.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#15-reference-documents","title":"1.5 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.1-1/.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#20-updating-the-ipu-platform-f2000x-pl","title":"2.0 Updating the IPU Platform F2000X-PL","text":"Every IPU Platform F2000X-PL ships with pre-programmed firmware for the FPGA user1, user2, and factory images, the Cyclone 10 BMC RTL and FW, the SoC NVMe, and the SoC BIOS. The combination of FW images in Table 4 compose the official ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell. Upon initial receipt of the board from manufacturing you will need to update two of these regions of flash to conform with the best known configuration for SoC Attach. As shown in Table 9, not all devices require firmware updates. To instruct users in the process of updating on-board IPU Platform F2000X-PL firmware, examples are provided in this guide illustrating the firmware update process for all devices.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-4-ipu-platform-f2000x-pl-fw-components","title":"Table 4: IPU Platform F2000X-PL FW Components","text":"HW Component File Name Version Update Required (Yes/No) Download Location FPGA SR Image1 user1: ofs_top_page1_unsigned_user1.binuser2: ofs_top_page2_unsigned_user2.binfactory: ofs_top_page0_unsigned_factory.bin PR Interface ID: c2dac77b-757c-5e27-a566-aad3ffba2f4e Yes ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell FPGA PR IMAGE2 ofs_pr_afu.green_region_unsigned.gbs N/A Yes Compiled with FIM ICX-D NVMe core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic N/A Yes ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell BMC RTL and FW AC_BMC_RSU_user_retail_1.2.4_unsigned.rsu 1.2.4 No ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell BIOS Version 0ACRH608_REL.BIN 0ACRH608_REL Yes ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell If a component does not have a required update, it will not have an entry on ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell.
1To check the PR Interface ID of the currently programmed FIM and the BMC RTL and FW version, use fpgainfo fme from the SoC. 2Must be programmed if using AFU-enabled exercisers, not required otherwise.
$ fpgainfo fme\nIntel IPU Platform F2000X-PL\n**Board Management Controller NIOS FW version: 1.2.4**\n**Board Management Controller Build version: 1.2.4**\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n**Bitstream Id : 360572752842090923**\nBitstream Version : 5.0.1\n**Pr Interface Id : c2dac77b-757c-5e27-a566-aad3ffba2f4e**\nBoot Page : user1\nUser1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nUser2 Image Info : None\nFactory Image Info : None\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#21-updating-the-f2000x-pl-icx-d-soc-nvme","title":"2.1 Updating the F2000X-PL ICX-D SoC NVMe","text":"The IPU Platform F2000X-PL ships with a Yocto image pre-programmed in NVMe, which is not the same as the SoC Attach OFS image that we will be using. The latter provides only the OPAE SDK and Linux DFL drivers and is fully open source. This section will show how you can use an attached USB drive to load a new image into flash. You will use a serial terminal to install the new image - Minicom and PuTTy terminal emulators have both been tested. minicom is used for demonstration purposes as the serial terminal to access the ICX-D SoC UART connection in this section. Information on compiling your own Yocto image for use with the IPU Platform F2000X-PL is discussed in section 4.0 Compiling a Custom Yocto SoC Image.
Note: Username and password for the default SoC NVMe boot image are \"root\" and \"root@123\".
-
First, make sure to complete the steps in section 1.5.5 Creating a Bootable USB Flash Drive for the SoC, and attach the USB drive either directly into the rear of the IPU Platform F2000X-PL, or into a USB Hub that itself is connected to the board.
-
Ensure your Minicom terminal settings match those shown below. You must direct Minicom to the USB device created in /dev associated with your RS-232 to USB Adapter cable. This cable must be attached to a server that is separate from the one housing your IPU Platform F2000X-PL. Check the server logs in dmesg to figure out which device is associated with your cable: [ 7.637291] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0. In this example the special character file /dev/ttyUSB0 is associated with our cable, and can be connected to using the following command: sudo minicom --color=on -D /dev/ttyUSB0.
-
Change the SoC boot order to boot from USB first. Reboot the server. From your serial Minicom terminal, watch your screen and press 'ESC' key to go into BIOS setup mode. Once BIOS setup comes up as shown below, click the right arrow key six times to move from 'Main' set up menu to 'Boot' setup:
Main setup menu:
Your order of boot devices may differ. You need to move the USB flash up to Boot Option #1 by first using the down arrow key to highlight the USB device then use '+' key to move the USB device to #1 as shown below:
Press 'F4' to save and exit.
-
You will boot into Yocto automatically. Log in with username root and an empty password using Minicom. Take note of the IP address of the board, you can use this to log in without needing the serial cable.
Verify that you have booted from the USB and not the on-board NVMe lsblk -no pkname $(findmnt -n / | awk '{ print $2 }'). You should see a device matching /dev/sd*, and not nvme*n*p*. If you see nvme*n*p*, then review the previous steps.
Record the IP address of the SoC at this time ip -4 addr. This will be used to log in remotely using SSH.
-
Check that 4 partitions created in 1.5.5 Creating a Bootable USB Flash Drive for the SoC are visible to the SoC in /dev/sd*:
$ lsblk -l\nroot@intel-corei7-64:/mnt# lsblk -l\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nsda 8:0 1 117.2G 0 disk\nsda1 8:1 1 38.5M 0 part /boot\nsda2 8:2 1 20G 0 part /\nsda3 8:3 1 44M 0 part [SWAP]\nsda4 8:4 1 97.1G 0 part /mnt\nnvme0n1 259:0 0 59.6G 0 disk\nnvme0n1p1 259:1 0 300M 0 part\nnvme0n1p2 259:2 0 22.1G 0 part\nnvme0n1p3 259:3 0 44M 0 part\n
Mount partition 4, and cd into it.
$ mount /dev/sda4 /mnt`\n$ cd /mnt\n
-
Install the Yocto release image in the SoC NVMe.
$ dd if=core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic of=/dev/nvme0n1 bs=1M status=progress conv=sync\n$ sync\n$ sgdisk -e /dev/nvme0n1\n
The transfer from USB to NVMe will take several minutes.
-
Reboot the SoC and update the SoC BIOS to boot from NVMe. Follow steps 2 and 3 from this section again, and this time move the NVME back to the front of the boot order. The NVMe is named UEFI OS (PCIe SSD) by the BIOS. Press F4 to save and exit.
You can use wget to retrieve a new version of the Yocto release image from meta-ofs once the SoC's network connection is up. Use wget to copy the image to the SoC over the network under /mnt. You may need to delete previous Yocto images to save on space: $ wget --no-check-certificate --user <Git username> --ask-password https://github.com/OFS/meta-ofs/releases/download/ofs-2024.1-2/core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.gz. Uncompress the newly retrieved file: gzip -d core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.gz. This may take several minutes.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#211-setting-the-time","title":"2.1.1 Setting the Time","text":" -
Use Linux command to set system time using format: date --set=\"STRING\".
date -s \"26 APRIL 2023 18:00:00\"\n
-
Set HWCLOCK to current system time:
hwclock --systohc\n
Verify time is set properly
date\n...\nhwclock --show\n...\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#30-updating-the-ipu-platform-f2000x-pl","title":"3.0 Updating the IPU Platform F2000X-PL","text":""},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#31-preparing-the-ipu-platform-f2000x-pl-soc-for-updates","title":"3.1 Preparing the IPU Platform F2000X-PL SoC for Updates","text":"Updating the IPU Platform F2000X-PL firmware often requires reboots of the SoC or reconfiguration of the FPGA region. If there are processes connected from the host to the SoC that do not expect the downtime to occur, or if the host is not tolerant to a surprise PCie link down, the following instructions can be used to properly orchestrate updates with the host when reboots occur.
Note: Intel IPU Platform F2000X-PL FPGA and BMC updates are initiated by commands issued on the IPU SoC. Issue the following commands from the host to remove any processes that would be impacted by this update. The instructions on properly removing the IPU from PCIe bus require the OPAE SDK to be installed on the host. Refer to section 6.0 Setting up the Host for this process.
-
From a host terminal shell, find PCIe Bus/Device/Function (BDF) address of your Intel IPU Platform F2000X-PL. Run the command lspci | grep bcce to print all boards with a DID that matches bcce.
$ lspci | grep bcce\n31:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n31:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\n# In this example, 31:00.0 is the proper PCIe BDF of our device\n
-
Shut down all VMs and software applications attached to any PFs/VFs on the host
- Issue the command
sudo pci_device <<PCIe BDF>> unplug on the host to remove the PCIe device from the PCIe bus - Shut down all software applications on the SoC accessing non-management PFs/VFs
- Issue your update command on the SoC, which will cause an SoC reboot and surprise PCIe link down on the host (ex.
reboot, rsu bmc/bmcimg/fpga) - Once you have completed all firmware updates, you may restart application software on the SoC
- Issue command
sudo pci_device <<PCIe BDF>> plug on the host to rescan the PCIe bus and rebind the device to its native driver - Restart software applications on the host
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#32-updating-fim-bmc-and-afu-with-fpgasupdate","title":"3.2 Updating FIM, BMC and AFU with fpgasupdate","text":"The fpgasupdate tool updates the Intel\u00ae C10 10 BMC image and firmware, root entry hash, FPGA Static Region (SR) and user image (PR). The fpgasupdate will only accept images that have been formatted using PACSign. If a root entry hash has been programmed onto the board, then the image will also need to be signed using the correct keys. Please refer to the Security User Guide: Intel Open FPGA Stack for information on creating signed images and on programming and managing the root entry hash. This repository requires special permissions to access - please reach out to your Intel representative to request. The fpgasupdate tool is used to program images into flash memory. The rsu tool is used to configure the FPGA/BMC with an image that is already stored in flash memory, or to switch between user1 and user2 images. All images received from an official Intel release will be \"unsigned\", as described in the Security User Guide: Intel Open FPGA Stack.
Note: 'Unsigned' in this context means the image has passed through PACsign and has had the proper security blocks prepended using a set of 'dummy' keys. FIMs with image signing enabled will require all programmable images to pass through PACsign even if the currently programmed FIM/BMC do not require specific keys to authenticate.
There are two regions of flash you may store FIM images for general usage, and one backup region. These locations are referred to as user1, user2, and factory. The factory region is not programmed by default and can only be updated once keys have been provisioned. The BMC FW and RTL will come pre-programmed with version 1.1.9.
Updating the FIM from the SoC requires the SoC be running a Yocto image that includes the OPAE SDK and Linux DFL drivers. Updating the FIM using fpgasupdate also requires an OFS enabled FIM to be configured on the F2000X-PL, which it will ship with from manufacturing. You need to transfer any update files to the SoC over SSH. The OPAE SDK utility fpgasupdate will be used to update all ofthe board's programmable firmware . This utility will accept files of the form *.rsu, *.bin, and *.gbs, provided the proper security data blocks have been prepended by PACSign. The default configuration the IPU platform ships with will match the below:
$ fpgainfo fme\nIntel IPU Platform F2000X-PL\n**Board Management Controller NIOS FW version: 1.2.4**\n**Board Management Controller Build version: 1.2.4**\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n**Bitstream Id : 360572752842090923**\nBitstream Version : 5.0.1\n**Pr Interface Id : c2dac77b-757c-5e27-a566-aad3ffba2f4e**\nBoot Page : user1\nUser1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nUser2 Image Info : None\nFactory Image Info : None\n
To load a new update image, you need to pass the IPU's PCIe BDF and the file name to fpgasupdate on the SoC. The below example will update the user1 image in flash:
$ fpgasupdate ofs_top_page1_unsigned_user1.bin 15:00.0\n
After loading an update image, rsu fpga/bmc/bmcimg can be used to reload the firmware and apply the change, as discussed below. An RSU of the BMC will always cause a reload of both the BMC and FPGA images.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#33-loading-images-with-rsu","title":"3.3 Loading images with rsu","text":"RSU performs a Remote System Update operation on an IPU Platform F2000X-PL, given its PCIe address. An rsu operation sends an instruction to the device to trigger a power cycle of the card only if run with bmcimg. rsu will force reconfiguration from flash for either the BMC or FPGA. PCIe Advanced Error Reporting (AER) is temporarily disabled for the card when RSU is in progress
The IPU Platform F2000X-PL contains two regions of flash you may store FIM images. These locations are referred to as user1 and user2. After an image has been programmed to either of these regions in flash using fpgasupdate, you may perform an rsu to reconfigure the Agilex 7 FPGA with the new image stored in flash. This operation will indicate to the BMC which region to configure the FPGA device from after power-on.
If the factory image has been updated, Intel strongly recommends you immediately RSU to the factory image to ensure the image is functional.
You can determine which region of flash was used to configure their FPGA device using the command fpgainfo fme and looking at the row labelled Boot Page.
When loading a new FPGA SR image, use the command rsu fpga. When loading a new BMC image, use the command rsu bmc. When using the RSU command, you may select which image will be configured to the selected device. For example, when performing an RSU for the Intel Agilex 7 FPGA, you may select to configure the user1, user2, or factory image. When performing an RSU for the C10 BMC, you may select to configure the user or factory image. You may also use RSU to reconfigure the SDM on devices that support it. The RSU command sends an instruction to the BMC to reconfigure the selected device from an image stored in flash.
$ rsu fpga --page=user1 15:00.0\n
Useage:
rsu bmc --page=(user|factory) [PCIE_ADDR]\nrsu fpga --page=(user1|user2|factory) [PCIE_ADDR]\nrsu sdm [PCIE_ADDR]\n
You can use RSU to change which page in memory the FPGA will boot from by default.
Synopsis:
rsu fpgadefault --page=(user1|user2|factory) --fallback=<csv> 15:00.0\n
Use to set the default FPGA boot sequence. The --page option determines the primary FPGA boot image. The --fallback option allows a comma-separated list of values to specify fallback images.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#34-updating-the-icx-d-soc-bios","title":"3.4 Updating the ICX-D SoC BIOS","text":"The ICX-D SoC NVMe comes pre-programmed with BIOS v7 (0ACRH007). This version will need to be replaced with 0ACRH608_REL. BIOS update files come in the form 0ACRH\\<\\<version>>.bin, and can be downloaded on ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell. This update process is in-band, and requires you to download and install a BIOS UEFI utility from AMI called \"APTIO V AMI Firmware Update Utility\", available here. This package will install a utility in the UEFI shell called AfuEfix64.efi which will be used to overwrite the ICX-D BIOS.
-
Check your BIOS Version. Reboot the SoC and wait for the BIOS version to be shown. In this example, the BIOS will need to be updated.
-
Download both the ICX-D SoC Update image and APTIO V AMI Firmware Update Utility. Unzip the BIOS update image and locate your BIOS update binary. Unzip Aptio_V_AMI_Firmware_Update_Utility.zip, and then navigate to Aptio_V_AMI_Firmware_Update_Utility\\afu\\afuefi\\64 and unzip AfuEfi64.zip. The file we need from this package is called AfuEfix64.efi.
-
Copy both files over to the SoC into /boot/EFI using the SoC's IP.
$ scp 0ACRH608_REL.BIN root@XX.XX.XX.XX:/boot/EFI\n$ scp AfuEfix64.efi root@XX.XX.XX.XX:/boot/EFI\n
-
Reboot the SoC from a TTY Serial terminal. Watch your screen and press 'ESC' key to go into BIOS setup mode. Select 'Built-in EFI Shell'.
-
At EFI prompt enter the following:
Shell> FS0:\nFS0:> cd EFI\nFS0:\\EFI\\> AfuEfix64.efi 0ACRH608_REL.BIN /p /n /b\n
Press 'E'. When the update has completed type 'reset -w' to reboot.
-
Watch your screen and press 'ESC' key to go into BIOS setup mode. Verify BIOS version matches expectation.
-
Click the right arrow key six times to move from 'Main' set up menu to 'Boot' setup. Select NVMe as the primary boot source.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#40-compiling-a-custom-yocto-soc-image","title":"4.0 Compiling a Custom Yocto SoC Image","text":"Custom Yocto image compilation steps are shown in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#50-verifying-the-icx-d-soc-opae-sdk","title":"5.0 Verifying the ICX-D SoC OPAE SDK","text":"The reference SoC Attach FIM and unaltered FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Full supported functionality of the HEMs is documented in the host_exerciser opae.io GitHub page. SoC Attach supports HEMs run both with and without an AFU image programmed into the board's one supported PR region. This image is available on the offial SoC Attach GitHub Page, and is programmed using fpgasupdate as shown in section 3.2. A few select examples run from the SoC and their expected results will be shown below.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#51-checking-telemetry-with-fpgainfo","title":"5.1 Checking Telemetry with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files.
The command argument is one of the following: errors, power, temp, port, fme, bmc, phy, mac, and security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
An example output for fpgainfo fme is shown below. Your IDs may not match what is shown here:
$ fpgainfo fme\nIntel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.1.9\nBoard Management Controller Build version: 1.1.9\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50103023DFBFC8E\nBitstream Version : 5.0.1\nPr Interface Id : bf74e494-ad12-5509-98ab-4105d27979f3\nBoot Page : user1\nUser1 Image Info : 98ab4105d27979f3bf74e494ad125509\nUser2 Image Info : None\nFactory Image Info : None\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#52-host-exercisers","title":"5.2 Host Exercisers","text":"Of these five tests listed below, the first three do not require an AFU be loaded into the board's PR region. They exercise data paths that pass exclusively through the FIM. The latter three tests exercise data through the AFU data path, and require SoC Attach release AFU Image to be configured using fpgasupdate.
- Run HE-MEM with 2 cachelines per request in
mem mode, exercising the FPGA's connection to DDR. No AFU required.Note: If you see the error message Allocate SRC Buffer, Test mem(1): FAIL, then you may need to manually allocate 2MiB Hugepages using the following: echo 20 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF0 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.1 <username>:<username>\n# Check for HE-MEM Accelerator GUID 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Run desired HE-MEM test(s)\n$ host_exerciser --cls cl_1 --mode lpbk mem\nstarting test run, count of 1\nAPI version: 2\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 9948\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.096 GB/s\n Test mem(1): PASS\n
- Generate traffic with HE-HSSI. No AFU required.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF2 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.2 <username>:<username>\n# Check for HE-HSSI Accelerator GUID 823c334c-98bf-11ea-bb37-0242ac130002\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Show number of configured ports\n$ fpgainfo phy | grep Port\n# Generate traffic for specific port number\n$ hssi --pci-address <PCIe Bus>:00.2 hssi_10g --num-packets 100 --port <port number>\n10G loopback test\nTx/Rx port: 1\nTx port: 1\nRx port: 1\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n...\n
This command will generate a log file in the directory is was run from. Check TxPackets and RxPackets for any loss. Two more supported HSSI commands are hssistats, which provides MAC statistics, and hssimac, which provides maximum TX and RX frame size.
- Test memory traffic generation using MEM-TG. This will exercise and test available memory channels with a configurable memory pattern. Does not require an AFU image.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF2 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.3 <username>:<username>\n# Check for MEM-TG Accelerator GUID 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Example MEM-TG Test\n$ mem_tg --loops 500 -w 1000 -r 0 -b 0x1 --stride 0x1 -m 0 tg_test\n
- HE-LPBK is designed to demo how AFUs move data between host memory and the FPGA. Will check latency, MMIO latency, MMIO bandwidth, and PCIe bandwidth. LPBK workload requires the SoC Attach AFU be loaded into the board's PR slot.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF1 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.1 <username>:<username>\n# Check for LPBK GUID 56e203e9-864f-49a7-b94b-12284c31e02b\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Example loopback test\n$ host_exerciser --mode lpbk lpbk\n
- Exercise He-HSSI subsystem from the AFU. This test will generate and receieve packets from any of the 8 available ports. This HSSI workload requires the SoC Attach AFU be loaded into the board's PR slot.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF6 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.2 <username>:<username>\n# Check for HSSI GUID 823c334c-98bf-11ea-bb37-0242ac130002\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Geneate traffic for specific port\n$ hssi --pci-address <bus_num>:00.6 hssi_10g --num-packets 100 --port <port_num>\n
This command will generate a log file in the directory is was run from. Check TxPackets and RxPackets for any loss. Two more supported HSSI commands are hssistats, which provides MAC statistics, and hssimac, which provides maximum TX and RX frame size.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#53-additional-opae-sdk-utilities","title":"5.3 Additional OPAE SDK Utilities","text":"This section will discuss OPAE SDK utilities that were not covered by previous sections. These commands are all available on the ICX-D SoC Yocto image by default. A full description and syntax breakdown for each command is located on the official OPAE SDK github.io repo.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-5-opae-sdk-utilities","title":"Table 5: OPAE SDK Utilities","text":"Utility Description fpgasupdate 3.2 Updating FIM, BMC and AFU with fpgasupdate rsu Section 3.3 Loading images with rsu host_exerciser Section 5.2 Host Exercisers hssi Section 5.2 Host Exercisers hssistats Section 5.2 Host Exercisers hssimac Section 5.2 Host Exercisers mem_tg Section 5.2 Host Exercisers usrclk userclk tool is used to set high and low clock frequency to an AFU. mmlink Remote signaltap is software tool used for debug RTL (AFU), effectively a signal trace capability that Quartus places into a green bitstream. Remote Signal Tap provides access the RST part of the Port MMIO space, and then runs the remote protocol on top. opaevfio The opaevfio command enables the binding/unbinding of a PCIe device to/from the vfio-pci device driver. See https://kernel.org/doc/Documentation/vfio.txt for a description of vfio-pci. opae.io An interactive Python environment packaged on top of libopaevfio.so, which provides user space access to PCIe devices via the vfio-pci driver. The main feature of opae.io is its built-in Python command interpreter, along with some Python bindings that provide a means to access Configuration and Status Registers (CSRs) that reside on the PCIe device. bitstreaminfo Prints bitstream metadata. fpgaconf Lower level programming utility that is called automatically by fpgasupdate. fpgaconf writes accelerator configuration bitstreams (also referred to as \"green bitstreams\") to an FPGA device recognized by OPAE. In the process, it also checks the green bitstream file for compatibility with the targeted FPGA and its current infrastructure bitstream (the \"blue bistream\"). fpgad Periodically monitors/reports the error status reflected in the device driver's error status sysfs files. Establishes the channel by which events are communicated to the OPAE application. Programs a NULL bitstream in response to AP6 event. fpgad is required to be running before API calls fpgaRegisterEvent and fpgaUnregisterEvent will succeed."},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#60-setting-up-the-host","title":"6.0 Setting up the Host","text":"The steps to set up the host and build and install the OPAE SDK are shown in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#61-verifying-the-soc-attach-solution-on-the-host","title":"6.1 Verifying the SoC Attach Solution on the Host","text":"The SoC Attach workload supports testing MMIO HW and Latency and PCIe BW and latency out of box. Execution of the host_exerciser binary on the host requires the OPAE SDK to be installed as shown in section 6.1 Installing the OPAE SDK On the Host. You will also need to have a proper SoC Attach FIM configured on your board as shown in section 3.2 Updating FIM, BMC and AFU with fpgasupdate.
-
Initialize PF attached to HSSI LPBK GUID with vfio-pci driver.
$ sudo opae.io init -d 0000:b1:00.0 <username>:<username>\n$ sudo opae.io init -d 0000:b1:00.1 <username>:<username>\n
-
Run host_exerciser loopback tests (only lpbk is supported). There are more methods of operation than are shown below - read the HE help message for more information.
# Example lpbk tests.\n$ sudo host_exerciser lpbk\n$ sudo host_exerciser --mode lpbk lpbk\n$ sudo host_exerciser --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --cls cl_4 lpbk\n# Number of cachelines per request 4.\n$ sudo host_exerciser --mode lpbk --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode lpbk --cls cl_4 lpbk\n# vNumber of cachelines per request 4.\n$ sudo host_exerciser --mode read --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode read --cls cl_4 lpbk\n# Number of cachelines per request 4.\n$ sudo host_exerciser --mode write --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode write --cls cl_4 lpbk\n# Number of cachelines per request 4.\n$ sudo host_exerciser --mode trput --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode trput --cls cl_4 lpbk\n# Enable interleave requests in throughput mode\n$ sudo host_exerciser --mode trput --interleave 2 lpbk\n$ sudo host_exerciser --perf true --mode trput --interleave 2 lpbk\n#with delay option.\n$ sudo host_exerciser --mode read --delay true lpbk\n$ sudo host_exerciser --mode write --delay true lpbk\n# Test all modes of operation\n$ host_exerciser --testall=true lpbk\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#62-fpga-device-access-permissions","title":"6.2 FPGA Device Access Permissions","text":"Access to FPGA accelerators and devices is controlled using file access permissions on the Intel\u00ae FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
sudo chmod a+rw /dev/dfl-port.0\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#63-memlock-limit","title":"6.3 Memlock limit","text":"Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/","title":"UVM Simulation User Guide: Open FPGA Stack for Intel Agilex 7 SoC Attach FPGAs","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#1-overview","title":"1 Overview","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the UVM simulation tool using OFS. After reviewing the document, you will be able to:
- Set-up the UVM verification tool suite
- Run pre-existing UVM unit tests and also create new UVM tests for your design
Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#2-introduction-to-uvm","title":"2 Introduction to UVM","text":"The Functional Verification Environment for OFS is UVM (Universal Verification Methodology) compliant and provides configurable setup for verifying various FIM features in simulation.
The purpose of this document is to demonstrate a full chip level and unit level UVM based verification environment for the current base shell FIM architecture as well as providing examples to extend the setup for different OFS variants by reusing the existing architecture and infrastructure for UVM based verification.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#3-universal-testbench-architecture","title":"3 Universal Testbench Architecture","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#31-overview","title":"3.1 Overview","text":"The main idea behind UVM is to develop modular, reusable, and a scalable testbench structure by providing an API framework that can be deployed across multiple projects.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#32-core-verification-concepts","title":"3.2 Core Verification Concepts","text":"The following is the list of verification components that will be used to design a UVM testbench architecture:
\u2022 Sequencer \u2022 Driver \u2022 Monitor \u2022 Scoreboard \u2022 Virtual Sequencer \u2022 Interface \u2022 Verification Environment \u2022 TOP Testbench
Figure 1 provides the general UVM Testbench and the verification components involved in the top-level architecture.
Figure 1 Typical UVM Testbench
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4-ofs-testbench-architecture","title":"4 OFS Testbench Architecture","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#41-overview","title":"4.1 Overview","text":"OFS (Open FPGA Stack) provides a UVM (Universal Verification Methodology) environment for the FIM with a modular, reusable, and scalable testbench structure via an API framework.
The framework consists of a FIM Testbench which is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this Testbench. UVM RAL(Register Abstraction Level) is used for CSR (Command and Status Registers) verification.
The qualified verification IPs will help to detect incorrect protocol behavior, help to focus on FIM features and accelerate the verification process.
Verification components include:
\u2022 FIM monitor to detect correct design behavior\n\u2022 FIM assertions for signal level integrity testing\n\u2022 Arm AMBA Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity\n\u2022 FIM coverage to collect functional data\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#42-base-fim-dut","title":"4.2 Base FIM DUT","text":"The hardware architecture of an Intel Agilex 7 FIM is based on the OFS hardware architecture. The following is the list of features and subsystems supported in the base shell.
\u2022 PCIe Subsystem\n\u2022 HSSI Subsystem\n\u2022 Memory Subsystem\n\u2022 HPS Subsystem\n\u2022 FME\n\u2022 AFU with PR support\n\u2022 QSFP Controllers\n\u2022 PMCI Controller, MCTP\n
Figure 2 DUT Base Shell Diagram
Figure 2 shows the high level architecture of an Intel Agilex 7 Base Shell. It has a Gen4x16, 100G Ethernet Datapath in a 2x4x25G configuration. The Intel Agilex 7 Base Shell is a shell that will enable a user to build other shell variants for a custom configuration. For the f2000x board there is one shell variant
base_x16
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43-full-chip-level-verification-archiecture-for-fim","title":"4.3 Full Chip Level Verification Archiecture for FIM","text":"Figure 3 shows a graphical representation a full chip testbench that includes major RTL blocks depicted in a OFS Intel Agilex 7 based UVM environment
Figure 3 OFS FIM Testbench
The major connection is the interface between the Xeon CPU and the FPGA where the PCIe Verification IP is connected to PCIe Subsystem. Therefore, as a full chip simulation environment, PCIe host VIP is the sole VIP/BFM used. PCIe host VIP connects to PCIe device which resides in FPGA in serial mode.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#431-testbench-components","title":"4.3.1 Testbench components","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4311-tb-top","title":"4.3.1.1 TB TOP","text":"TOP is the top level testbench and consists of a FIM DUT instance and top-level UVM Verification Environment instance. It binds RTL inputs with the verification environmnet interfaces to drive stimulus. It also has clock generation and reset driving logic.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4312-fim-verification-environment","title":"4.3.1.2 FIM Verification Environment","text":"This is the top most verification environment class and consists of the protocol specific PCI Express and AXI UVM environment VIP instances, Monitors, Scoreboards, Assertions, Functional coverage Modules and other verification components. It instantiates Virtual sequencers to control stimuli for FIM traffic from different sequencers of the VIPs.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4313-synopsys-vips","title":"4.3.1.3 Synopsys VIPs","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43131-pci-vip-as-host","title":"4.3.1.3.1 PCI VIP as Host","text":"This is Synopsys Licensed PCI Express Gen4 VIP and acts as Root Port. The PCI Express link is connected to the DUT using TX-RX lanes. Agent is an active component and includes a sequencer to generate TLPs, Driver to drive it on a PCI Express link and Monitor to check the protocol correctness.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43132-axi-streaming-vip-monitors","title":"4.3.1.3.2 AXI-Streaming VIP Monitors","text":"This is Synopsys Licensed AXI streaming interface Verification IP used as a Passive Agent to monitor AXI-ST links at various points. Please refer to Figure 3 to see all the AXI-ST monitors connected to different modules.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43133-axi4-memory-mapped-vip-monitors","title":"4.3.1.3.3 AXI4-Memory Mapped VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 memory mapped interface Verification IP used in passive mode to observe memory requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and performing data integrity checks.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43134-axi4-lite-vip-monitors","title":"4.3.1.3.4 AXI4-Lite VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used in passive mode to observe MMIO requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and perfoming data integrity checks. Please refer to Figure 3 to see all the Arm\u00ae AMBA\u00ae 4 AXI4-Lite monitors connected to different modules.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43135-axi4-lite-vip-as-pmci-master","title":"4.3.1.3.5 AXI4-Lite VIP as PMCI Master","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used to drive and observe MMIO requests as PMCI master. For each master-slave pair, the verification environment has a VIP instantiated in active mode and includes a sequencer to generate MMIO requests, driver to drive these requests on AXI-4 Lite interface to BPF and a monitor for observing protocol violations and data integrity checks.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4314-axi4-s-scoreboard","title":"4.3.1.4 AXI4-S Scoreboard","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-S scoreboard checks data integrity of source and sink components. It has input transactions from VIP monitors and a TLP to AXI converter for PCIe TLP packets. It makes sure the valid TLPs or AXI transactions are not missed while traversing from Host to AFU and reverse. The scoreboard will be disabled for error testing especially for invalid TLP requests and UR responses.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4315-virtual-sequencer","title":"4.3.1.5 Virtual Sequencer","text":"The virtual sequencer is the top-level sequencer which controls Enumeration, MMIO Requests, downstream and Upstream traffic as well as HSSI and Memory transactions. It makes sure that the transactions are ordered correctly as per the design specification while running a UVM test simulation.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4316-fim-monitor","title":"4.3.1.6 FIM Monitor","text":"The FIM monitor is used for checking the correctness of a specific FIM feature. As an example, a user can add interrupt related checks, error generation and clearing checks, Protocol checker impacts etc. in this component.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4317-fim-assertions","title":"4.3.1.7 FIM Assertions","text":"The assertion component is used for checking the signal level integrity of a specific FIM feature or behavior. As an example, we can add interrupt signal triggering and clear assertions, Error detection to error register bit assertions, protocol checker and clearing logic, FLR behavior assertion etc. in this top-level module. There are alos assertions to make sure the inputs are driven correctly to catch errors in a users simulation.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4318-fim-functional-coverage","title":"4.3.1.8 FIM Functional Coverage","text":"The FIM functional coverage component will have the functional coverage of each feature like interrupts, CSR's, MMIO requests, Byte align transactions, error reporting etc. to demonstrate a variety of FIM features. This will help us to find holes in the design as well as wide verification coverage to make sure thorough testing of a design. It will also provide a user what the FIM is capable of and how much code coverage they are testing.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#5-uvm-verification-set-up","title":"5 UVM Verification set-up","text":"To run the tutorial steps in this guide requires the following development environment:
Item Version Intel Quartus Prime Pro Intel Quartus Prime Pro 23.4 Simulator Synopsys VCS S-2021.09-SP1 or newer for UVM simulation of top level FIM Simulator (Questasim) Questasim 2023.4 or newer for UVM simulation of top level FIM"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#51-uvm-prerequisite","title":"5.1 UVM Prerequisite","text":"Retrieve OFS repositories.
The OFS FIM source code is included in the OTCShare GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories. Cloning the repo using the HTTPS method requires a personal access token. Please see this blog post for information about obtaining a personal access token Token authentication requirements for Git operations.
Navigate to location for storage of OFS source, create the top-level source directory and clone OFS repositories.
$ mkdir ofs-2024.1-1\n$ cd ofs-2024.1-1\n$ export OFS_BUILD_ROOT=$PWD\n$ git clone --branch --recurse-submodules https://github.com/ofs-f2000x-pl.git\n\nCloning into 'ofs-f2000x-pl'...'\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\n\n$ cd ofs-f2000x-pl\n$ git checkout tags/ofs-2024.1-1\n
Verify that the correct tag/branch have been checked out
$ git describe --tags\n\n$ ofs-2024.1-1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#52-license-requirements","title":"5.2 License Requirements","text":"The FIM Testbench is UVM compliant and integrates third party VIP's from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this TB. UVM RAL (Register Abstraction Layer) is used for CSR Verification.
The Qualified Verification IPs will help to detect incorrect protocol behavior easily, help to focus on FIM features and accelerate the verification process.
\u2022 VCS & DVE\n\u2022 SNPS-Assertions\n\u2022 Verdi\n\u2022 VerdiCoverage\n\u2022 VerdiSimDB\n\u2022 VerdiTransactionDebugUltra\n\u2022 VIP-AMBA-AXI-SVT\n\u2022 VIP-AMBA-STREAM-SVT\n\u2022 VIP-PCIE-SVT\n\u2022 VIP-PCIE-TS-SVT\n\u2022 VIP-PCIE-G3-OPT-SVT\n\u2022 VIP-Ethernet-SVT\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#53-software-tools-requirements","title":"5.3 Software Tools Requirements","text":"The following tools are required for successful UVM set-up
- Python 3.6.8
- Synopsys PCIE and AMBA AXI UVM VIP Q-2020.03A License
- VCS R-2020.12-SP2 License Note: Makefile can be modified to use DVE instead of Verdi
- VCS R-2020.12-SP2 License
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#54-creating-a-software-tools-script","title":"5.4 Creating a Software Tools Script","text":"The UVM tool set-up is best done by creating a simple set-up script so all applicable tools are sourced before running the tests.
The following environment variables can be pasted into a script and used prior to running the UVM verification environment
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#license-files","title":"License Files","text":"export LM_LICENSE_FILE=\nexport SNPSLMD_LICENSE_FILE=\n
The license environment variables LM_LICENSE_FILE and SNPSLMD_LICENSE_FILE can point to a server license on your system.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#general-environment-variables","title":"General Environment Variables","text":"export IOFS_BUILD_ROOT=$PWD\nexport OFS_ROOTDIR=<user_path>/ofs-f2000x-pl\nexport WORKDIR=$OFS_ROOTDIR\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#quartus-tools","title":"Quartus Tools","text":"export QUARTUS_HOME=<user_path>/intelFPGA_pro/23.4/quartus\nexport QUARTUS_ROOTDIR=$QUARTUS_HOME\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-verification-tools","title":"Synopsys Verification Tools","text":"export DESIGNWARE_HOME=<user_path>/synopsys/vip_common/vip_Q-2020.03A\nexport PATH=$DESIGNWARE_HOME/bin:$PATH\nexport UVM_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm\nexport VCS_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel\nexport PATH=$VCS_HOME/bin:$PATH\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-verification-tools","title":"QuestaSIM Verification Tools","text":"export MTI_HOME=<user_path>/mentor/questasim/2023.4/linux64\nexport PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\nexport QUESTA_HOME=$MTI_HOME\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#6-running-a-uvm-simulation-test-and-analysing-results","title":"6 Running a UVM Simulation Test and Analysing Results","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#61-simulation","title":"6.1 Simulation","text":"The default simulator used in the simulation script is Synopsys VCS-MX. Users can refer to the options and adopt the options for other simulators. The script is a makefile that calls vlogan, vcs and simv for compilation, elaboration and simulation, respectively.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#62-file-structure","title":"6.2 File Structure","text":"After cloning the repo, the verification and ofs-common directories contain all UVM verification related files. The directory structure is shown in Figure 4 below.
Figure 4 UVM Verification Directory File Structure
ofs-f2000x-pl/verification/testbench has a testbench, uvm env, virtual sequencer, RAL etc.
ofs-f2000x-pl/tests contains all uvm tests and sequences.
Users can run the simulation under \"ofs-f2000x-pl/verification/scripts\" directory and the simulation result is outputted to a \"sim\" directory for Synopsys VCS or sim_msim for Questasim.
The simulation result folder is named after the test name with increasing suffix number. If user runs the same test multiple times, the suffix is incremented by 1 each time.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#63-uvm-test-suite","title":"6.3 UVM Test Suite","text":"The UVM environment contains a variety of tests that have been developed to test out the FIM portion of OFS.
The table below has four columns which describe the \"Test Name\", \"DUT Scope\", \"Test Scenario\" and the \"Checking Criteria\".
Tests are located at ofs-f2000x-pl/ofs-common/verification/fpga_family/agilex/tests
Test Name DUT Scope Test Scenario Checking Criteria afu_mmio_flr_pf0_test PF0 FLR Reset Apply FLR Reset for PF0 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF0 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf0_test PF0_VF0_FLR Reset Apply FLR Reset for PF0_VF0 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf1_test PF0_VF1_FLR Reset Apply FLR Reset for PF0_VF1 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf2_test PF0_VF2_FLR Reset Apply FLR Reset for PF0_VF2 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf1_vf0_test PF1_VF0_FLR Reset Apply FLR Reset for PF1_VF0 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf2_test PF2 FLR Reset Apply FLR Reset for PF2 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF2 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf3_test PF3 FLR Reset Apply FLR Reset for PF3 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF3 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf4_test PF4 FLR Reset Apply FLR Reset for PF4 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF4 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_stress_5bit_tag_test AFU-StressAFU-Stress To check the AFU Stress by sending traffic with 5bit tag from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_8bit_tag_test AFU-Stress To check the AFU Stress by sending traffic with 5bit tag from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_test AFU-Stress 1. Initiate transactions to all the supported PF/VF from PCIE VIP and ensure that traffic is sent to all blocks of the AFU. 2. Ensure that CE/HE-LB/HE-MEM/HSSI/BPF/FME are seeing traffic. 3. Ensure that HE-LB/HE-MEM/CE sends DMWR/DMRD requests to PCIE VIP. 4. Ensure the Mux/DeMux blocks is able to handle the traffic based on the PF's/VF's and proper muxing/demuxing happens. Data checking bar_32b_test PCIe MMIO Path Set the BAR values to 32bit and test mmio access BAR address, Register Base Offset bar_64b_test PCIe MMIO Path Set the BAR values to 64bit and test mmio access BAR address, Register Base Offset dfh_walking_test DFH DFH walking offset checking, eol checking -> tb emif_csr_test EMIF CSR access data checking fme_csr_test FME CSR CSR accesses data checking fme_err_intr_test Interrupt FME Interrupt request using FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_intr_test Interrupt FME interrupt request using RAS ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_multi_err_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_hssi_csr_test HE-HSSI CSR accesses data checking he_hssi_rx_lpbk_25G_10G_test HE-HSSI Sending back to back ethernet data traffic with 25G speed on HSSI RX Port0-7 lanes using Ethernet VIPs Enable the loopback mode in HE-HSSI and compare the pkts recived on HSSI TX Port(DATA CHECKING) he_hssi_tx_lpbk_P0_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port0 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P1_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port1 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P2_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port2 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P3_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port3 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P4_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port4 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P5_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port5 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P6_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port6 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P7_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port7 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_lpbk_cont_test HE-LPBK Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_lpbk_csr_test HE-LPBK CSR accesses data checking he_lpbk_flr_rst_test HE-LPBK FLR RST is generated in HE_LPBK access data checking, counter checking he_lpbk_long_rst_test HE-LPBK Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_lpbk_long_test HE-LPBK Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_lpbk_multi_user_intr_test HE-LPBK generate user HE_LB interrupt interrupt checking he_lpbk_rd_cont_test HE-LPBK Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_rd_test HE-LPBK Read only mode. Randomize num_lines, addresses, req_len counter checking he_lpbk_reqlen16_test HE-LPBK To check the behavior of HE_LPK block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_lpbk_reqlen1_test HE-LPBK Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_lpbk_reqlen2_test HE-LPBK Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_lpbk_reqlen4_test HE-LPBK Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_lpbk_reqlen8_test HE-LPBK Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_lpbk_rst_in_middle_test PCIe MMIO Path Set HE_LPK in all the modes randomly and iterate the transactions in loop. At the end of every loop assert the Soft reset in the middle of the transactions Register Base Offset he_lpbk_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_thruput_contmode_test HE-LPBK Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking he_lpbk_thruput_test HE-LPBK Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_lpbk_wr_cont_test HE-LPBK Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_wr_test HE-LPBK Write only mode. Randomize num_lines, addresses, req_len counter checking he_mem_cont_test HE-MEM Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking, counter checking he_mem_csr_test HE-MEM CSR accesses data checking he_mem_lpbk_long_rst_test HE-MEM Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_mem_lpbk_long_test HE-MEM Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_mem_lpbk_reqlen16_test HE-MEM To check the behavior of HE_LPK block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_mem_lpbk_reqlen1_test HE-MEM Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_mem_lpbk_reqlen2_test HE-MEM Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_mem_lpbk_reqlen4_test HE-MEM Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_mem_lpbk_reqlen8_test HE-MEM Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_mem_lpbk_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_multi_user_intr_test Interrupt Back to back multiple User interrupt request from HE MEM Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read for multiple back to back request he_mem_rd_cont_test HE-MEM Read only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_test HE-MEM Read only mode. Randomize num_lines, addresses, req_len counter checking he_mem_thruput_contmode_directed_test HE-MEM Set HE_MEM in Thrput Continuous mode and test data checking, counter checking he_mem_thruput_contmode_test HE-MEM Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_thruput_test HE-MEM Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_mem_user_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_mem_wr_cont_test HE-MEM Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_wr_test HE-MEM Write only mode. Randomize num_lines, addresses, req_len counter checkingt he_random_test All HEs Enable all HEs and randomize modes data checking if in lpbk mode, counter checking helb_rd_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Read only mode Measure the performance helb_rd_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Read only mode Measure the performance helb_rd_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Read only mode Measure the performance helb_thruput_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_4cl_5bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_8bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Thruput mode Measure the performance helb_wr_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Write only mode Measure the performance helb_wr_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Write only mode Measure the performance helb_wr_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Write only mode Measure the performance hssi_ss_test HSSI CSR accesses data checking malformedtlp_pcie_rst_test Protocol Checker generate mllformed TLP protocol error and initiate pcie reset to clear the error Check for error malformedtlp_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MaxTagError_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transactions are completing. mem_tg_csr_test MEM-TG CSR access data checking mem_tg_traffic_gen_test MEM-TG This test checks the MEM_TG traffic generator flow data checking mini_smoke_test All HEs shorter simpler version of random test for turn-in sanity check data checking if in lpbk mode, counter checking mix_intr_test Interrupt Mix interrupt testcase to send multiple FME and User interrupt request simultaneously Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests through different sources - FME and HE-MEM modules mmio_pcie_mrrs_128B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_128B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Read Checking valid possible combination of MPS & MRRS mmio_stress_nonblocking_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux with non-blocking MMIO reads data checking mmio_stress_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux data checking mmio_test PCIe MMIO Path MMIO targeting PF0(ST2MM, FME, PMCI, QSFP, HSSI SS), PF1, PF2,PF3, PF4, PF1.VF1, PF1.VF2 data checking mmio_unimp_test PCIe MMIO Path MMIO acccess to unimplemented addresses data checking MMIODataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOTimedout_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. pcie_csr_test PCIESS Earlier msix registers were in fme block but now it has moved from fme to pciess. Hence coded a seperate test for msix data checking pcie_pmci_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pcie_pmci_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM (Vendor Defined Message) single packet received from PCIe HIA subsystem to PMCI controller over APF and BPF via AXI4-lite memory write request pmci_csr_test PMCI CSR CSR accesses data checking pmci_fme_csr_test PMCI FME CSR CSR accesses data checking pmci_pcie_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pcie_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM single packet received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pciess_csr_test PMCI PCIESS CSR CSR accesses data checking pmci_qsfp_csr_test PMCI QSFP CSR CSR accesses data checking port_gasket_csr_test PORT GASKET CSR accesses data checking qsfp_csr_test QSFP CSR accesses data checking TxMWrDataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. TxMWrInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. uart_intr_test UART Checking Generates UART interrupt Check interrupt UnexpMMIORspErr_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. vdm_err_vid_test Vendor ID check generate a packet with undefined Vendor-ID from Host to PMCI_SS ID check The next section describes how to compile and build the UVM environment prior to running each UVM test and analyinsg the results in the log files
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#64-ip-compile","title":"6.4 IP Compile","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs","title":"Synopsys VCS","text":"To compile all IPs for the Synopsys VCS simulater:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk cmplib_adp\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd","title":"Questasim (TBD)","text":"To compile all IPs for the Questasim simulater:
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk cmplib_adp\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#65-rtl-test-bench-compile","title":"6.5 RTL & Test Bench Compile","text":"The RTL file list for compilation is located here: verification/scripts/rtl_comb.f
The TB file list for compilation is located here: verification/scripts/ver_list.f
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_1","title":"Synopsys VCS","text":"To compile RTL and Testbench for the Synopsys VCS simulater
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_adp DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_1","title":"Questasim (TBD)","text":"To compile RTL and Testbench for the Questasim simulater
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk build_adp DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#66-ip-and-rtl-test-bench-compile","title":"6.6 IP and RTL & Test Bench Compile","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_2","title":"Synopsys VCS","text":"If the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_all DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_2","title":"Questasim (TBD)","text":"If the user wants to compile all IPs and RTL Testbench in one command for Questasim then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk build_all DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_3","title":"Synopsys VCS","text":"To run a simulation for Synopsys VCS:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk run TESTNAME=ofs_mmio_test DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_3","title":"Questasim (TBD)","text":"To run a simulation for Questasim:
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk run TESTNAME=ofs_mmio_test DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_4","title":"Synopsys VCS","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation.
gmake -f Makefile_VCS.mk build_adp DUMP=1\n\n gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> DUMP=1\n
Or
gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_4","title":"Questasim (TBD)","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Questasim build and simulation.
gmake -f Makefile_MSIM.mk build_adp DUMP=1\n\n gmake -f Makefile_MSIM.mk run TESTNAME=<test_case_name> DUMP=1\n
Or
gmake -f Makefile_MSIM.mk build_run TESTNAME=<test_case_name> DUMP=1\n
There are some optimizations in the Table below for convenience if you want to bypass some commands for both Synopsys VCS and Questasim:
Command (Synopsys VCS) Command (Questasim) Details gmake -f Makefile_VCS.mk build_all DUMP=1 gmake -f Makefile_MSIM.mk build_all DUMP=1 compile IP + compile RTL gmake -f Makefile_VCS.mk build_run TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk build_run TESTNAME= DUMP=1 compile RTL + run test gmake -f Makefile_VCS.mk do_it_all TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk do_it_all TESTNAME= DUMP=1 compile IP, RTL and run test gmake -f Makefile_VCS.mk rundb TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk rundb TESTNAME= DUMP=1 run test in sim dir + over-writes content"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#67-uvm-regression-test","title":"6.7 UVM Regression Test","text":"cd $VERDIR/scripts\n\npython uvm_regress.py -l -n 8 -p adp -k top_pkg -s vcs -c none\n\nFor Regression in VCS with top/test package, execute the following command \n python uvm_regress.py -l -n 8 -p adp -k top_pkg -s vcs -c none\n\nResults are created in a sim directory ($VERDIR/sim) with individual testcase log dir\n\nFor Regression in MSIM with top/test package, execute the following command \n python uvm_regress.py -l -n 8 -p adp -k top_pkg -s msim -c none\n
Results are created in a sim directory ($VERDIR/sim_msim) with individual testcase log dir
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#68-uvm-waveform-and-transcript-analysis","title":"6.8 UVM Waveform and Transcript Analysis","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_5","title":"Synopsys VCS","text":"Running Synopsys VCS UVM tests will generate a ofs-f2000x-pl/verification/sim directory
\u2022 All build time logs are located at ofs-f2000x-pl/verification/sim\n\n\u2022 Each testcase will have separate directory inside sim ofs-f2000x-pl/verification/sim/<test_case_name>\n
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Synopsys VCS. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 5
Figure 5 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 6
Figure 6 trans.log
The waveform generated is named as \"inter.vpd\". To open the waveform, go to simulation result directory and run
dve -full64 -vpd inter.vpd &\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_5","title":"Questasim (TBD)","text":"Running Questasim UVM tests will generate a ofs-f2000x-pl/verification/sim_msim directory
\u2022 All build time logs are at ofs-f2000x-pl/verification/sim_msim\n\n\u2022 Each testcase will have separate directory inside sim ofs-f2000x-pl/verification/sim_msim/<test_case_name>\n
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Questasim. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 7
Figure 7 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 8
Figure 8 trans.log
The waveform generated is named as \"vsim.wlf\". To open the waveform, go to simulation result directory and run
vsim -view vsim.wlf &\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#69-uvm-coverage-analysis","title":"6.9 UVM Coverage Analysis","text":"The following command allows to run a single testcase with coverage enabled
gmake -f Makefile_VCS.mk cmplib_adp && gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1 COV_FUNCTIONAL=1&& gmake -f Makefile_VCS.mk run TESTNAME=<TESTCASE-NAME> DUMP=1 DEBUG=1 COV_FUNCTIONAL=1 &\n
The following command shows how to merge and generate the coverage report
urg -dir <$VERDIR/sim/simv.vdb> <$VERDIR/sim/regression.vdb> -format both -dbname <regression_database_name>\n
This will generate both urgreport directory and .vdb file Multiple regression.vdb from different regressions can be merged with the same command.
e.g \"urg -dir <path1_till_simv.vdb> <path1_till_regression.vdb> <path2_till_regression.vdb> -report <dir> -format both -dbname <dirname>\"\n
The following commands shows how to launch DVE and check the coverage reports
To open DVE of a single regression or testcase, execute:\n\n dve -full64 -cov -covdir simv.vdb regression.vdb &\n\nTo open DVE of a merged regression test, execute:\n\n dve -full64 -cov -covdir <dirname.vdb> &\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#7-csr-verification-using-uvm-ral","title":"7 CSR Verification using UVM RAL","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#71-overview","title":"7.1 Overview","text":"The UVM Register Layer provides a standard base class library that enable users to implement the object-oriented model to access the DUT registers and memories. The UVM Register Layer is also referred to as UVM Register Abstraction Layer (UVM RAL). Design registers can be accessed independently of the physical bus interface. i.e. by calling read/write methods. This is shown in Figure 9 below.
Figure 9 RAL UVM Testbench
The RAL register models for different CSR's mimics the design registers. All RAL files are located here.
ofs-f2000x-pl/verification/testbench/ral\n
The RAL model is generated through the Synopsys RALGEN tool and is used for CSR verification.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#72-ral-integration","title":"7.2 RAL Integration","text":"For UVM RAL model integration to the environment, adapters for each CSR is implemented and integrated into the Testbench Environment. It is used to convert the PCIe bus sequence items into uvm_reg_bus_op and vice versa. The CSR test cases pick up all the registers from the respective CSR blocks and perform a default value, wr/rd check on them.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#73-ral-model-generation","title":"7.3 RAL Model Generation","text":"Steps for RAL model generation
Excel(xls) file containing the registers is required. Make sure there are separate xls files for each CSR and the worksheet name must contain the name \"reg_fields\".
Excel sheet snapshot example below for EMIF_CSR.xls located at /ipss/mem/rtl/adp
\u2022 Navigate to ofs-f2000x-pl/ofs-common/verification/common/scripts/ral\n\u2022 Copy the excel file (xls) to the above area\n\u2022 In the bash terminal run mk_ral.sh <Excel sheet name without extension > <output *.sv file name without ral_ prepended >\n\u2022 The above steps generate two ral *.sv files. File with _cov suffix is a coverage enabled ral model. \n\u2022 Copy *.sv files to ofs-f2000x-pl/verification/testbench/ral\n
\u2022 As an example to generate ral_ac_ce.sv from AC_CE_CSR.xls file the command is\n\n mk_ral.sh AC_CE_CSR ac_ce\n
This generates two ral models (ral_ac_ce.sv and ral_ac_ce_cov.sv)
To add new registers
\u2022 To create new registers, copy existing ones and modify the cells in the xls. Make sure the last line is also a copied blank line\n\u2022 Follow all the steps of RAL model generation\n
To Generate a RAL model when a new xls sheet is created for a new component
\u2022 Copy the relevant xls sheet to ofs-f2000x-pl/ofs-common/verification/common/scripts/ral\n\u2022 Follow all the steps of RAL model generation\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#74-top-level-verification-architecture-for-csr-testing","title":"7.4 Top Level Verification Architecture for CSR testing","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#741-testbench-components","title":"7.4.1 Testbench components","text":"The testbench components for RAL are defined below
\u2022 ral_reg_iofs_* (uvm_reg) generated by the steps as mentioned in section 5.3\n
The uvm register class is written by extending the uvm_reg. A register represents a set of fields that are accessible as a single entity Each register contains any number of fields, which mirror the values of the corresponding elements in hardware
\u2022 ral_block_iofs_* (uvm_block) generated in the same register file\n
A register model is an instance of a register block, which may contain any number of registers, register files, memories, and other blocks
\u2022 ral_block_ofs (uvm_block) \u2013 Contains all the CSR block instantiations\n\u2022 Reg2vip_*_adapter (uvm_reg_adapter) \u2013 This class defines an interface for converting between uvm_reg_bus_op and a specific bus transaction. For each CSR a respective adapter is present\n
All the components are defined in ofs-f2000x-pl/ofs-common/verification/testbench
Integration of components in testbench
\u2022 The RAL blocks and adapters for each CSR is instantiated in tb_env\n\u2022 The bar range for each CSR is also set in the tb_env\n
Sample Environment Integration snippets
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#8-modifying-uvm-testbench","title":"8 Modifying UVM Testbench","text":"The next sections describe what needs to be considered when modifying the UVM, adding a new interface to the testbench and creating a new UVM test for a customised OFS Accelerator platform.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#81-modifying-uvm-environment-for-new-shell-variant","title":"8.1 Modifying UVM environment for new Shell Variant","text":"OFS f2000x comprises a shell based on PCIe Gen4x16 and is named base_x16
This base_x16 shell is described by an RTL file list, IP File lists and setup scripts to complete the build flow
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#82-modifying-uvm-environment-and-setting-up-compile-and-run-flow","title":"8.2 Modifying UVM environment and setting up Compile and Run flow","text":"All the variants can mostly reuse the existing UVM infrastructure to setup the build and run flow
\u2022 Create directory under $OFS_BUILD_ROOT new variant e.g ofs-n9000\n\u2022 Change directory to $OFS_BUILD_ROOT/ofs-n9000/verification/scripts\n\u2022 modify Makefile it to point to the new RTL, IP and script files.\n
Following these three steps above will enable the build and sim flow to run the existing UVM TB and tests with new IOFS n9000 variant.
Adding a new interface requires signal connections in the testbench. An additional BFM or verification IP is needed to drive the new interface. The main testbench file tb_top.sv is found at the following location
ofs-f2000x-pl/verification/testbench\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#83-adding-a-new-ral-directory","title":"8.3 Adding a new RAL directory","text":"In case the design has many register changes and the user decides to generate all the new RAL models instead of reusing from existing base RAL models, the following steps will help to create and integrate a new RALDIR in the UVM environment.
\u2022 Generate the new RAL files in desired directory. Preferably under the \"ofs-f2000x-pl/verification/common/testbench\" \n\u2022 By default, the Makefile points to base FIM RAL so set the RALDIR path in the Makefile to the new generated RAL file directory\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#84-modifying-tbtest-files-for-new-variant","title":"8.4 Modifying TB/Test files for new Variant","text":"Create a define for the new variant. e.g 'define FIM_NEW. If you are modifying common files then add new logic under this define so other projects will not get affected with variant specific change.
If there are more changes, please create separate \"testbench\" and \"test\" folders under this new variant.
\u2022 Extend variant specific env file from base env file to add variant specific changes.\n\u2022 Create new test/seq lib files in \"tests\" folder.\n\u2022 Create new variant package, add new TB/Tests/Sequence lib files and also import the base package files.\n
If you are adding new files then make sure it's included in Makefile for the build+run flow.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#85-uvm-pcie-drivers","title":"8.5 UVM PCIe Drivers","text":"The \"svt_pcie_driver_app_transaction_base_sequence\" is part of Synopsys PCIe VIP library. You can find the sequence definition in the following two directories
\u2022 Navigate to \"$DESIGNWARE_HOME/vip/svt/pcie_svt/Q-2020.03/sverilog/src/vcs/svt_pcie_driver_app_transaction_sequence_collection.svp\" file. All the base and PCIe sequences are defined in this file.\n\n\u2022 When the OFS UVM build command is executed, it creates \"vip\" directory under \"$OFS_BUILD_ROOT/ofs-f2000x-pl/verification\". You can also find the same sequence file at \"$IOFS_BUILD_ROOT/ofs-f2000x/verification/vip/pcie_vip/src/sverilog/vcs/svt_pcie_driver_app_transaction_sequence_collection.sv\"\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/ftile_devkit/","title":"F-Series (2xF-tile) Development Kit Collateral for OFS","text":"This folder contains applicable collateral for OFS PCIe Attach reference shell targeting the F-Series (2xF-tile) Development Kit DK-DEV-AGF027F1.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/","title":"Shell Developer Guide for Open FPGA Stack: Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a guide for OFS Agilex PCIe Attach developers targeting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). The following topics are covered in this guide:
- Compiling the OFS Agilex PCIe Attach FIM design
- Simulating the OFS Agilex PCIe Attach design
- Customizing the OFS Agilex PCIe Attach FIM design
- Configuring the FPGA with an OFS Agilex PCIe Attach FIM design
The FIM Development Walkthroughs Table lists all of the walkthroughs provided in this guide. These walkthroughs provide step-by-step instructions for performing different FIM Development tasks.
Table: FIM Development Walkthroughs
Walkthrough Name Category Install Quartus Prime Pro Software Setup Clone FIM Repository Setup Set Development Environment Variables Setup Set Up Development Environment Setup Compile OFS FIM Compilation Manually Generate OFS Out-Of-Tree PR FIM Compilation Change the Compilation Seed Compilation Run Individual Unit Level Simulation Simulation Run Regression Unit Level Simulation Simulation Add a new module to the OFS FIM Customization Modify and run unit tests for a FIM that has a new module Customization Hardware test a FIM that has a new module Customization Debug the FIM with Signal Tap Customization Compile the FIM in preparation for designing your AFU Customization Resize the Partial Reconfiguration Region Customization Modify PCIe Configuration Using OFSS Customization Modify PCIe Configuration Using IP Presets Customization Create a Minimal FIM Customization Migrate to a Different Agilex Device Number Customization Modify the Ethernet Sub-System to 2x4x10GbE Customization Modify the Ethernet Sub-System to 3x4x10GbE Customization Remove the HPS Customization Set up JTAG Configuration Program the FPGA via JTAG Configuration"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#111-knowledge-pre-requisites","title":"1.1.1 Knowledge Pre-Requisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex 7 PCIe Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)).
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Signal Tap Logic Analyzer tool software.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex PCIe Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex PCIe Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new acceleration card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1211-top-level","title":"1.2.1.1 Top Level","text":"Figure: OFS Agilex PCIe Attach fseries-dk FIM Top-Level Diagram
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex PCIe Attach design are listed in the Release Capabilities Table. It describes the capabilities of the fseries-dk hardware as well as the capabilities of the default OFS Agilex PCIe Attach design targeting the fseries-dk.
Table: Release Capabilities
Interface fseries-dk Hardware Capabilities OFS Agilex PCIe Attach Default Design Implementation Host Interface[1] PCIe Gen4x8 PCIe Gen4x16 Network Interface 1 QSFP-56 cage1 QSFPDD-56 Cage 2x4x25GbE External Memory 3xDDR4 DIMMs sockets - 72-bits (1 available for HPS) 2xDDR4 - 2400MHz - 8Gb (1Gbx8) - 64-bits - No ECC1xDDR4 - 2400MHz - 16Gb (2Gbx8) - 40-bits - With ECC - For HPS [1] F-Tile ES version development kit has a form factor of PCIe Gen4x16, however it only supports PCIe Gen4x8. The OFS Agilex PCIe Attach design implements PCIe Gen4x16 with the assumption that it will train down to PCIe Gen4x8.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex PCIe Attach fseries-dk FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem AXI Streaming IP for PCI Express User Guide 790711 Memory Subsystem Memory Subsystem Intel FPGA IP User Guide for Intel Agilex OFS 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workload in the OFS Agilex PCIe Attach fseries-dk FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI Used to exercise and characterize HSSI interfaces. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex PCIe Attach fseries-dk FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the host software and AFU and board peripherals. The APF Address Map Table describes the address mapping of the APF, followed by the BPF Address Map Table which describes the address mapping of the BPF.
Table: APF Address Map
Address Size (Bytes) Feature 0x00000\u20130x3FFFF 256K Board Peripherals (See BPF Address Map table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket:4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x80000 \u2013 0x80FFF 4K AFU Error Reporting Table: BPF Address Mapping
Address Size (Bytes) Feature 0x00000 - 0x0FFFF 64K FME 0x10000 - 0x10FFF 4K PCIe 0x11000 - 0x11FFF 4K Reserved 0x12000 - 0x12FFF 4K QSFP0 0x13000 - 0x13FFF 4K QSFP1 0x14000 - 0x14FFF 4K HSSI 0x15000 - 0x15FFF 4K EMIF"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customizations Table lists the general user flows for OFS Agilex PCIe Attach fseries-dk FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customizations
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify PCIe Configuration Using OFSS Modify PCIe Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 2x4x10GbE Modify the Ethernet Sub-System to 3x4x10GbE Remove the HPS"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA acceleration card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Please see the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up the environment for deployment machines.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 N/A Quartus Prime Software Quartus Prime Pro Version 24.1 for Linux + Patches 0.18, 0.26 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A git 1.8.3.1 or later Section 1.3.1.2 FIM Source Files ofs-2024.2-1 Section 1.3.2.1"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"The source files for the OFS Agilex PCIe Attach FIM are provided in the following repository: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
Some essential directories in the repository are described as follows:
ofs-agx7-pcie-attach\n| syn // Contains files related to synthesis\n| | board // Contains synthesis files for several cards, including the fseries-dk | | | fseries-dk // Contains synthesis files for fseries-dk\n| | | | setup // Contains setup files, including pin constraints and location constraints\n| | | | syn_top // Contains Quartus project files\n| verification // Contains files for UVM testing\n| ipss // Contains files for IP Sub-Systems\n| | qsfp // Contains source files for QSFP Sub-System\n| | hssi // Contains source files for HSSI Sub-System\n| | pmci // Contains source files for PMCI Sub-System (not used in F-Tile FIM)\n| | pcie // Contains source files for PCIe Sub-System\n| | mem // Contains source files for Memory Sub-System\n| sim // Contains simulation files\n| | unit_test // Contains files for all unit tests\n| | | scripts // Contains script to run regression unit tests\n| license // Contains Quartus patch\n| ofs-common // Contains files which are common across OFS platforms\n| | verification // Contains common UVM files\n| | scripts // Contains common scripts\n| | | common\n| | | | syn // Contains common scripts for synthesis, including build script\n| | | | sim // Contains common scripts for simulation\n| | tools // Contains common tools files\n| | | mk_csr_module // Contains common files for CSR modules\n| | | fabric_generation // Contains common files for APF/BPF fabric generation\n| | | ofss_config // Contains common files for OFSS configuration tool\n| | | | ip_params // Contains default IP parameters for certain Sub-Systems when using OFSS\n| | src // Contains common source files, including host exercisers\n| tools //\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| | | hssi // Contains OFSS files for Ethernet-SS configuraiton\n| | | memory // Contains OFSS files for Memory-SS configuration\n| | | pcie // Contains OFSS files for PCIe-SS configuration\n| | | iopll // Contains OFSS files for IOPLL configuration\n| src // Contains source files for Agilex PCIe Attach FIM\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | includes // Contains source file header files\n| | top // Contains top-level source files, including design top module\n| | afu_top // Contains top-level source files for AFU\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 PCIe Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n
-
Check out the correct tag of the repository
cd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.2-1)\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-agx7-pcie-attach\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.FTILE_DK_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.FTILE_DK_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 24.1 for Linux with Agilex FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 24.1 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
git 1.8.3.1 or later.
-
Verify version number
git --version\n
Example output:
git version 1.8.3.1\n
-
Clone the ofs-agx7-pcie-attach repository. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.18, 0.26. All required patches are provided in the Assets of the OFS FIM Release Tag: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
-
Extract and unzip the patch-agx7-2024-1.tar.gz file.
tar -xvzf patch-agx7-2024-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-24.1-0.18-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
- Change the Compilation Seed
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This sections describes the theory behind FIM compilation.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2121-top-level-ofss-file","title":"2.1.2.1 Top Level OFSS File","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2122-base-ofss-file","title":"2.1.2.2 Base OFSS File","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2123-pcie-ofss-file","title":"2.1.2.3 PCIe OFSS File","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2124-iopll-ofss-file","title":"2.1.2.4 IOPLL OFSS File","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2125-memory-ofss-file","title":"2.1.2.5 Memory OFSS File","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is: $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/fseries-dk/syn_top/output_files.
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes the images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Name Description ofs_top.sof The FIM design SRAM Object File; a binary file of the compiled FIM image. ofs_top_hps.sof The build assembly process combines the FPGA ofs_top.sof programming file with u-boot-spl-dtb.hex to produce this file."},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <PATH_OF_RELOCATABLE_PR_TREE> <BOARD_TARGET> <WORK_DIRECTORY>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <PATH_OF_RELOCATABLE_PR_TREE> String Specifies the location of the relocatable PR directory tree to be created. <BOARD_TARGET> fseries-dk Specifies the name of the board target. <WORK_DIRECTORY> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_hps.sof\n\u2514\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#224-he_null-fim","title":"2.2.4 HE_NULL FIM","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex PCIe Attach FIM for fseries-dk:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. The following is used build the default fseries-dk design:
./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk\n
The build command options allow for many modifications to the shell design at build time. The following tool is provided to help you conveniently get the build command for a specific shell configuration:
OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command. Note: If no OFSS file is specified, the build script will use the <target>.ofss file by default.
- Once the build script is complete, the build summary should report that the build is complete and passes timing. For example:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: fseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#226-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM. This can be automatically done for you if you run the build script with the -p option. This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the fseries-dk OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_fseries-dk/pr_build_template fseries-dk work_fseries-dk\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
set_global_assignment -name SEED 1\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#3-fim-simulation","title":"3. FIM Simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config> OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> fseries-dk | n6001 Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional Refer to the Run Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files automatically; refer to the Run Regression Unit Level Simulation section for step-by-step instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#321-walkthrough-run-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Run Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the fseries-dk
./gen_sim_files.sh fseries-dk\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"You may use the regression script regress_run.py to run some or all of the unit tests available with a single command. The regression script is in the following location:
$OFS_ROOTDIR/sim/unit_test/scripts/regress_run.py\n
The usage of the regression script is as follows:
regress_run.py [-h] [-l] [-n <num_procs>] [-k <test_package>] [-s <simulator>] [-g] [--ofss <ip_config>] [-b <board_name>] [-e]\n
The Regression Unit Test Script Options table describes the options for the regress_run.py script.
Table: Regression Unit Test Script Options
Field Options Description -h | --help N/A Show the help message and exit -l | --local N/A Run regression locally -n | --n_procs Integer Maximum number of processes/tests to run in parallel when run locally. This has no effect on farm runs. -k | --pack all | fme | he | hssi | list | mem | pmci Test package to run during regression. The \"list\" option will look for a text file named \"list.txt\" in the \"unit_test\" directory for a text list of tests to run (top directory names). The default test package is all. -s | --sim vcs | vcsmx | msim Specifies the simulator used for the regression tests. The default simulator is vcs -g | --gen_sim_files N/A Generate simulation files. This only needs to be done once per repo update. This is the equivalent of running the gen_sim_files.sh script. -o | --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. -b | --board_name n6000 | n6001 | fseries-dk Specifies the board target -e | --email_list String Specifies email list to send results to multiple recipients The log for each unit test that is run by the regression script is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#331-walkthrough-run-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Run Regression Unit Level Simulation","text":"Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a test list file to only run the unit level simulations that are supported for the fseries-dk FIM.
touch $OFS_ROOTDIR/sim/unit_test/list.txt\n
Copy the following list into the new file. You may remove tests from this list as desired.
./bfm_test/set_params.sh\n./csr_test/set_params.sh\n./dfh_walker/set_params.sh\n./flr/set_params.sh\n./fme_csr_access/set_params.sh\n./fme_csr_directed/set_params.sh\n./he_lb_test/set_params.sh\n./he_mem_lb_test/set_params.sh\n./he_null_test/set_params.sh\n./hssi_csr_test/set_params.sh\n./hssi_kpi_test/set_params.sh\n./hssi_test/set_params.sh\n./indirect_csr/set_params.sh\n./mem_ss_csr_test/set_params.sh\n./mem_ss_rst_test/set_params.sh\n./mem_tg_test/set_params.sh\n./pcie_ats_basic_test/set_params.sh\n./pcie_csr_test/set_params.sh\n./pcie_ss_axis_components/set_params.sh\n./pf_vf_access_test/set_params.sh\n./port_gasket_test/set_params.sh\n./qsfp_test/set_params.sh\n./remote_stp_test/set_params.sh\n./uart_csr/set_params.sh\n
-
Navigate to the unit test scripts directory.
cd $OFS_ROOTDIR/sim/unit_test/scripts\n
-
Run regression test with the your desired options. For example, to simulate with the options to generate simulation files, run locally, use 8 processes, run all tests, use VCS simulator, and target the fseries-dk:
python regress_run.py -g -l -n 8 -k list -s vcs -b fseries-dk\n
Note: You may run all available tests by using -k all instead of creating and using -k list, however not all tests are supported depending on the target board.
-
Once all tests are complete, check that the tests have passed.
Example output:
Passing Unit Tests:24/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n bfm_test:............... PASS -- Time Elapsed:0:01:14.600452\n csr_test:............... PASS -- Time Elapsed:0:01:30.972054\n dfh_walker:............. PASS -- Time Elapsed:0:01:15.179127\n flr:.................... PASS -- Time Elapsed:0:01:14.579890\n fme_csr_access:......... PASS -- Time Elapsed:0:00:48.545993\n fme_csr_directed:....... PASS -- Time Elapsed:0:00:54.702789\n he_lb_test:............. PASS -- Time Elapsed:0:02:11.371956\n he_mem_lb_test:......... PASS -- Time Elapsed:0:41:32.226191\n he_null_test:........... PASS -- Time Elapsed:0:01:11.791063\n hssi_csr_test:.......... PASS -- Time Elapsed:0:44:10.611323\n hssi_kpi_test:.......... PASS -- Time Elapsed:2:28:24.465424\n hssi_test:.............. PASS -- Time Elapsed:2:23:52.603328\n indirect_csr:........... PASS -- Time Elapsed:0:01:02.535460\n mem_ss_csr_test:........ PASS -- Time Elapsed:0:23:37.683859\n mem_ss_rst_test:........ PASS -- Time Elapsed:0:45:19.603426\n mem_tg_test:............ PASS -- Time Elapsed:0:28:59.823955\n pcie_ats_basic_test:.... PASS -- Time Elapsed:0:01:10.104139\n pcie_csr_test:.......... PASS -- Time Elapsed:0:01:10.891950\n pcie_ss_axis_components: PASS -- Time Elapsed:0:02:04.448343\n pf_vf_access_test:...... PASS -- Time Elapsed:0:01:09.465886\n port_gasket_test:....... PASS -- Time Elapsed:0:01:11.912088\n qsfp_test:.............. PASS -- Time Elapsed:0:05:10.887379\n remote_stp_test:........ PASS -- Time Elapsed:0:01:14.684407\n uart_csr:............... PASS -- Time Elapsed:0:01:34.763679\nFailing Unit Tests: 0/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n Number of Unit test results captured: 24\nNumber of Unit test results passing.: 24\nNumber of Unit test results failing.: 0\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4-fim-customization","title":"4. FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Table: FIM Customization Walkthroughs
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify PCIe Configuration Using OFSS Modify PCIe Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 2x4x10GbE Modify the Ethernet Sub-System to 3x4x10GbE Remove the HPS"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a information for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example will add a hello_fim module to the design. The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the Host. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the fseries-dk card. The process for these are described in this section.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can be mastered by the Host. The BPF is an interconnect generated by Platform Designer. The Hello FIM BPF Interface Diagram figure shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
Figure: Hello FIM BPF Interface Diagram
The BPF fabric is defined in $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt.
We will add the Hello FIM module to an un-used address space in the MMIO region. The Hello FIM MMIO Address Layout table below shows the MMIO region for the Host with the Hello FIM module added at base address 0x16000.
Table: Hello FIM MMIO Address Layout
Offset Feature CSR set 0x00000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 HSSI Interface 0x15000 EMIF Interface 0x16000 Hello FIM"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the Hello FIM CSR table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Table: Hello FIM CSR
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Make hello_fim source directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create hello_fim_top.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_top.sv\n
Copy the following code into hello_fim_top.sv:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : September 2023\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h001,\nparameter bit [3:0] FEAT_VER = 4'h1,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create hello_fim_com.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_com.sv\n
Copy the following code to hello_fim_com.sv:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h001,\nparameter bit [3:0] FEAT_VER = 4'h1,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Create hello_fim_design_files.tcl file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_design_files.tcl\n
Copy the following code into hello_fim_design_files.tcl
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf to include Hello FIM module
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top_sources.tcl to include Hello FIM design files
###########################################\n# Design Files\n###########################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_design_files.tcl\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/fabric_design_files.tcl to include BPF Hello FIM Slave IP.
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE $::env(BUILD_ROOT_REL)/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify $OFS_ROOTDIR/src/includes/fabric_width_pkg.sv to add Hello FIM slave information and update EMIF slave next offset.
localparam bpf_hello_fim_slv_baseaddress = 'h16000; // New\nlocalparam bpf_hello_fim_slv_address_width = 12; // New\nlocalparam bpf_emif_slv_next_dfh_offset = 'h1000; // Old value: 'hB000\nlocalparam bpf_hello_fim_slv_next_dfh_offset = 'hA000; // New\nlocalparam bpf_hello_fim_slv_eol = 'b0; // New\n
-
Modify $OFS_ROOTDIR/src/top/top.sv
-
Add bpf_hello_fim_slv_if to AXI interfaces
// AXI4-lite interfaces\n...\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width), .ARADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width)) bpf_hello_fim_slv_if();\n
-
Add Hello FIM instantiation
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (fabric_width_pkg::bpf_hello_fim_slv_address_width),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`else\ndummy_csr #(\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif\n
-
Add interfaces for Hello FIM slv to bpf instantiation
bpf bpf (\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\n);\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt to add the hello_fim module as a slave to the apf.
# NAME FABRIC BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 18 fme,pcie,pmci,qsfp0,qsfp1,emif,hssi,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute helper script to generate BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\n\nsh gen_fabrics.sh\n
-
Once the script completes, the following new IP is created: $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip.
-
[OPTIONAL] You may verify the BPF changes have been made correctly by opening bpf.qsys to analyze the BPF.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\n\nqsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qpf\n
Find the bpf_hello_fim_slv instance:
-
Compile the Hello FIM design
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p fseries-dk work_fseries-dk_hello_fim\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/dfh_walker/test_csr_defs.sv
-
Add HELLO_FIM_IDX entry to t_dfh_idx enumeration.
...\ntypedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX, // New\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nVUART_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n...\n
-
Add HELLO_FIM_DFH to get_dfh_names function.
...\nfunction automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\n...\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\"; // New\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\n...\nreturn dfh_names;\n...\n
-
Add expected DFH value for Hello FIM to the get_dfh_values function.
...\nfunction automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\n...\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_xxxxxx_1012;\ndfh_values[PMCI_DFH_IDX][39:16] = fabric_width_pkg::bpf_pmci_slv_next_dfh_offset;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_xxxxxx_0100; // New\ndfh_values[HELLO_FIM_DFH_IDX][39:16] = fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset; // New\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_xxxxxx_0014;\ndfh_values[ST2MM_DFH_IDX][39:16] = fabric_width_pkg::apf_st2mm_slv_next_dfh_offset;\n...\nreturn dfh_values;\n...\n
-
Generate simulation files
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\n./gen_sim_files.sh fseries-dk\n
-
Run DFH Walker Simulation
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\nsh run_sim.sh TEST=dfh_walker\n
-
Verify that the test passes, and that the output shows the Hello FIM in the DFH sequence
********************************************\n Running TEST(0) : test_dfh_walking\n********************************************\n...\n\nREAD64: address=0x00015000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000010001009\n\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nREAD64: address=0x00016000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x30000000a0000100\n\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000000a0000100)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000200001012\n\nPMCI_DFH\n Address (0x20000)\nDFH value (0x3000000200001012)\n...\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356791250000\nV C S S i m u l a t i o n R e p o r t\nTime: 356791250000 fs\nCPU Time: 61.560 seconds; Data structure size: 47.4Mb\nTue Aug 15 16:29:45 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#414-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
[OPTIONAL] In the work directory where the FIM was compiled, determine the PR Interface ID of your design. You can use this value at the end of the walkthrough to verify that the design has been configured to the FPGA.
cd $OFS_ROOTDIR/<work_directory>/syn/board/fseries-dk/syn_top/\n\ncat fme-ifc-id.txt\n
Example output:
98ed516f-d24d-5b71-ae12-e78cd641e4be\n
-
Switch to your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Run fpgainfo to determine the PCIe B:D.F of your board, and to verify the PR Interface ID matches the ID you found in Step 1.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
-
Initialize opae.io
sudo opae.io init -d <B:D.F>\n
For example:
sudo opae.io init -d b1:00.0\n
-
Run DFH walker. Note the value read back from offset 0x16000 indicates the DFH ID is 0x100 which matches the Hello FIM module.
sudo opae.io walk -d <B:D.F>\n
For example:
sudo opae.io walk -d b1:00.0\n
Example output:
...\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000000a0000100\n dfh: id = 0x100, rev = 0x0, next = 0xa000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000200001012\n dfh: id = 0x12, rev = 0x1, next = 0x20000, eol = 0x0, reserved = 0x0, feature_type = 0x3\n...\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d b1:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d b1:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d b1:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:b1:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration JTAG PCI Development Kit (Driver: dfl-pci)\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#415-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.5 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Perform the following steps in your development environment:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Synthesize the design using the -e build script option. You may skip this step if you are using a pre-synthesized design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -e fseries-dk work_hello_fim_with_stp\n
-
Open the design in Quartus. The Compilation Dashboard should show that the Analysis & Synthesis step has completed.
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/ofs_top.qpf\n
-
Open Tools -> Signal Tap Logic Analyzer
-
Select the Default template and click Create
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note that the clock selected should correspond to the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the block of interest and open the design instance for review.
-
In the Signal Configuration -> Clock box of the Signal Tap Logic Analyzer window, click the \"...\" button
-
In the Node Finder tool that opens, type clock into the Named field, and select hello_fim_top_inst in the Look in field, then click Search. Select the hello_fim_top_inst|clk signal from the Matching Nodes box and click the \">\" button to move it to the Nodes Found box. Click OK to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM.
-
In the Named field, enter csr_lite_if*. In the Look In field, select hello_fim_top_inst.
-
Select the nets that appear in the Matching Nodes box. Click the > button to add them to the Nodes Found box.
-
Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto_signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, stp_for_hello_fim.
-
Save the newly created Signal Tap file, in the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/ directory, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will aurtomatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE stp_for_hello_fim.stp\nset_global_assignment -name SIGNALTAP_FILE stp_for_hello_fim.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
./ofs-common/scripts/common/syn/build_top.sh -p -k fseries-dk work_hello_fim_with_stp\n
-
Ensure that the compile completes successfully and meets timing:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: fseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the fseries-dk. Refer to the Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the fseries-dk via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open the Quartus Signal Tap GUI
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap Logic Analyzer window, select File -> Open, and choose the stp_for_hello_fim.stp file.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable for the fseries-dk. In the Device: selection box select the Agilex device.
-
If the Agilex Device is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
Create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'stp_for_hello_fim' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, walk the DFH list or peek/poke the Hello FIM registers.
opae.io init -d 0000:b1:00.0\nopae.io walk -d 0000:b1:00.0\nopae.io release -d 0000:b1:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":"In the FIM build, the standard AFUs in the port gasket can be replaced with PIM-based AFUs, wrapped by the ofs_plat_afu() top-level module. Before invoking build_top.sh, set the environment variable AFU_WITH_PIM to the path of a sources text file -- the same sources file from examples such as sources.txt in hello_world:
export AFU_WITH_PIM=<path to>/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt\n
When set during the build_top.sh setup stage, the FIM build is configured to load the specified AFU. PR will continue to work, if enabled. The only difference with AFU_WITH_PIM is the change to the AFUs present at power-on. Keep in mind that the default exercisers were chosen to attach to all devices and force reasonable routing decisions during the FIM build across the PR boundary. AFU_WITH_PIM is best used for FIMs that disable PR.
Configuration of the user clock from an AFU's JSON file is ignored in a FIM build.
The AFU_WITH_PIM setting matters only during the setup stage of build_top.sh. It is not consumed by later stages.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#422-compiling-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2 Compiling the FIM in preparation for designing your AFU","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" blocks. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination, order or all can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4221-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p fseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_fseries-dk\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to adjust the PR region to fit the additional logic into the static region. Similarly, if you reduce logic in the FIM region, then you can adjust the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions as reported in the following file:
$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/fseries-dk/syn_top/output_files/ofs_top.fit.rpt\n
An example report of the resources usage by partitions defined for the FIM is shown as follows:
+------------------------------------------------------------------------------------------+\n; Logic Lock Region Constraints ;\n+--------------------------------------+-------------------------+-------------------------+\n; Name ; Place Region Constraint ; Route Region Constraint ;\n+--------------------------------------+-------------------------+-------------------------+\n; afu_top|port_gasket|pr_slot|afu_main ; (90, 40) to (350, 220) ; (0, 0) to (385, 329) ;\n+--------------------------------------+-------------------------+-------------------------+\n+----------------------------------------------------------------------------------------------+\n; Logic Lock Region Usage Summary ;\n+-------------------------------------------------------+--------------------------------------+\n; Statistic ; afu_top|port_gasket|pr_slot|afu_main ;\n+-------------------------------------------------------+--------------------------------------+\n; ALMs needed [=A-B+C] ; 48011.2 / 351140 ( 13 % ) ;\n; [A] ALMs used in final placement ; 53324.4 / 351140 ( 15 % ) ;\n; [B] Estimate of ALMs recoverable by dense packing ; 5452.3 / 351140 ( 1 % ) ;\n; [C] Estimate of ALMs unavailable ; 139.0 / 351140 ( < 1 % ) ;\n; ALMs used for memory ; 450.0 ;\n; Combinational ALUTs ; 67166 ;\n; Dedicated Logic Registers ; 87533 / 1404560 ( 6 % ) ;\n; I/O Registers ; 0 ;\n; Block Memory Bits ; 1737568 ;\n; M20Ks ; 137 / 5049 ( 2 % ) ;\n; DSP Blocks needed [=A-B] ; 0 / 3439 ( 0 % ) ;\n; [A] DSP Blocks used in final placement ; 0 / 3439 ( 0 % ) ;\n; [B] Estimate of DSPs recoverable by dense merging ; 0 / 3439 ( 0 % ) ;\n; Pins ; 0 ;\n; IOPLLs ; 0 ;\n; ; ;\n; Region Placement ; (90, 40) to (350, 220) ;\n+-------------------------------------------------------+--------------------------------------+\n
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to first analyze the PR logic lock regions in a default FIM design, then resize the PR region:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
The OFS flow provides the TCL file $OFS_ROOTDIR/syn/board/fseries-dk/setup/pr_assignments.tcl which defines the PR partition where the AFU is allocated. Each region is a rectangle defined by the origin coordinate (X0, Y0) and the top right corner coordinate (X1, Y1).
#####################################################\n# Main PR Partition -- green_region\n#####################################################\nset_instance_assignment -name PARTITION green_region -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name CORE_ONLY_PLACE_REGION ON -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name RESERVE_PLACE_REGION ON -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name PARTIAL_RECONFIGURATION_PARTITION ON -to afu_top|port_gasket|pr_slot|afu_main\n\nset_instance_assignment -name PLACE_REGION \"X90 Y40 X350 Y220\" -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name ROUTE_REGION \"X0 Y0 X385 Y329\" -to afu_top|port_gasket|pr_slot|afu_main\n
-
[OPTIONAL] Use Quartus Chip Planner to visualize the default PR region allocation.
-
Compile the design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk\n
-
Open the project in the Quartus GUI
quartus $OFS_ROOTDIR/work_fseries-dk/syn/board/fseries-dk/syn_top/ofs_top.qpf\n
-
Switch to the ofs_top view if necessary.
-
Click Tools -> Chip Planner to open the Chip Planner.
-
Analyze the regions shown. Note that the regions are made of rectangles described by an origin coordinate, region height, and region width. If you are modifying the regions, you will need to identify the coordinates of your desired region.
-
Close the Quartus GUI.
-
Make changes to the partial reconfiguraton region in the $OFS_ROOTDIR/syn/board/fseries-dk/setup/pr_assignments.tcl file. You can modify the size and location of existing lock regions or add new ones and assign them to the AFU PR partition.
-
Recompile your FIM and create the PR relocatable build tree using the following commands.
cd $OFS_ROOTDIR ofs-common/scripts/common/syn/build_top.sh -p fseries-dk work_fseries-dk_resize_pr\n
-
Analyze the resource utilization report $OFS_ROOTDIR/work_fseries-dk/syn/board/fseries-dk/syn_top/output_files/ofs_top.fit.rpt produced after recompiling the FIM.
-
Perform further modification to the PR regions until the results are satisfactory. Make sure timing constraints are met.
For more information on how to optimize the floor plan of your Partial Reconfiguration design refer to the following online documentation.
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3 - Floorplan the Design
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe sub-system IP and PF/VF MUX can be modified either using the OFSS flow or the IP Presets flow. The OFSS flow supports a subset of all available PCIe Sub-system settings, while the IP Preset flow can make any available PCIe Sub-system settings change. With PCIe-SS modifcations related to the PFs and VFs, the PF/VF MUX logic is automatically configured based on the PCIe-SS configuration.
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section.
The sections below describe each modification.
- PCIe Configuration Using OFSS
- PCIe Configuration Using IP Presets
- [PCIe Sub-System Target IP]
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS PCIe Attach FIM for the n6001 can support up to 8 PFs and 2000 VFs distributed accross all PFs.
For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 (if at least 1 VF present) | 2 (if no VFs present) Max # of PFs 8 Min # of VFs 1 on PF0 (if only 1 PF present) | 0 (if 2PFs present) Max # of VFs 2000 distributed across all PFs Note that as the number of VFs goes up, timing closure can become more difficult.
The scripts provided in ${OFS_ROOTDIR}/ofs-common/tools/ofss_config allows you to easily reconfigure the number of PFs and VFs, bar addresses, vendor/device ID values and more. The PCIe Subsystem IP parameters that can be modified can be seen by reviewing ${OFS_ROOTDIR}/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py
New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe-SS configuration registers contain the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00000001 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00000001"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4431-walkthrough-modify-pcie-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify PCIe Configuration Using OFSS","text":"Perform the following steps to use OFSS files to configure the PCIe Sub-system and PF/VF MUX. In this example, we will add a PF with 2 VFs to the default configuration.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment to test the design. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
View the default OFS PCIe Attach FIM for the fseries-dk PF/VF configuration in the the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n
-
Create a new PCIe OFSS file from the existing pcie_host.ofss file
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_pfvf_mod.ofss\n
-
Configure the new pcie_pfvf_mod.ofss with your desired PCIe settings. In this example we will add PF5 with 2 VFs.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n[pf5]\nnum_vfs = 2\n
-
Compile the FIM with the new PCIe OFSS file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_pfvf_mod.ofss fseries-dk work_fseries-dk_pfvf_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_pfvf_mod/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Verify the number of VFs on the newly added PF5. In this example, we defined 2 VFs on PF5 in Step 5.
sudo lspci -vvv -s b1:00.5 | grep VF\n
Example output:
Initial VFs: 2, Total VFs: 2, Number of VFs: 0, Function Dependency Link: 05\nVF offset: 4, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
-
Verify communication with the newly added PF5. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
-
Initialize the driver on PF5
sudo opae.io init -d b1:00.5\n
-
Read the GUID for the PF5 CSR stub.
sudo opae.io -d b1:00.5 -r 0 peek 0x8\nsudo opae.io -d b1:00.5 -r 0 peek 0x10\n
Example output:
0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#444-pcie-configuration-using-ip-presets","title":"4.4.4 PCIe Configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4441-walkthrough-modify-pcie-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Configuration Using IP Presets","text":"Perform the following steps to use an IP Preset file to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID of PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment to test the design. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script using your desired OFSS configration to create a working directory for the fseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup fseries-dk work_fseries-dk\n
-
Open the PCIe-SS in the work directory using Quartus Parameter Editor.
qsys-edit $OFS_ROOTDIR/work_fseries-dk/ipss/pcie/qip/pcie_ss.ip\n
-
In the IP Parameter Editor window, scroll down and select the PCIe Interfaces Port Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab. Modify the settings as desired. In this case, we are changing the Revision ID to 0x00000002. You may make any desired modifications.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, fseries-dk-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 6.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = fseries-dk-rev2\n
-
Compile the design with the modified new pcie_host_mod_preset.ofss file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_mod_preset.ofss fseries-dk work_fseries-dk_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_pcie_mod/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Determing the PCIe B:D.F of your board. You may use the OPAE command fpgainfo fme to determine this.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
-
Use lspci to verify that the PCIe changes have been implemented. In this example, the Rev for PF0 is 02.
lspci -nvmms b1:00.0\n
Example output:
Slot: b1:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 0001\nPhySlot: 1-1\nRev: 01\nNUMANode: 1\nIOMMUGroup: 190\n
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the Intel FPGA IP Subsystem for PCI Express which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. You must also reduce the IOPLL frequency to a maximum of 470 MHz.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#445-pcie-sub-system-target-ip","title":"**4.4.5 PCIe Sub-System Target IP **","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4451-walkthrough-change-the-pcie-sub-system-target-ip","title":"4.4.5.1 Walkthrough: Change the PCIe Sub-System Target IP","text":"Perform the following steps to change the target PCIe IP from AXI Streaming Intel FPGA IP for PCI Express to Intel FPGA IP Subsystem for PCI Express.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Add the following line to the [settings] section of the PCIe OFSS file you are using. In this example, it will be added to the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file, which is the default PCIe OFSS file used for fseries-dk.
ip_component = pcie_ss\n
Note: To change back to using the AXI Streaming Intel FPGA IP for PCI Express, simply remove this line.
-
Build the FIM using the 470 MHz IOPLL and the PCIe OFSS file that was modified in step 3.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/iopll/iopll_470MHz.ofss,tools/ofss_config/pcie/pcie_host.ofss fseries-dk work_fseries-dk\n
-
When the build completes, the output files can be found in $OFS_ROOTDIR/work_fseries-dk/syn/board/fseries-dk/output_files.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#45-minimal-fim","title":"4.5 Minimal FIM","text":"In a minimal FIM, the exercisers and Ethernet subsystem are removed from the design, and the PF/VF configuration is reduced to either 2PF/0VF or 1PF/1VF.
There are two pre-provided PCIe configurations that can be used to create minimal FIM:
- 2PF: this PCIe configuration has two physical functions, with the APF/BPF on PF0, and the AFU PR region on PF1.
-
1PF/1VF: This PCIe configuration has a single physical function, with the APF/BPF on PF0, and the AFU PR region on PF0-VF0.
The FIM source repository contains OFSS file that can be used to configure the PCIe IP for the minimal FIM.
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2pf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#451-walkthrough-create-a-minimal-fim","title":"4.5.1 Walkthrough: Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment for hardware testing. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with Null HEs compile option, the No HSSI compile option, and the desired PCIe PF/VF configuration.
cd $OFS_ROOTDIR\n
-
2PF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2pf.ofss fseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_fseries-dk_minimal_fim\n
-
1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss fseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_fseries-dk_minimal_fim\n
-
Review the $OFS_ROOTDIR/work_fseries-dk_minimal_fim/syn/board/fseries-dk/syn_top/output_files/ofs_top.fit.rpt utilization report to see the utilization statistics for the Minimal fim. Refer to Appendix A: Resource Utilization Tables Table A-4 for the expected utilization for this Minimal FIM.
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_minimal_fim/syn/board/fseries-dk/syn_top/output_files/ofs_top_hps.sof image to your deployment environmenment.
-
Switch to your deployment environment, if different than your development environment.
-
Program the ofs_top_hps.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Use lspci to verify that the PCIe changes have been implemented.
sudo lspci -vvv -s b1:00.0 | grep VF\n
Example output for a 1PF/1VF image:
Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 1, stride: 1, Device ID: bcce\nVF Migration: offset: 00000000, BIR: 0\n
-
You may wish to adjust the PR logic lock regions to maximize the resources available for the AFU. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#46-migrating-to-a-different-agilex-device-number","title":"4.6 Migrating to a Different Agilex Device Number","text":"The following instructions enable a user to change the device part number of the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). Please be aware that this release works with Agilex\u00ae 7 FPGA devices that have F-tile for PCIe and Ethernet. Other tiles will take further work.
You may wish to change the device part number for the following reasons
- Migrate to same device package but with a different density
- Migrate to a different package and with a different or same density
The default device for the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is AGFB027R24C2E2VR2
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"Perform the following steps to migrate to a different Agilex Device. In this example, we will migrate from the default AGFB027R24C2E2VR2 device to AGFB027R31C2E2V. The package will change from R24C to R31C.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/fseries-dk_base.ofss file to use AGFB027R31C2E2V.
[ip]\ntype = ofs\n\n[settings]\nplatform = n6001\nfamily = agilex\nfim = base_x16\npart = AGFB027R31C2E2V device_id = 6001\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf file to use AGFB027R31C2E2V.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY \"Agilex 7\"\nset_global_assignment -name DEVICE AGFB027R31C2E2V
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_pr_afu.qsf file to use AGFB027R31C2E2V.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex 7\nset_global_assignment -name DEVICE AGFB027R31C2E2V
-
If the device you are migrating to uses the same package and pinout, you do not need to modify the pinout constraints. In this example, because we are migrating from package R24C to R31C, we need to modify the pinout to match the new device. If you would like Quartus to attempt to pin out the design automatically, you may remove all pin assignments. Typically, you will still need to set certain pins manually in order to guide Quartus for a successful compile (e.g. transceiver reference clocks).
-
Commment out all pin assignments in the following files: * $OFS_ROOTDIR/syn/board/fseries-dk/setup/emif_loc.tcl * $OFS_ROOTDIR/syn/board/fseries-dk/setup/hps_loc.tcl * $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl
-
Identify the pins in the design that will be constrained. In this example we will manually constrain the QSFP reference clock and the PCIe reference clock to help guide the fitter. The Device Migration Pinout Table shows th pin assignments that will be set, along with the pin number for both the old R24C package and the new R31C package.
Net Name Pin Name AGF 027 R24C Pin # AGF 027 R31C Pin # \"qsfp_ref_clk(n)\" REFCLK_FGTL12C_Q1_RX_CH2n AV48 BD57 qsfp_ref_clk REFCLK_FGTL12C_Q1_RX_CH2p AW49 BB57 \"PCIE_REFCLK0(n)\" REFCLK_FGTR13A_Q1_RX_CH2n BU7 BP13 PCIE_REFCLK0 REFCLK_FGTR13A_Q1_RX_CH2p BR7 BN14 -
Re-pin the pins identified in the previous step in the $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl file for the new pinout for the AGF 027 R31C package.
set_location_assignment PIN_BB57 -to qsfp_ref_clk\nset_location_assignment PIN_BD57 -to \"qsfp_ref_clk(n)\"\nset_location_assignment PIN_BP13 -to \"PCIE_REFCLK0(n)\"\nset_location_assignment PIN_BN14 -to PCIE_REFCLK0\n
-
Uncomment the instance assignments related to he QSFP reference clock in the $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl file.
set_instance_assignment -name IO_STANDARD \"CURRENT MODE LOGIC (CML)\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=156250000\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=TRUE\" -to qsfp_ref_clk\n
-
Compile a flat design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh fseries-dk:flat work_fseries-dk\n
-
Verify that the build completes successfuly. If the compilation fails with errors relating to the pinout, review the error messages and modify the pinout accordingly. If there are timing violations, try building with a different seed. Refer to the Change the Compilation Seed section for instructions on changing the build seed.
-
When you are satisfied with the pinout, preserve it by hard-coding the desired pinout back to the followig files:
$OFS_ROOTDIR/syn/board/fseries-dk/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/fseries-dk/setup/hps_loc.tcl$OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl
-
When you are ready to re-incorporate PR into the design, modify the PR region to be compatible with the new device. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#471-walkthrough-add-or-remove-the-memory-sub-system","title":"4.7.1 Walkthrough: Add or remove the Memory Sub-System","text":"The Memory Sub-System can be added or removed to the FIM design using the INCLUDE_DDR4 macro defined in the ofs_top.qsf file. The Memory-SS is enabled by default in the fseries-dk.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the ofs_top.qsf file to either enable or disable the INCLUDE_DDR4 macro. Comment out this macro assignemnt if you wish to remove the Memory-SS.
Note: The default Memory-SS has connections to the HPS. When enabling the Memory-SS, you must either ensure that the INCLUDE_HPS and INCLUDE_UART macros are also enabled, or you must remove the connections from the Memory-SS to the HPS. Refer to the Remove the HPS section for step-by-step instructions on removing the HPS from the design.
-
Compile the design. Refer to the Compile OFS FIM section for step-by-step instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System.
Note: The default HSSI-SS configuration for the fseries-dk is 2x4x25GbE.
Note: The fseries-dk does not support 2x200GbE or 1x400GbE configurations.
Note: Due to Ethernet differential pair routing on the ES version of the Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit, some differential pairs were swapped to improve signal routing. To account for the pair swap, there is a requirement to run a script to invert the differential traces. Refer to the \"HE-HSSI\" Section of the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on swapping the polarity of Ethernet differential pairs at run-time.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#481-walkthrough-modify-the-ethernet-sub-system-to-2x4x10gbe-using-ip-presets","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System to 2x4x10GbE Using IP Presets","text":"Perform the following steps to modify the Ethernet Sub-System to 2x4x10GbE. In this walkthrough, we will create a new IP presets file that will be used during the build flow. Note that this modification can be done with OFSS, but the presets flow for demonstration purposes.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script to create a work directry for the fseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss fseries-dk work_fseries-dk\n
-
Open the HSSI-SS IP in the work directory using qsys-edit
qsys-edit $OFS_ROOTDIR/work_fseries-dk/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Change the PORT_PROFILE parameter from 25GbE to 10GbE for Ports 0 through 7.
-
Create an IP Presets file for this Ethernet Subsystem IP configuration.
-
Click New in the bottom right corner of the Presets window.
-
In the New Preset window, set the Preset name. In this example, we will name it 2x4x10g-fseries-dk.
-
Click the ... button to select the save location of the Preset file.
-
In the Save As window, navigate to the $OFS_ROODIR/ipss/hssi/qip/hssi_ss/presets directory. Set the File Name to 2x4x10g-fseries-dk.qprs. Then click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close out of the IP Parameter Editor GUI. Do not save or generate the IP.
-
Create a new HSSI OFSS file named $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_2x4x10_ftile.ofss with the following contents. This will call the IP Presets file generated in Step 6.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\n num_channels = 8\ndata_rate = 10GbE\n preset = 2x4x10g-fseries-dk\n
-
Run the FIM build script with the new HSSI OFSS file created in Step 8.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_2x4x10_ftile.ofss fseries-dk work_fseries-dk_eth_2x4x10\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#482-walkthrough-modify-the-ethernet-sub-system-using-ofss","title":"4.8.2 Walkthrough: Modify the Ethernet Sub-System Using OFSS","text":"Perform the following steps to modify the Ethernet Sub-System to 3x4x25GbE. In this walkthrough, we will create HSSI OFSS file that will be used during the build flow.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a new HSSI OFSS file for the 3x4x25GbE configuration.
-
Create the hssi_3x4x25_ftile.ofss file
vim $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_3x4x25_ftile.ofss\n
-
Copy the following into the new HSSI OFSS file. Note that num_channels is set to 12:
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 12\ndata_rate = 25GbE\n\neth_f_rsfec = IEEE_802.3_RS_528_514\nclient_interface = MAC Avalon ST\n
-
Edit the pinout constraints to support the new channels.
-
Open the top_loc.tcl file
vim $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl\n
-
Add hssi_if[11:8] connections, and change the qsfp_ref_clk pin to the global ref clock which can be accessed by all transceivers in the F-Tile.
set_location_assignment PIN_AW55 -to hssi_if[8].rx_p\nset_location_assignment PIN_BC55 -to hssi_if[9].rx_p\nset_location_assignment PIN_BG55 -to hssi_if[10].rx_p\nset_location_assignment PIN_BL55 -to hssi_if[11].rx_p\n\nset_location_assignment PIN_AY54 -to hssi_if[8].rx_n\nset_location_assignment PIN_BD54 -to hssi_if[9].rx_n\nset_location_assignment PIN_BH54 -to hssi_if[10].rx_n\nset_location_assignment PIN_BM54 -to hssi_if[11].rx_n\n\nset_location_assignment PIN_BB52 -to hssi_if[8].tx_p\nset_location_assignment PIN_BF52 -to hssi_if[9].tx_p\nset_location_assignment PIN_BK52 -to hssi_if[10].tx_p\nset_location_assignment PIN_BL49 -to hssi_if[11].tx_p\n\nset_location_assignment PIN_BA51 -to hssi_if[8].tx_n\nset_location_assignment PIN_BE51 -to hssi_if[9].tx_n\nset_location_assignment PIN_BJ51 -to hssi_if[10].tx_n\nset_location_assignment PIN_BM48 -to hssi_if[11].tx_n\n\nset_location_assignment PIN_AW49 -to qsfp_ref_clk\nset_location_assignment PIN_AV48 -to \"qsfp_ref_clk(n)\"\n
-
Change the number of QSFP ports from 2 to 3 in the $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv file.
localparam NUM_QSFP_PORTS_USED = 3; // Number of QSFP cages on board used by target hssi configuration\n
-
Edit $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/hssi_wrapper.sv so that the QSFP LED signals use NUM_QSFP_PORTS_USED defined in the previous step.
// Speed and activity LEDS\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_green, // Link up in Nx25G or 2x56G or 1x100G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_yellow, // Link up in Nx10G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_green, // Link up and activity seen\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_red // LOS, TX Fault etc\n
-
Build the flat FIM. It is recommended to build the flat FIM when making changes such as this to reduce complexity until the changes are finalized.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_3x4x25_ftile.ofss fseries-dk:flat work_fseries-dk_eth_12x25g\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#49-modifying-the-hps","title":"4.9 Modifying the HPS","text":"This section describes ways to modify the HPS.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#491-walkthrough-remove-the-hps","title":"4.9.1 Walkthrough: Remove the HPS","text":"Perform the following steps to remove the HPS from the FIM design.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script to create a work directry for the fseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup fseries-dk work_fseries-dk\n
-
Create a Memory Sub-system IP presets file with the connection to the HPS removed.
-
In the work directory created in Step 3, open the mem_ss.ip
qsys-edit $OFS_ROOTDIR/work_fseries-dk/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
In the IP Parameter Editor window that opens, remove the entries corresponding to the HPS (row #2) in the Memory Interfaces and Application Interfaces sections. To do this, click the row to be removed, then click the minus (-) button.
-
In the Presets pane, click New... to create a new IP presets file.
-
In the New Preset window, create a unique preset name. For example, fseries-dk-mem-no-hps.
-
Click the ... button to select the save location of the IP presets file. In the Save As window, set the Look In field to the memory IP presets directory $OFS_ROOTDIR/ipss/mem/qip/presets. Set the File Name field to match the name selected in Step 4. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Edit the Memory OFSS file $OFS_ROOTDIR/tools/ofss_config/memory/memory_ftile.ofss to use the IP presets file generated in Step 4.
[ip]\ntype = memory\n\n[settings]\noutput_name = mem_ss\npreset = fseries-dk-mem-no-hps\n
-
Edit $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf to comment out the INCLUDE_HPS macro definition.
#set_global_assignment -name VERILOG_MACRO \"INCLUDE_HPS\"\n
-
Build the FIM.
./ofs-common/scripts/common/syn/build_top.sh -p fseries-dk work_fseries-dk_no_hps\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the fseries-dk is done by programming a SOF image to the FPGA via JTAG using Quartus Programer.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"The Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) has an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG.
Perform the following steps to establish a JTAG connection with the fseries-dk.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
Steps:
-
Refer to the following figure for Steps 2 and 3.
-
Locate Single DIP Switch SW2 and 4-position DIP switch SW3 on the fseries-dk. These switches control the JTAG setup for the board. Ensure that both SW2 and SW3.3 are set to ON.
-
Locate the J10 Micro-USB port on the fseries-dk. Connect a Micro-USB to USB-A cable between the J10 port and the workstation that has Quartus Prime Pro tools installed.
-
Use the jtagconfig tool to check that the JTAG chain contains the AGFB027R24C2E2VR2 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
1) Agilex F-Series FPGA Dev Kit [1-6]\n0343B0DD AGFB027R24C(.|R2|R0)/..\n020D10DD VTAP10\n
This concludes the walkthrough for establishing a JTAG connection on the fseries-dk.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the fseries-dk. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is connected via JTAG. The Quartus programmer version must be the same as the version of Quartus used to build the design.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the fseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
Note: the Quartus programmer version must be the same as the version of Quartus used to build the design.
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the fseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB027R24C2E2VR2 device. The JTAG chain should show the device.
-
Right click the AGFB027R24C2E2VR2 row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGFB027R24C2E2VR2 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#appendix","title":"Appendix","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#appendix-a-resource-utilization-tables","title":"Appendix A: Resource Utilization Tables","text":"Table A-1 Default FIM Resource Utilization with PR
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 189241.5 20.73 585 4.41 PCIE_RST_CTRL.rst_ctrl 463.5 0.05 0 0.0 afu_top 129761.2 14.22 320 2.41 auto_fab_0 1295.2 0.14 8 0.06 bpf 661.2 0.07 0 0.0 fme_top 638.1 0.07 6 0.05 hps_ss 0.0 0.0 0 0.0 hssi_wrapper 15902.6 1.74 80 0.6 local_mem_wrapper 5517.6 0.6 30 0.23 ofs_top_auto_tiles 14591.8 1.6 20 0.15 pcie_wrapper 18469.5 2.02 113 0.85 pmci_dummy_csr 684.8 0.08 0 0.0 qsfp_0 626.1 0.07 4 0.03 qsfp_1 626.0 0.07 4 0.03 sys_pll 0.5 0.0 0 0.0 Table A-2 Minimal FIM Resource Utilization
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 82271.4 9.01 345 2.6 PCIE_RST_CTRL.rst_ctrl 351.4 0.04 0 0.0 afu_top 45522.5 4.99 190 1.43 auto_fab_0 1281.0 0.14 8 0.06 bpf 775.6 0.08 0 0.0 fme_top 727.8 0.08 6 0.05 hps_ss 0.0 0.0 0 0.0 hssi_dummy_csr 689.2 0.08 0 0.0 local_mem_wrapper 5512.2 0.6 30 0.23 ofs_top_auto_tiles 6994.1 0.77 10 0.08 pcie_wrapper 18365.5 2.01 101 0.76 pmci_dummy_csr 682.7 0.07 0 0.0 qsfp0_dummy_csr 684.0 0.07 0 0.0 qsfp1_dummy_csr 684.3 0.07 0 0.0 sys_pll 0.5 0.0 0 0.0"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/ftile_devkit/doc_modules/ftile_wt_program_fpga_via_jtag/","title":"Ftile wt program fpga via jtag","text":"This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)))) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the fseries-dk. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
sudo fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010202A8769764\nBitstream Version : 5.0.1\nPr Interface Id : b541eb7c-3c7e-5678-a660-a54f71594b34\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the fseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the fseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB027R24C2E2VR2 device. The JTAG chain should show the device.
-
Right click the AGFB027R24C2E2VR2 row and selct Change File.
-
In the Select New Programming File window that opens, select ofs_top_hps.sof and click Open.
-
Check the Program/Configure box for the AGFB027R24C2E2VR2 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
"},{"location":"hw/ftile_devkit/doc_modules/ftile_wt_set_up_jtag/","title":"Ftile wt set up jtag","text":"The Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) has an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG.
Perform the following steps to establish a JTAG connection with the fseries-dk.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))] for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
Steps:
-
Refer to the following figure for Steps 2 and 3.
-
Locate Single DIP Switch SW2 and 4-position DIP switch SW3 on the fseries-dk. These switches control the JTAG setup for the board. Ensure that both SW2 and SW3.3 are set to ON.
-
Locate the J10 Micro-USB port on the fseries-dk. Connect a Micro-USB to USB-A cable between the J10 port and the workstation that has Quartus Prime Pro tools installed.
-
Use the jtagconfig tool to check that the JTAG chain contains the AGFB027R24C2E2VR2 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
1) Agilex F-Series FPGA Dev Kit [1-6]\n0343B0DD AGFB027R24C(.|R2|R0)/..\n020D10DD VTAP10\n
This concludes the walkthrough for establishing a JTAG connection on the fseries-dk.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/","title":"Getting Started Guide: Open FPGA Stack for Intel Agilex 7 FPGAs Targeting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)","text":"Last updated: November 19, 2024
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in evaluating the 2024.2-1 version of the PCIe Attach release targeting the F-Series Development Kit. This document will not cover Board Installation Guidelines or OFS Software Installation. Instead it will recommend you use a software installer to allow for fast evaluation. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Load and verify firmware targeting the FIM and AFU regions of the AGFB027R24C2E2VR2 FPGA
- Verify full stack functionality offered by the PCIe Attach OFS solution
- Learn where to find additional information on other PCIe Attach ingredients
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell targeting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). This platform is a Development Kit intended to be used as a starting point for evaluation and development of the Intel Agilex 7 FPGA F-Series with two F-Tiles.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#13-references-and-versions","title":"1.3 References and Versions","text":""},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-2-software-and-component-version-summary-for-ofs-pcie-attach-targeting-the-f-series-development-kit","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach targeting the F-Series Development Kit","text":"The OFS 2024.2-1 PCIe Attach release targeting the F-Series Development Kit is built upon tightly coupled software and Operating System version(s). The repositories listed below are used to manually build the Shell and the AFU portion of any potential workloads. Use this section as a general reference for the versions which compose this release. Specific instructions on building the FIM or AFU are discussed in their respective documents, but are shown here for the sake of completion.
Component Version Download Link Quartus Quartus Prime Pro Version 24.1 https://www.intel.com/content/www/us/en/software-kit/794624/intel-quartus-prime-pro-edition-design-software-version-24-1-for-linux.html, patches: 0.18, 0.26 Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.10/x86_64/product-software OneAPI-ASP ofs-2024.2-2 https://github.com/OFS/oneapi-asp/releases/tag/ofs-2024.2-1, patches: 0.02 OFS Platform AFU BBB ofs-2024.2-1 https://github.com/OFS/ofs-platform-afu-bbb/releases/tag/ofs-2024.2-1 OFS FIM Common Resources 2024.2-1 https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.2-1 AFU Examples tag: ofs-2024.2-1 https://github.com/OFS/examples-afu/releases/tag/ofs-2024.2-1 OPAE-SIM tag: 2.13.0-2 https://github.com/OPAE/opae-sim OFS SW Installer OFS 2024.2-1 Release Page"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-3-hardware-bkc-for-ofs-pcie-attach-targeting-the-f-series-development-kit","title":"Table 3: Hardware BKC for OFS PCIe Attach targeting the F-Series Development Kit","text":"The following table highlights the hardware which composes the Best Known Configuration (BKC) for the OFS 2024.2-1 PCIe Attach release targeting F-Series Development Kit.
Note: The Dell R750 server product line is known not to work with this release.
Component Link Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/agilex/agf027-and-agf023.html Intel FPGA Download Cable II https://www.intel.com/content/www/us/en/products/sku/215664/intel-fpga-download-cable-ii/specifications.html SuperMicro SYS-220HE-FTNR https://www.supermicro.com/en/products/system/hyper/2u/sys-220he-ftnr"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#14-reference-documents","title":"1.4 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.1-1/.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
OFS is a hardware and software infrastructure that provides an efficient approach to developing a customer FPGA-based platform or workload using an Intel, 3rd party, or custom board.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM), or shell of the FPGA provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The OFS architecture for Intel Agilex 7 FPGA provides modularity, configurability, and scalability. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem - a hierarchical design that targets the P-tile PCIe hard IP and is configured to support Gen4 speeds and Arm AXI4-Stream Data Mover functional mode.
- Ethernet Subsystem - provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
- Memory Subsystem - composed of 5 DDR4 channels; two HPS DDR4 banks, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each, and four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB
- Hard Processor System - 64-bit quad core ARM\u00ae Cortex*-A53 MPCore with integrated peripherals.
- Reset Controller
- FPGA Management Engine - Provides a way to manage the platform and enable acceleration functions on the platform.
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration. Each feature of the FME exposes itself to the kernel-level OFS drivers on the host through a Device Feature Header (DFH) register that is placed at the beginning of Control Status Register (CSR) space. Only one PCIe link can access the FME register space in a multi-host channel design architecture at a time.
Note: For more information on the FIM and its external connections, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required to support partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Like the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your entire AFU resides in a partial reconfiguration region of the FPGA.
- The AFU is part of the static region and is compiled as a flat design.
- Your AFU contains both static and PR regions.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components: ST2MM module, Virtio LB stub, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), Memory Host Exerciser (HE-MEM), Traffic Generator to memory (HE-MEM-TG), Port Gasket (PRG) and HPS Copy Engine.
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Basic HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
The AFU has the option to consume native packets from the host or interface channels or to instantiate a shim provided by the Platform Interface Manager (PIM) to translate between protocols.
Note: For more information on the Platform Interface Manager and AFU development and testing, refer to the Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#30-board-installation-and-server-requirements","title":"3.0 Board Installation and Server Requirements","text":"Instructions detailing the board installation guidelines for an F-Tile Dev Kit including server BIOS settings and regulatory information can be found in the Board Installation Guidelines: Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). This document also covers the installation of a JTAG cable, which is required for shell programming.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#40-f-series-development-kit-jtag-driver-setup","title":"4.0 F-Series Development Kit JTAG Driver Setup","text":"A specific JTAG driver needs to be installed on the host OS. Follow the instructions under the driver setup for Red Hat 5+ on Intel\u00ae FPGA Download Cable (formerly USB-Blaster) Driver for Linux*.
View the JTAG Chain after installing the proper driver and Quartus Programmer.
cd ~/intelFPGA_pro/quartus/bin\n./jtagconfig -D\n1) AGI FPGA Development Kit [1-13]\n(JTAG Server Version 23.3.0 Build 104 09/20/2023 SC Pro Edition)\n034BB0DD AGIB027R29A(.|R2|R3)/.. (IR=10)\n020D10DD VTAP10 (IR=10)\nDesign hash 27AA3E0B7CE0A5B9F366\n + Node 08586E00 (110:11) #0\n+ Node 0C006E00 JTAG UART #0\n+ Node 19104600 Nios II #0\n+ Node 30006E02 Signal Tap #2\n+ Node 30006E01 Signal Tap #1\n+ Node 30006E00 Signal Tap #0\nCaptured DR after reset = (0069761BB020D10DD) [65]\nCaptured IR after reset = (000D55) [21]\nCaptured Bypass after reset = (2) [3]\nCaptured Bypass chain = (0) [3]\nJTAG clock speed auto-adjustment is enabled. To disable, set JtagClockAutoAdjust parameter to 0\nJTAG clock speed 24 MHz\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#41-upgrading-the-f-series-development-kit-fim-via-jtag","title":"4.1 Upgrading the F-Series Development Kit FIM via JTAG","text":"Intel provides a pre-built FIM that can be used out-of-box for platform bring-up. This shell design is available on the OFS 2024.2-1 Release Page. After programming the shell and installing both the OPAE SDK and Backport Linux DFL kernel drivers as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach, you can confirm the correct FIM has been configured by checking the output of fpgainfo fme against the following table:
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-4-fim-version","title":"Table 4: FIM Version","text":"Identifier Value Pr Interface ID 98ed516f-d24d-5b71-ae12-e78cd641e4be Bitstream ID 360571656856467345 You will need to download and unpack the artifact images for this release before upgrading your device. You also need to set up the F-Series Development Kit as outlined in the Board Installation Guidelines: Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). The file ofs_top_hps.sof is the base OFS FIM file. This file is loaded into the FPGA using the development kit built in USB Blaster. Please be aware this FPGA is not loaded into non-volatile storage. If the server is power cycled you will need to reload the FPGA .sof file.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/fseries-dk-images_ofs-2024-2-1.tar.gz\ntar xf fseries-dk-images.tar.gz\ncd fseries-dk-images/\n
This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)))) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the fseries-dk. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
sudo fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010202A8769764\nBitstream Version : 5.0.1\nPr Interface Id : b541eb7c-3c7e-5678-a660-a54f71594b34\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the fseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the fseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB027R24C2E2VR2 device. The JTAG chain should show the device.
-
Right click the AGFB027R24C2E2VR2 row and selct Change File.
-
In the Select New Programming File window that opens, select ofs_top_hps.sof and click Open.
-
Check the Program/Configure box for the AGFB027R24C2E2VR2 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#50-ofs-software-installation-from-script","title":"5.0 OFS Software Installation from Script","text":"An overview of the OFS software stack responsibilities and components can be found in the Software Installation Guide: Open FPGA Stack for PCIe Attach. In this document, we will instead use the provided PCIe Attach software installation script to quickly bring up the platform.
Before running the software installer, it is recommended you perform the following steps to lock your OS version to 8.10 and enable related repositories:
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
Download the OFS PCIe Attach installation script from the OFS 2024.2-1 Release Page. Unpack the files:
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/pcieattach_sw_installer_2024.1.zip\n\nunzip pcieattach_sw_installer_2024.1.zip\n
Executing the script without any input arguments will cause it to set up your local environment and will pull down and install pre-built OFS software artifacts. You can explore its other functionality as looking at the help output ofs_sw_install.py -h or though the included README. Execute the script.
./ofs_sw_install.py\n
You can check that all OPAE packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\n
To verify the Backport DFL kernel and driver stack have been installed, perform a warm reboot and check with uname.
Note: Your kernel version and tags may differ from the below, depending on script arguments
uname -r\n4.18.0-553.5.1.el8_10.x86_64\n
You can also check the contents of /usr/lib/modules/<kernel name>/kernel/drivers, or run modinfo <driver name>. Below is a list of the drivers which are loaded as a part of the F Tile Dev Kit Shell design:
lsmod | grep dfl\n\nqsfp_mem_dfl 16384 0\nqsfp_mem_core 20480 1 qsfp_mem_dfl\nuio_dfl 20480 0\nuio 28672 1 uio_dfl\ndfl_fme_mgr 20480 1\ndfl_emif 16384 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_afu 36864 0\ndfl_fme 49152 0\ndfl_pci 20480 0\ndfl 40960 7 dfl_pci,uio_dfl,dfl_fme,dfl_fme_br,qsfp_mem_dfl,dfl_afu,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 24576 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#51-memlock-limit","title":"5.1 Memlock limit","text":"Depending on the requirements of any loaded applications, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using:
ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#60-opae-tools-overview","title":"6.0 OPAE Tools Overview","text":"The following section offers a brief introduction including expected output values for the utilities included with OPAE. A full explanation of each command with a description of its syntax is available in the opae-sdk GitHub repo.
A list of all tools included in the OPAE SDK release can be found on the OPAE FPGA Tools tab of ofs.github.io.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#601-board-management-with-fpgainfo","title":"6.0.1 Board Management with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files.
Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Note: Your Bitstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo. As the F-Series Development Kit does not contain a traditional BMC as used by other OFS products, those lines in fpgainfo's output will not return valid objects. The subcommand fpgainfo bmc will likewise fail to report telemetry data.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x501020241BF165B\nBitstream Version : 5.0.1\nPr Interface Id : b4eda250-cdb7-5891-a06e-13d28d09bc32\nBoot Page : N/A\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#602-updating-with-fpgasupdate","title":"6.0.2 Updating with fpgasupdate","text":"The fpgasupdate tool is used to program AFU workloads into an open slot in a FIM. The fpgasupdate tool only accepts images that have been formatted using PACsign.
As the F-Series Development Kit does not contain a traditional BMC, you do not have access to a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL. Only the programming of a GBS workload is supported for this release.
The process of programming a SOF with a new FIM version is shown in section 4.0 Upgrading the F-Series Development Kit FIM via JTAG
sudo fpgasupdate ofs_pr_afu.gbs <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:42:31.58] [INFO ] updating from file ofs_pr_afu.gbs with size 19928064 [2022-04-14 16:42:31.60] [INFO ] waiting for idle [2022-04-14 16:42:31.60] [INFO ] preparing image file [2022-04-14 16:42:38.61] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] [2022-04-14 16:42:54.63] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] [2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:49:11.03] [INFO ] Secure update OK [2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#603-verify-fme-interrupts-with-hello_events","title":"6.0.3 Verify FME Interrupts with hello_events","text":"The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors.
Sample output from sudo hello_events.
sudo hello_events\nWaiting for interrupts now...\ninjecting error\nFME Interrupt occurred\nSuccessfully tested Register/Unregister for FME events!\nclearing error\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#61-host-exerciser-modules","title":"6.1 Host Exerciser Modules","text":"The reference FIM and unchanged FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. There are three HEMs present in the Intel OFS Reference FIM - HE-LPBK, HE-MEM, and HE-HSSI. These exercisers are tied to three different VFs that must be enabled before they can be used. Execution of these exercisers requires you bind specific VF endpoint to vfio-pci. The host-side software looks for these endpoints to grab the correct FPGA resource.
Refer to the Intel Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for a full description of these modules.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-5-module-pfvf-mappings","title":"Table 5: Module PF/VF Mappings","text":"Module PF/VF ST2MM PF0 HE-MEM PF0-VF0 HE-HSSI PF0-VF1 HE-MEM_TG PF0-VF2 HE-LB Stub PF1-VF0 HE-LB PF2 VirtIO LB Stub PF3 HPS Copy Engine PF4"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#611-he-mem-he-lb","title":"6.1.1 HE-MEM / HE-LB","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. The Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: - Latency (AFU to Host memory read) - MMIO latency (Write+Read) - MMIO BW (64B MMIO writes) - BW (Read/Write, Read only, Wr only)
The Host Exerciser Loopback Memory (HE-MEM) AFU is used to exercise use of FPGA connected DDR, data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host.
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. When using the F-Series Development Kit SmartNIC Platform, optimal performance requires the exercisers be run at 400 MHz.
Execution of these exercisers requires you to bind specific VF endpoint to vfio-pci. The following commands will bind the correct endpoint for a device with B/D/F 0000:b1:00.0 and run through a basic loopback test.
Note: While running the opae.io init command listed below, the command has failed if no output is present after completion. Double check that Intel VT-D and IOMMU have been enabled in the kernel as discussed in section 3.0 OFS DFL Kernel Drivers.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci Binding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci iommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188 Assigning /dev/vfio/188 to user Changing permissions for /dev/vfio/188 to rw-rw----\n\n$ sudo host_exerciser --clock-mhz 400 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 3002\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 8.732 GB/s\n Test lpbk(1): PASS\n
The following example will run a loopback throughput test using one cache line per request.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\n\n$ sudo host_exerciser --clock-mhz 400 --mode trput --cls cl_1 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2014\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 13.016 GB/s\n Test lpbk(1): PASS\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#612-traffic-generator-afu-test-application","title":"6.1.2 Traffic Generator AFU Test Application","text":"Beginning in OPAE version 2.0.11-1+ the TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank. The supported arguments for test configuration are:
- Number of test loops: --loops
- Number of read transfers per test loop: -r,--read
- Number of write transfers per test loop: -w,--write
- Burst size of each transfer: -b,--bls
- Address stride between each transfer: --stride
- Target memory TG: -m,--mem-channel
Below are some example commands for how to execute the test application. To run the preconfigured write/read traffic test on channel 0:
mem_tg tg_test\n
Target channel 1 with a 1MB single-word write only test for 1000 iterations
mem_tg --loops 1000 -r 0 -w 2000 -m 1 tg_test\n
Target channel 2 with 4MB write/read test of max burst length for 10 iterations
mem_tg --loops 10 -r 8 -w 8 --bls 255 -m 2 tg_test\n
sudo mem_tg --loops 1000 -r 2000 -w 2000 --stride 2 --bls 2 -m 1 tg_test\n[2022-07-15 00:13:16.349] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nTG PASS\nMem Clock Cycles: 17565035\nWrite BW: 4.37232 GB/s\nRead BW: 4.37232 GB/s\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#613-he-hssi","title":"6.1.3 HE-HSSI","text":"HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic.
The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode- specific options, and displays the ethernet statistics by invoking ethtool --statistics INTERFACE.
Due to Ethernet differential pair routing on the ES version of the Intel Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit, some differential pairs were swapped to improve signal routing. To account for the pair swap, there is a requirement to run a script to invert the differential traces. If you run the command \u201cfpgainfo phy B:d.f\u201d when the Ethernet ports are connected to known good sources and observe the following three ports are down as shown below:
$ sudo fpgainfo phy b1:00.0\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102020CB5B309\nBitstream Version : 5.0.1\nPr Interface Id : d0f93544-a487-5a92-8632-d43f27dbccee\nQSFP0 : Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
Create the following script called \u201cset_tx_inverse_polarity.sh\u201d to set make Transceiver PAM register settings:
#!/bin/sh\n# Port 3\nbase_addr=$(printf \"%08d\" \"0x500000\")\naddress=`expr $base_addr + 589884`\n#address=`expr $base_addr + 589884`\noffset=`expr $address/4`\nhex_number=$(printf \"0x%06x\" \"$(($offset))\")\necho $hex_number\ncmd_sts=$(printf \"%32x\" \"$(($offset2))\")\ncsraddr=\"${hex_number}0500000002\"\ncsraddr1=\"${hex_number}0600000001\"\ndata=1a040\necho $csraddr\nsudo opae.io poke 0x140b0 0x0001a26500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\nsleep 1\nsudo opae.io poke 0x140b0 0x0001226500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\n# Port 6\nbase_addr=$(printf \"%08d\" \"0xb00000\")\naddress=`expr $base_addr + 589884`\n#address=`expr $base_addr + 589884`\noffset=`expr $address/4`\nhex_number=$(printf \"0x%06x\" \"$(($offset))\")\necho $hex_number\ncmd_sts=$(printf \"%32x\" \"$(($offset2))\")\ncsraddr=\"${hex_number}0500000002\"\ncsraddr1=\"${hex_number}0600000001\"\ndata=1a040\necho $csraddr\nsudo opae.io poke 0x140b0 0x0001a16500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\nsleep 1\nsudo opae.io poke 0x140b0 0x0001216500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\n# Port 7\nbase_addr=$(printf \"%08d\" \"0x1100000\")\naddress=`expr $base_addr + 589884`\n#address=`expr $base_addr + 589884`\noffset=`expr $address/4`\nhex_number=$(printf \"0x%06x\" \"$(($offset))\")\necho $hex_number\ncmd_sts=$(printf \"%32x\" \"$(($offset2))\")\ncsraddr=\"${hex_number}0500000002\"\ncsraddr1=\"${hex_number}0600000001\"\ndata=1a040\necho $csraddr\nsudo opae.io poke 0x140b0 0x0001a26500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\nsleep 1\nsudo opae.io poke 0x140b0 0x0001226500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\n
The script set_tx_inverse_polarity.sh requires the VFIO driver on PF0 to access the Transceiver registers. You will use the opae.io command prior to running set_tx_inverse_polarity.sh to bind the VFIO driver. Once the script completes, release the VFIO driver with opae.io release.
The listing below shows the script being run:
sudo opae.io init -d 0000:b1:00.0 $USER\nUnbinding (0x8086,0xbcce) at 0000:b1:00.0 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.0 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.0 is 8\nsh set_tx_inverse_polarity.sh\n0x16400f\n0x16400f0500000002\n0x16400f0500000006\n0x16400f0500000006\n0x2e400f\n0x2e400f0500000002\n0x2e400f0500000006\n0x2e400f0500000006\n0x46400f\n0x46400f0500000002\n0x46400f0500000006\n0x46400f0500000006\n\nsudo opae.io release -d 0000:b1:00.0\nReleasing (0x8086,0xbcce) at 0000:b1:00.0 from vfio-pci\nRebinding (0x8086,0xbcce) at 0000:b1:00.0 to dfl-pci\n\nsudo fpgainfo phy b1:00.0\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x501020241BF165B\nBitstream Version : 5.0.1\nPr Interface Id : 767712e5-b1d0-5777-aea9-592572a6817f\nQSFP0 : Connected\nQSFP1 : Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE UP\nPort1 :25GbE UP\nPort2 :25GbE UP\nPort3 :25GbE UP\nPort4 :25GbE UP\nPort5 :25GbE UP\nPort6 :25GbE UP\nPort7 :25GbE UP\n
The following example walks through the process of binding the VF corresponding with the HE-HSSI exerciser to vfio-pci, sending traffic, and verifying that traffic was received.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-6-accelerator-pfvf-and-guid-mappings","title":"Table 6: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID F Series Dev Kit base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to user\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to user\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
-
Check Ethernet PHY settings with fpgainfo.
sudo fpgainfo phy -B 0xb1 Intel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102020CB5B309\nBitstream Version : 5.0.1\nPr Interface Id : d0f93544-a487-5a92-8632-d43f27dbccee\nQSFP0 : Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
-
Set loopback mode.
sudo hssiloopback --loopback enable --pcie-address 0000:b1:00.0 args Namespace(loopback='enable', pcie_address='0000:b1:00.0', port=0)\nsbdf: 0000:b1:00.0\nFPGA dev: {'segment': 0, 'bus': 177, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:b1:00.0'}\nargs.hssi_grps{0: ['dfl_dev.6', ['/sys/bus/pci/devices/0000:b1:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.6/uio/uio0']]}\nfpga uio dev:dfl_dev.6\n\n--------HSSI INFO START-------\nDFH :0x3000000010002015\nHSSI ID :0x15\nDFHv :0.5\nguidl :0x99a078ad18418b9d\nguidh :0x4118a7cbd9db4a9b\nHSSI version :1.0\nFirmware Version :1\nHSSI num ports :8\nPort0 :25GbE\nPort1 :25GbE\nPort2 :25GbE\nPort3 :25GbE\nPort4 :25GbE\nPort5 :25GbE\nPort6 :25GbE\nPort7 :25GbE\n--------HSSI INFO END-------\n\nhssi loopback enabled to port0\n
-
Send traffic through the 10G AFU.
$sudo hssi --pci-address b1:00.6 hssi_10g --num-packets 100 10G loopback test\nTx/Rx port: 99\nTx port: 0\nRx port: 0\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth:\n\nNo eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.\n0x40000 ETH_AFU_DFH: 0x1000010000001000\n0x40008 ETH_AFU_ID_L: 0xbb370242ac130002\n0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea\n0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000\n0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000\n0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000\n0x40048 AFU_SCRATCHPAD: 0x0000000045324511\n\n0x3c00 number_packets: 0x00000064\n0x3c01 random_length: 0x00000000\n0x3c02 random_payload: 0x00000000\n0x3c03 start: 0x00000000\n0x3c04 stop: 0x00000000\n0x3c05 source_addr0: 0x44332211\n0x3c06 source_addr1: 0x00006655\n0x3c07 dest_addr0: 0xaa998877\n0x3c08 dest_addr1: 0x0000ccbb\n0x3c09 packet_tx_count: 0x00000064\n0x3c0a rnd_seed0: 0x5eed0000\n0x3c0b rnd_seed1: 0x5eed0001\n0x3c0c rnd_seed2: 0x00025eed\n0x3c0d pkt_length: 0x00000040\n0x3cf4 tx_sta_tstamp: 0x02ab5f6d\n0x3cf5 tx_end_tstamp: 0x02ab63dd\n0x3d00 num_pkt: 0xffffffff\n0x3d01 pkt_good: 0x00000064\n0x3d02 pkt_bad: 0x00000000\n0x3d07 avst_rx_err: 0x00000000\n0x3d0b rx_sta_tstamp: 0x02ab6058\n0x3d0c rx_end_tstamp: 0x02ab64dc\n0x3e00 mac_loop: 0x00000000\n\nHSSI performance:\n Selected clock frequency : 402.832 MHz\n Latency minimum : 583.37 ns\n Latency maximum : 633.018 ns\n Achieved Tx throughput : 15.8863 GB/s\n Achieved Rx throughput : 15.6115 GB/s\n
The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. hssi_loopback tests both external and internal loopbacks.
The hssistats tool provides the MAC statistics.
"},{"location":"hw/iseries_devkit/","title":"I-Series (2xR-tile and 1xF-Tile) Development Kit Collateral for OFS","text":"This folder contains applicable collateral for OFS PCIe Attach reference shell targeting the I-Series (2xR-tile and 1xF-Tile) Development Kit DK-DEV-AGI027RBES.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/","title":"FPGA Interface Manager Developer Guide for Open FPGA Stack: Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a guide for OFS Agilex PCIe Attach developers targeting the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). The following topics are covered in this guide:
- Compiling the OFS Agilex PCIe Attach FIM design
- Simulating the OFS Agilex PCIe Attach design
- Customizing the OFS Agilex PCIe Attach FIM design
- Configuring the FPGA with an OFS Agilex PCIe Attach FIM design
The FIM Development Walkthroughs Table lists all of the walkthroughs provided in this guide. These walkthroughs provide step-by-step instructions for performing different FIM Development tasks.
Table: FIM Development Walkthroughs
Walkthrough Name Category Install Quartus Prime Pro Software Setup Clone FIM Repository Setup Set Development Environment Variables Setup Set Up Development Environment Setup Compile OFS FIM Compilation Manually Generate OFS Out-Of-Tree PR FIM Compilation Change the Compilation Seed Compilation Running Individual Unit Level Simulation Simulation Running Regression Unit Level Simulation Simulation Add a new module to the OFS FIM Customization Modify and run unit tests for a FIM that has a new module Customization Hardware test a FIM that has a new module Customization Debug the FIM with Signal Tap Customization Compile the FIM in preparation for designing your AFU Customization Resize the Partial Reconfiguration Region Customization Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Customization Modify PCIe Configuration Using IP Presets Customization Migrate to a Different Agilex Device Number Customization Modify the Ethernet Sub-System to 1x400GbE Customization Set up JTAG Configuration Program the FPGA via JTAG Configuration"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#111-knowledge-pre-requisites","title":"1.1.1 Knowledge Pre-Requisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex 7 PCIe Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Signal Tap Logic Analyzer tool software.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex PCIe Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex PCIe Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new acceleration card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":""},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1211-top-level","title":"1.2.1.1 Top Level","text":"Figure: OFS Agilex PCIe Attach iseries-dk FIM PCIe 1x16 Top-Level Diagram
Figure: OFS Agilex PCIe Attach iseries-dk FIM PCIe 2x8 Top-Level Diagram
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex PCIe Attach design are listed in the Release Capabilities Table. It describes the capabilities of the iseries-dk hardware as well as the capabilities of the default OFS Agilex PCIe Attach design targeting the iseries-dk.
Table: Release Capabilities
Interface iseries-dk Hardware Capabilities1 OFS Agilex PCIe Attach Provided Design Implementation Host Interface PCIe Gen5x16 \u2022 PCIe 1xGen5x16 (Default) \u2022 Bifurcated PCIe 2xGen5x8\u2022 PCIe 1xGen4x16 Network Interface 2 - QSFP-DD 3 Build Options: 1. QSFP 1,0 = 25 GbE2. QSFP 1,0 = 200 GbE 3. QSFP 0 = 400 GbE External Memory 2 - board mounted independent single rank DDR4-2666 8GB (1 Gb x 64 + 8b ECC)2 - DIMM sockets where each socket is single memory channels or independent channels (Check Dev Kit OPN for support option)2. 4xDDR4-2666 - 8GB (1Gb x 64 bits) - ECC not implemented by default 1 The iseries-dk FIM design was validated on DK-DEV-AGI027RBES with Agilex 7 device number AGIB027R29A1E2VR3. If you wish to use the production develpment kit (DK-DEV-AGI027-RA), you will need to migrate to device number AGIB027R29A1E1VB. Refer to the Migrate to a Different Agilex Device Number section for general instructions on this process.
2 The iseries-dk was validated with 2 Micron MTA8ATF1G64AZ-2G6E1 DDR4 SDRAM modules in the DIMM sockets.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex PCIe Attach iseries-dk FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem AXI Streaming IP for PCI Express User Guide 790711 Memory Subsystem Memory Subsystem Intel FPGA IP User Guide for Intel Agilex OFS 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workload in the OFS Agilex PCIe Attach iseries-dk FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI Used to exercise and characterize HSSI interfaces. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex PCIe Attach iseries-dk FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the host software and AFU and board peripherals. The APF Address Map Table describes the address mapping of the APF, followed by the BPF Address Map Table which describes the address mapping of the BPF.
Table: APF Address Map
Address Size (Bytes) Feature 0x00000\u20130x3FFFF 256K Board Peripherals (See BPF Address Map table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART (not used) 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket:4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x80000 \u2013 0x80FFF 4K AFU Error Reporting Table: BPF Address Mapping
Address Size (Bytes) Feature 0x00000 - 0x0FFFF 64K FME 0x10000 - 0x10FFF 4K PCIe 0x11000 - 0x11FFF 4K Reserved 0x12000 - 0x12FFF 4K QSFP0 0x13000 - 0x13FFF 4K QSFP1 0x14000 - 0x14FFF 4K HSSI 0x15000 - 0x15FFF 4K EMIF 0x20000 - 0x3FFFF 128K PMCI (note, PMCI is not implemented)"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customizations Table lists the general user flows for OFS Agilex PCIe Attach iseries-dk FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customizations
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Configuration Using IP Presets Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 1x400GbE"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA acceleration card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Please see the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up the environment for deployment machines.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 N/A Quartus Prime Software Quartus Prime Pro Version 24.1 for Linux + Patches 0.18, 0.26 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A git 1.8.3.1 or later Section 1.3.1.2 FIM Source Files ofs-2024.2-1 Section 1.3.2.1"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"The source files for the OFS Agilex PCIe Attach FIM are provided in the following repository: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
Some essential directories in the repository are described as follows:
ofs-agx7-pcie-attach\n| syn // Contains files related to synthesis\n| | board // Contains synthesis files for several cards, including the iseries-dk | | | iseries-dk // Contains synthesis files for iseries-dk\n| | | | setup // Contains setup files, including pin constraints and location constraints\n| | | | syn_top // Contains Quartus project files\n| verification // Contains files for UVM testing\n| ipss // Contains files for IP Sub-Systems\n| | qsfp // Contains source files for QSFP Sub-System\n| | hssi // Contains source files for HSSI Sub-System\n| | pmci // Contains source files for PMCI Sub-System (not used in F-Tile FIM)\n| | pcie // Contains source files for PCIe Sub-System\n| | mem // Contains source files for Memory Sub-System\n| sim // Contains simulation files\n| | unit_test // Contains files for all unit tests\n| | | scripts // Contains script to run regression unit tests\n| license // Contains Quartus patch\n| ofs-common // Contains files which are common across OFS platforms\n| | verification // Contains common UVM files\n| | scripts // Contains common scripts\n| | | common\n| | | | syn // Contains common scripts for synthesis, including build script\n| | | | sim // Contains common scripts for simulation\n| | tools // Contains common tools files\n| | | mk_csr_module // Contains common files for CSR modules\n| | | fabric_generation // Contains common files for APF/BPF fabric generation\n| | | ofss_config // Contains common files for OFSS configuration tool\n| | | | ip_params // Contains default IP parameters for certain Sub-Systems when using OFSS\n| | src // Contains common source files, including host exercisers\n| tools //\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| | | hssi // Contains OFSS files for Ethernet-SS configuraiton\n| | | memory // Contains OFSS files for Memory-SS configuration\n| | | pcie // Contains OFSS files for PCIe-SS configuration\n| | | iopll // Contains OFSS files for IOPLL configuration\n| src // Contains source files for Agilex PCIe Attach FIM\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | includes // Contains source file header files\n| | top // Contains top-level source files, including design top module\n| | afu_top // Contains top-level source files for AFU\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 PCIe Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n
-
Check out the correct tag of the repository
cd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.2-1)\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-agx7-pcie-attach\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.ISERIES_DK_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.ISERIES_DK_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 24.1 for Linux with Agilex FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 24.1 Build 94 06/14/2023 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
git 1.8.3.1 or later.
-
Verify version number
git --version\n
Example output:
git version 1.8.3.1\n
-
Clone the ofs-agx7-pcie-attach repository. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.18, 0.26. All required patches are provided in the Assets of the OFS FIM Release Tag: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
-
Extract and unzip the patch-agx7-2024-1.tar.gz file.
tar -xvzf patch-agx7-2024-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-23.4-0.17-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
- Change the Compilation Seed
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This section describes the theory behind FIM compilation.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2121-top-level-ofss-file","title":"2.1.2.1 Top Level OFSS File","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2122-base-ofss-file","title":"2.1.2.2 Base OFSS File","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2123-pcie-ofss-file","title":"2.1.2.3 PCIe OFSS File","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2124-iopll-ofss-file","title":"2.1.2.4 IOPLL OFSS File","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2125-memory-ofss-file","title":"2.1.2.5 Memory OFSS File","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is: $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/iseries-dk/syn_top/output_files.
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes the images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Name Description ofs_top.sof The FIM design SRAM Object File; a binary file of the compiled FIM image."},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <PATH_OF_RELOCATABLE_PR_TREE> <BOARD_TARGET> <WORK_DIRECTORY>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <PATH_OF_RELOCATABLE_PR_TREE> String Specifies the location of the relocatable PR directory tree to be created. <BOARD_TARGET> iseries-dk Specifies the name of the board target. <WORK_DIRECTORY> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u2514\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#224-he_null-fim","title":"2.2.4 HE_NULL FIM","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex PCIe Attach FIM for iseries-dk:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. The following is used build the default iseries-dk design:
./ofs-common/scripts/common/syn/build_top.sh iseries-dk work_iseries-dk\n
The build command options allow for many modifications to the shell design at build time. The following tool is provided to help you conveniently get the build command for a specific shell configuration:
OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command. Note: If no OFSS file is specified, the build script will use the <target>.ofss file by default.
- Once the build script is complete, the build summary should report that the build is complete and passes timing. For example:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: iseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 1\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#226-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM. This can be automatically done for you if you run the build script with the -p option. This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the iseries-dk OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
./ofs-common/scripts/common/syn/build_top.sh iseries-dk work_iseries-dk\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_iseries-dk/pr_build_template iseries-dk work_iseries-dk\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
set_global_assignment -name SEED 2\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#3-fim-simulation","title":"3. FIM Simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config> OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> iseries-dk Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional Refer to the Running Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files automatically; refer to the [Run Regression Unit Level Simulation] section for step-by-step instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#321-walkthrough-running-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Running Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the iseries-dk
./gen_sim_files.sh iseries-dk\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"You may use the regression script regress_run.py to run some or all of the unit tests available with a single command. The regression script is in the following location:
$OFS_ROOTDIR/sim/unit_test/scripts/regress_run.py\n
The usage of the regression script is as follows:
regress_run.py [-h] [-l] [-n <num_procs>] [-k <test_package>] [-s <simulator>] [-g] [--ofss <ip_config>] [-b <board_name>] [-e]\n
The Regression Unit Test Script Options table describes the options for the regress_run.py script.
Table: Regression Unit Test Script Options
Field Options Description -h | --help N/A Show the help message and exit -l | --local N/A Run regression locally -n | --n_procs Integer Maximum number of processes/tests to run in parallel when run locally. This has no effect on farm runs. -k | --pack all | fme | he | hssi | list | mem | pmci Test package to run during regression. The \"list\" option will look for a text file named \"list.txt\" in the \"unit_test\" directory for a text list of tests to run (top directory names). The default test package is all. -s | --sim vcs | vcsmx | msim Specifies the simulator used for the regression tests. The default simulator is vcs -g | --gen_sim_files N/A Generate simulation files. This only needs to be done once per repo update. This is the equivalent of running the gen_sim_files.sh script. -o | --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. -b | --board_name iseries-dk Specifies the board target -e | --email_list String Specifies email list to send results to multiple recipients The log for each unit test that is run by the regression script is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#331-walkthrough-running-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Running Regression Unit Level Simulation","text":"Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a test list file to only run the unit level simulations that are supported for the iseries-dk FIM.
touch $OFS_ROOTDIR/sim/unit_test/list.txt\n
Copy the following list into the new file. You may remove tests from this list as desired.
./bfm_test/set_params.sh\n./csr_test/set_params.sh\n./dfh_walker/set_params.sh\n./flr/set_params.sh\n./fme_csr_access/set_params.sh\n./fme_csr_directed/set_params.sh\n./he_lb_test/set_params.sh\n./he_mem_lb_test/set_params.sh\n./he_null_test/set_params.sh\n./hssi_csr_test/set_params.sh\n./hssi_kpi_test/set_params.sh\n./hssi_test/set_params.sh\n./indirect_csr/set_params.sh\n./mem_ss_csr_test/set_params.sh\n./mem_ss_rst_test/set_params.sh\n./mem_tg_test/set_params.sh\n./pcie_ats_basic_test/set_params.sh\n./pcie_csr_test/set_params.sh\n./pcie_ss_axis_components/set_params.sh\n./pf_vf_access_test/set_params.sh\n./port_gasket_test/set_params.sh\n./qsfp_test/set_params.sh\n./remote_stp_test/set_params.sh\n./uart_csr/set_params.sh\n
-
Navigate to the unit test scripts directory.
cd $OFS_ROOTDIR/sim/unit_test/scripts\n
-
Run regression test with the your desired options. For example, to simulate with the options to generate simulation files, run locally, use 8 processes, run the specified test list, use VCS simulator, and target the iseries-dk:
python regress_run.py -g -l -n 8 -k list -s vcs -b iseries-dk\n
Note: You may run all available tests by using -k all instead of creating and using -k list, however not all tests are supported depending on the target board.
-
Once all tests are complete, check that the tests have passed.
Passing Unit Tests:24/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n bfm_test:............... PASS -- Time Elapsed:0:01:14.600452\n csr_test:............... PASS -- Time Elapsed:0:01:30.972054\n dfh_walker:............. PASS -- Time Elapsed:0:01:15.179127\n flr:.................... PASS -- Time Elapsed:0:01:14.579890\n fme_csr_access:......... PASS -- Time Elapsed:0:00:48.545993\n fme_csr_directed:....... PASS -- Time Elapsed:0:00:54.702789\n he_lb_test:............. PASS -- Time Elapsed:0:02:11.371956\n he_mem_lb_test:......... PASS -- Time Elapsed:0:41:32.226191\n he_null_test:........... PASS -- Time Elapsed:0:01:11.791063\n hssi_csr_test:.......... PASS -- Time Elapsed:0:44:10.611323\n hssi_kpi_test:.......... PASS -- Time Elapsed:2:28:24.465424\n hssi_test:.............. PASS -- Time Elapsed:2:23:52.603328\n indirect_csr:........... PASS -- Time Elapsed:0:01:02.535460\n mem_ss_csr_test:........ PASS -- Time Elapsed:0:23:37.683859\n mem_ss_rst_test:........ PASS -- Time Elapsed:0:45:19.603426\n mem_tg_test:............ PASS -- Time Elapsed:0:28:59.823955\n pcie_ats_basic_test:.... PASS -- Time Elapsed:0:01:10.104139\n pcie_csr_test:.......... PASS -- Time Elapsed:0:01:10.891950\n pcie_ss_axis_components: PASS -- Time Elapsed:0:02:04.448343\n pf_vf_access_test:...... PASS -- Time Elapsed:0:01:09.465886\n port_gasket_test:....... PASS -- Time Elapsed:0:01:11.912088\n qsfp_test:.............. PASS -- Time Elapsed:0:05:10.887379\n remote_stp_test:........ PASS -- Time Elapsed:0:01:14.684407\n uart_csr:............... PASS -- Time Elapsed:0:01:34.763679\nFailing Unit Tests: 0/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n Number of Unit test results captured: 24\nNumber of Unit test results passing.: 24\nNumber of Unit test results failing.: 0\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4-fim-customization","title":"4. FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Table: FIM Customization Walkthroughs
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Configuration Using IP Presets Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 1x400GbE"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a information for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example will add a hello_fim module to the design. The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the Host. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the iseries-dk card. The process for these are described in this section.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can be mastered by the Host. The BPF is an interconnect generated by Platform Designer. The Hello FIM BPF Interface Diagram figure shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
Figure: Hello FIM BPF Interface Diagram
The BPF fabric is defined in $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt.
We will add the Hello FIM module to an un-used address space in the MMIO region. The Hello FIM MMIO Address Layout table below shows the MMIO region for the Host with the Hello FIM module added at base address 0x16000.
Table: Hello FIM MMIO Address Layout
Offset Feature CSR set 0x00000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 HSSI Interface 0x15000 EMIF Interface 0x16000 Hello FIM"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the Hello FIM CSR table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Table: Hello FIM CSR
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Make hello_fim source directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create hello_fim_top.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_top.sv\n
Copy the following code into hello_fim_top.sv:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : September 2023\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create hello_fim_com.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_com.sv\n
Copy the following code to hello_fim_com.sv:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Create hello_fim_design_files.tcl file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_design_files.tcl\n
Copy the following code into hello_fim_design_files.tcl
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qsf to include Hello FIM module
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top_sources.tcl to include Hello FIM design files
############################################\n# Design Files\n############################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_design_files.tcl\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/fabric_design_files.tcl to include BPF Hello FIM Slave IP.
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE $::env(BUILD_ROOT_REL)/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify $OFS_ROOTDIR/src/includes/fabric_width_pkg.sv to add Hello FIM slave information and update EMIF slave next offset.
localparam bpf_hello_fim_slv_baseaddress = 'h16000; // New\nlocalparam bpf_hello_fim_slv_address_width = 12; // New\nlocalparam bpf_emif_slv_next_dfh_offset = 'h1000; // Old value: 'hB000\nlocalparam bpf_hello_fim_slv_next_dfh_offset = 'hA000; // New\nlocalparam bpf_hello_fim_slv_eol = 'b0; // New\n
-
Modify $OFS_ROOTDIR/src/top/top.sv
-
Add bpf_hello_fim_slv_if to AXI interfaces
// AXI4-lite interfaces\n...\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width), .ARADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width)) bpf_hello_fim_slv_if();\n
-
Add Hello FIM instantiation
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (fabric_width_pkg::bpf_hello_fim_slv_address_width),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`else\ndummy_csr #(\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif\n
-
Add interfaces for Hello FIM slv to bpf instantiation
bpf bpf (\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\n);\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt to add the hello_fim module as a slave to the apf.
# NAME FABRIC BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 18 fme,pcie,pmci,qsfp0,qsfp1,emif,hssi,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute helper script to generate BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\n\nsh gen_fabrics.sh\n
-
Once the script completes, the following new IP is created: $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip.
-
[OPTIONAL] You may verify the BPF changes have been made correctly by opening bpf.qsys to analyze the BPF.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\n\nqsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
Find the bpf_hello_fim_slv instance:
-
Compile the Hello FIM design
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p iseries-dk work_iseries-dk_hello_fim\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/dfh_walker/test_csr_defs.sv
-
Add HELLO_FIM_IDX entry to t_dfh_idx enumeration.
...\ntypedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX, // New\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nVUART_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n...\n
-
Add HELLO_FIM_DFH to get_dfh_names function.
...\nfunction automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\n...\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\"; // New\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\n...\nreturn dfh_names;\n...\n
-
Add expected DFH value for Hello FIM to the get_dfh_values function.
...\nfunction automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\n...\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_xxxxxx_1012;\ndfh_values[PMCI_DFH_IDX][39:16] = fabric_width_pkg::bpf_pmci_slv_next_dfh_offset;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_xxxxxx_0100; // New\ndfh_values[HELLO_FIM_DFH_IDX][39:16] = fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset; // New\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_xxxxxx_0014;\ndfh_values[ST2MM_DFH_IDX][39:16] = fabric_width_pkg::apf_st2mm_slv_next_dfh_offset;\n...\nreturn dfh_values;\n...\n
-
Generate simulation files
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\n./gen_sim_files.sh iseries-dk\n
-
Run DFH Walker Simulation
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\nsh run_sim.sh TEST=dfh_walker\n
-
Verify that the test passes, and that the output shows the Hello FIM in the DFH sequence
********************************************\n Running TEST(0) : test_dfh_walking\n********************************************\n\n...\n\nTX: Tag Search Comparison: CPLD Tag: 00a Active Tag: 00a\nTX: PU Completion ADDED to transaction!\nTX: Match found for PU CPLD!\nRead64 Method Data Transaction: Address: 0000_0000_0001_5000 Data: 3000_0000_1000_1009\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nTX: Tag Search Comparison: CPLD Tag: 00b Active Tag: 00b\nTX: PU Completion ADDED to transaction!\nTX: Match found for PU CPLD!\nRead64 Method Data Transaction: Address: 0000_0000_0001_6000 Data: 3000_0000_a000_0100\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000000a0000100)\nTX: Tag Search Comparison: CPLD Tag: 00c Active Tag: 00c\nTX: PU Completion ADDED to transaction!\nTX: Match found for PU CPLD!\nRead64 Method Data Transaction: Address: 0000_0000_0002_0000 Data: 3000_0002_0000_1012\nPMCI_DFH\n Address (0x20000)\nDFH value (0x3000000200001012)\n...\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\n$finish called from file \"/home/applications.fpga.ofs.fim-n6001/sim/unit_test/dfh_walker/unit_test.sv\", line 236.\n$finish at simulation time 13720.141ns\n V C S S i m u l a t i o n R e p o r t\nTime: 13720141000 fs\nCPU Time: 3.290 seconds; Data structure size: 5.8Mb\nMon Dec 4 09:27:50 2023\nTotal of 5 minutes elapsed for dfh_walker\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#414-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
[OPTIONAL] In the work directory where the FIM was compiled, determine the PR Interface ID of your design. You can use this value at the end of the walkthrough to verify that the design has been configured to the FPGA.
cd $OFS_ROOTDIR/<work_directory>/syn/board/iseries-dk/syn_top/\n\ncat fme-ifc-id.txt\n
Example output:
b7855572-6ca9-58b8-bd11-44e1f1ab329f\n
-
Switch to your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Run fpgainfo to determine the PCIe B:D.F of your board, and to verify the PR Interface ID matches the ID you found in Step 1.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
-
Initialize opae.io
sudo opae.io init -d <B:D.F>\n
For example:
sudo opae.io init -d b1:00.0\n
-
Run DFH walker. Note the value read back from offset 0x16000 indicates the DFH ID is 0x100 which matches the Hello FIM module.
sudo opae.io walk -d <B:D.F>\n
For example:
sudo opae.io walk -d b1:00.0\n
Example output:
...\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000000a0000100\n dfh: id = 0x100, rev = 0x0, next = 0xa000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000200001012\n dfh: id = 0x12, rev = 0x1, next = 0x20000, eol = 0x0, reserved = 0x0, feature_type = 0x3\n...\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d b1:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d b1:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d b1:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:b1:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration JTAG PCI Development Kit (Driver: dfl-pci)\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#415-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.5 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Perform the following steps in your development environment:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Synthesize the design using the -e build script option. You may skip this step if you are using a pre-synthesized design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -e iseries-dk work_hello_fim_with_stp\n
-
Open the design in Quartus. The Compilation Dashboard should show that the Analysis & Synthesis step has completed.
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
-
Open Tools -> Signal Tap Logic Analyzer
-
Select the Default template and click Create
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note that the clock selected should correspond to the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the block of interest and open the design instance for review.
-
In the Signal Configuration -> Clock box of the Signal Tap Logic Analyzer window, click the \"...\" button
-
In the Node Finder tool that opens, type clock into the Named field, and select hello_fim_top_inst in the Look in field, then click Search. Select the hello_fim_top_inst|clk signal from the Matching Nodes box and click the \">\" button to move it to the Nodes Found box. Click OK to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM.
-
In the Named field, enter csr_lite_if*. In the Look In field, select hello_fim_top_inst.
-
Select the nets that appear in the Matching Nodes box. Click the > button to add them to the Nodes Found box.
-
Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto_signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, stp_for_hello_fim.
-
Save the newly created Signal Tap file, in the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/ directory, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will aurtomatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE STP_For_Hello_FIM.stp\nset_global_assignment -name SIGNALTAP_FILE STP_For_Hello_FIM.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
./ofs-common/scripts/common/syn/build_top.sh -p -k iseries-dk work_hello_fim_with_stp\n
-
Ensure that the compile completes successfully and meets timing:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: iseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 1\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the iseries-dk. Refer to Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the iseries-dk via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/output_files/ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open the Quartus Signal Tap GUI
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap Logic Analyzer window, select File -> Open, and choose the stp_for_hello_fim.stp file.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable for the iseries-dk. In the Device: selection box select the Agilex device.
-
If the Agilex Device is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
Create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'stp_for_hello_fim' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, walk the DFH list or peek/poke the Hello FIM registers.
opae.io init -d 0000:b1:00.0\nopae.io walk -d 0000:b1:00.0\nopae.io release -d 0000:b1:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":"In the FIM build, the standard AFUs in the port gasket can be replaced with PIM-based AFUs, wrapped by the ofs_plat_afu() top-level module. Before invoking build_top.sh, set the environment variable AFU_WITH_PIM to the path of a sources text file -- the same sources file from examples such as sources.txt in hello_world:
export AFU_WITH_PIM=<path to>/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt\n
When set during the build_top.sh setup stage, the FIM build is configured to load the specified AFU. PR will continue to work, if enabled. The only difference with AFU_WITH_PIM is the change to the AFUs present at power-on. Keep in mind that the default exercisers were chosen to attach to all devices and force reasonable routing decisions during the FIM build across the PR boundary. AFU_WITH_PIM is best used for FIMs that disable PR.
Configuration of the user clock from an AFU's JSON file is ignored in a FIM build.
The AFU_WITH_PIM setting matters only during the setup stage of build_top.sh. It is not consumed by later stages.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#422-compiling-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2 Compiling the FIM in preparation for designing your AFU","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" blocks. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination, order or all can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4221-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_iseries-dk\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to adjust the PR region to fit the additional logic into the static region. Similarly, if you reduce logic in the FIM region, then you can adjust the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions as reported in the following file:
$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/iseries-dk/syn_top/output_files/ofs_top.fit.rpt\n
An example report of the resources usage by partitions defined for the FIM is shown as follows:
+------------------------------------------------------------------------------------------+\n; Logic Lock Region Constraints ;\n+--------------------------------------+-------------------------+-------------------------+\n; Name ; Place Region Constraint ; Route Region Constraint ;\n+--------------------------------------+-------------------------+-------------------------+\n; afu_top|port_gasket|pr_slot|afu_main ; (90, 40) to (350, 220) ; (0, 0) to (385, 329) ;\n+--------------------------------------+-------------------------+-------------------------+\n+----------------------------------------------------------------------------------------------+\n; Logic Lock Region Usage Summary ;\n+-------------------------------------------------------+--------------------------------------+\n; Statistic ; afu_top|port_gasket|pr_slot|afu_main ;\n+-------------------------------------------------------+--------------------------------------+\n; ALMs needed [=A-B+C] ; 48011.2 / 351140 ( 13 % ) ;\n; [A] ALMs used in final placement ; 53324.4 / 351140 ( 15 % ) ;\n; [B] Estimate of ALMs recoverable by dense packing ; 5452.3 / 351140 ( 1 % ) ;\n; [C] Estimate of ALMs unavailable ; 139.0 / 351140 ( < 1 % ) ;\n; ALMs used for memory ; 450.0 ;\n; Combinational ALUTs ; 67166 ;\n; Dedicated Logic Registers ; 87533 / 1404560 ( 6 % ) ;\n; I/O Registers ; 0 ;\n; Block Memory Bits ; 1737568 ;\n; M20Ks ; 137 / 5049 ( 2 % ) ;\n; DSP Blocks needed [=A-B] ; 0 / 3439 ( 0 % ) ;\n; [A] DSP Blocks used in final placement ; 0 / 3439 ( 0 % ) ;\n; [B] Estimate of DSPs recoverable by dense merging ; 0 / 3439 ( 0 % ) ;\n; Pins ; 0 ;\n; IOPLLs ; 0 ;\n; ; ;\n; Region Placement ; (90, 40) to (350, 220) ;\n+-------------------------------------------------------+--------------------------------------+\n
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to first analyze the PR logic lock regions in a default FIM design, then resize the PR region:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
The OFS flow provides the TCL file $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl which defines the PR partition where the AFU is allocated. Each region is a rectangle defined by the origin coordinate (X0, Y0) and the top right corner coordinate (X1, Y1).
#####################################################\n# Main PR Partition -- green_region\n#####################################################\nset_instance_assignment -name PARTITION green_region -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name CORE_ONLY_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name RESERVE_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name PARTIAL_RECONFIGURATION_PARTITION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n\nset_instance_assignment -name PLACE_REGION \"X90 Y40 X350 Y220\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name ROUTE_REGION \"X0 Y0 X385 Y329\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n
-
[OPTIONAL] Use Quartus Chip Planner to visualize the default PR region allocation.
-
Compile the design.
./ofs-common/scripts/common/syn/build_top.sh iseries-dk work_iseries-dk\n
-
Once the design is compiled, open it in Quartus.
quartus $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
-
Switch to the ofs_top view if necessary.
-
Click Tools -> Chip Planner to open the Chip Planner.
-
Analyze the regions shown. Note that the regions are made of rectangles described by an origin coordinate, region height, and region width. If you are modifying the regions, you will need to identify the coordinates of your desired region.
-
Close the Quartus GUI.
-
Make changes to the partial reconfiguraton region in the $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl file. You can modify the size and location of existing lock regions or add new ones and assign them to the AFU PR partition.
-
Recompile your FIM and create the PR relocatable build tree using the following commands.
cd $OFS_ROOTDIR ofs-common/scripts/common/syn/build_top.sh -p iseries-dk work_iseries-dk_resize_pr\n
-
Analyze the resource utilization report $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/output_files/ofs_top.fit.rpt produced after recompiling the FIM.
-
Perform further modification to the PR regions until the results are satisfactory. Make sure timing constraints are met.
For more information on how to optimize the floor plan of your Partial Reconfiguration design refer to the following online documentation.
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3 - Floorplan the Design
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe Subsystem can be easily modified using OFS provided script and the PCIe subsystem IP core. In this section both the PCIe SR-IOV configuration and PCIe configuration registers will be modified. You can use this process for setting up your specific settings.
The PCIe configuration for the iseries-dk can be set to 1x16 or bifurcated 2x8. You must ensure that the server you are using is set according to the design you are using. You can use OFSS to select which configuration to build with. Refer to the PCIe Configuration Using OFSS section for details. The following OFSS files are provided to you in the source repository:
- PCIe 1xGen5x16: $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss
- PCIe 2xGen5x8: $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link.ofss
- PCIe 1xGen4x16: $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_gen4.ofss
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section. Note that the older IP does not support the PCIe Gen5x16 configuration; it only supports Gen5 2x8 or Gen4 1x16. A walkthrough for these changes is provided later in this section.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS PCIe Attach FIM for the n6001 can support up to 8 PFs and 2000 VFs distributed accross all PFs.
For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 (if at least 1 VF present) | 2 (if no VFs present) Max # of PFs 8 Min # of VFs 1 on PF0 (if only 1 PF present) | 0 (if 2PFs present) Max # of VFs 2000 distributed across all PFs Note that as the number of VFs goes up, timing closure can become more difficult.
The scripts provided in ${OFS_ROOTDIR}/ofs-common/tools/ofss_config allows you to easily reconfigure the number of PFs and VFs, bar addresses, vendor/device ID values and more. The PCIe Subsystem IP parameters that can be modified can be seen by reviewing ${OFS_ROOTDIR}/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py
New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe-SS configuration registers contain the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. pcie_gen 4 | 5 N/A Specifies the PCIe generation pcie_instances 1 | 2 N/A Specifies the number of PCIe instances. When set to 1 the PCIe configuration will be 1x16. When set to 2 the PCIe configuration will be x8; the pcie_instances_enabled parameter will determine if it is 2x8 or 1x8. pcie_instances_enabled 1 | 2 N/A Specifies the number of PCIe instances that are enabled. When set to 2 the PCIe configuration will be 2x8 if; when set to 1 the PCIe configuration will be 1x16. Use only when the pcie_instances parameter is set to 2. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00000001 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00000001"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4431-walkthrough-modify-the-pcie-sub-system-and-pfvf-mux-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS","text":"Perform the following steps to use OFSS files to configure the PCIe Sub-system and PF/VF MUX. In this example both the PCIe SR-IOV configuration and PCIe configuration registers will be modified.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- To demonstrate updated PCIe PF/VF in hardware, use an OFS Agilex I-Series PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
View the default OFS PCIe Attach FIM for the iseries-dk PF/VF configuration in the the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n
-
Create a new PCIe OFSS file from the existing pcie_host.ofss file
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_pfvf_mod.ofss\n
-
Configure the new pcie_pfvf_mod.ofss with your desired PCIe settings. In this example we will add PF5 with 2 VFs.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n[pf5]\nnum_vfs = 2\n
-
Compile the FIM with the new PCIe OFSS file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_pfvf_mod.ofss iseries-dk work_iseries-dk_pfvf_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_iseries-dk_pfvf_mod/syn/board/iseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Verify the number of VFs on the newly added PF5. In this example, we defined 2 VFs on PF5 in Step 5.
sudo lspci -vvv -s b1:00.5 | grep VF\n
Example output:
Initial VFs: 2, Total VFs: 2, Number of VFs: 0, Function Dependency Link: 05\nVF offset: 4, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
-
Verify communication with the newly added PF5. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
-
Initialize the driver on PF5
sudo opae.io init -d b1:00.5\n
-
Read the GUID for the PF5 CSR stub.
sudo opae.io -d b1:00.5 -r 0 peek 0x8\nsudo opae.io -d b1:00.5 -r 0 peek 0x10\n
Example output:
0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4432-walkthrough-build-the-2x8-pcie-sub-system","title":"4.4.3.2 Walkthrough: Build the 2x8 PCIe Sub-System","text":"Perform the following steps to use OFSS files to build the FIM with a 2xGen5x8 PCIe configuration.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- To demonstrate updated PCIe PF/VF in hardware, use an OFS Agilex I-Series PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Build the FIM with the 2xGen5x16 PCIe configuration.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link.ofss iseries-dk work_iseries-dk\n
-
When the build script completes, copy the resulting $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/output_files/ofs_top.sof to your deployment environment if it is different than your development environment.
-
Switch to your deployment environment.
Note: Ensure that your server is configured for PCIe 2x8 bifurcated mode before continuing.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Verify the 6 PFs (3 for each link) are present with proper device ID. Not the PCIe B:D.F for each as this will be used later.
$ lspci | grep bcce\n
Example output:
ac:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nac:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nac:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nad:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nad:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nad:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
-
Verify the new VFs can be added. Use the OPAE SDK command pci_device to create VFs. Verify PF 0 has the proper number of VFs and have device ID of bccf.
sudo pci_device ac:00.0 vf 3\nsudo pci_device ad:00.3 vf 3\n
-
Verify that the VFs have been added.
```bash sudo lspci -vvv -s ac:00.0 | grep VF sudo lspci -vvv -s ad:00.0 | grep VF
Example Outputs:\n\n```bash\nInitial VFs: 3, Total VFs: 3, Number of VFs: 3, Function Dependency Link: 00\nVF offset: 3, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n\nInitial VFs: 3, Total VFs: 3, Number of VFs: 3, Function Dependency Link: 00\nVF offset: 3, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#444-pcie-configuration-using-ip-presets","title":"4.4.4 PCIe Configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4441-walkthrough-modify-pcie-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Configuration Using IP Presets","text":"Perform the following steps to use OFSS files to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID of PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- To demonstrate updated PCIe PF/VF in hardware, use an OFS Agilex I-Series PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script using your desired OFSS configration to create a working directory for the iseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup iseries-dk work_iseries-dk\n
-
Open the PCIe-SS in the work directory using Quartus Parameter Editor.
qsys-edit $OFS_ROOTDIR/work_iseries-dk/ipss/pcie/qip/pcie_ss.ip\n
-
In the IP Parameter Editor window, scroll down and select the PCIe Interfaces Port Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab. Modify the settings as desired. In this case, we are changing the Revision ID to 0x00000002. You may make any desired modifications.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, iseries-dk-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 8.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = iseries-dk-rev2\n
-
Compile the design with the modified new pcie_host_mod_preset.ofss file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_mod_preset.ofss iseries-dk work_iseries-dk_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_pcie_mod/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Determing the PCIe B:D.F of your board. You may use the OPAE command fpgainfo fme to determine this.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
-
Use lspci with the PCIe B:D.F of your board to verify that the PCIe changes have been implemented. In this example, the Rev for PF0 is 02.
lspci -nvmms 84:00.0\n
Example output:
Slot: 84:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 0001\nPhySlot: 1-1\nRev: 01\nNUMANode: 1\nIOMMUGroup: 190\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#445-pcie-sub-system-target-ip","title":"**4.4.5 PCIe Sub-System Target IP **","text":"As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section. Note that the older IP does not support the PCIe Gen5x16 configuration; it only supports Gen5 2x8 or Gen4 1x16.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4451-walkthrough-change-the-pcie-sub-system-target-ip","title":"4.4.5.1 Walkthrough: Change the PCIe Sub-System Target IP","text":"Perform the following steps to change the target PCIe IP from AXI Streaming Intel FPGA IP for PCI Express to Intel FPGA IP Subsystem for PCI Express.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Add the following line to the [settings] section of the PCIe OFSS file you are using. In this example, it will be added to the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link.ofss file. Note that the older IP does not support the PCIe Gen5x16 configuration; it only supports Gen5 2x8 or Gen4 1x16. By using the pcie_host_2link.ofss file, the PCIe configuration will be Gen5 2x8.
ip_component = pcie_ss\n
Note: To change back to using the AXI Streaming Intel FPGA IP for PCI Express, simply remove this line.
-
Build the FIM using the 470 MHz IOPLL and the PCIe OFSS file that was modified in step 3.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/iopll/iopll_470MHz.ofss,tools/ofss_config/pcie/pcie_host_2link.ofss iseries-dk work_${{ env.ISERIS_DK_MODEL}}\n
-
When the build completes, the output files can be found in $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/output_files.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#45-minimal-fim","title":"4.5 Minimal FIM","text":"In a minimal FIM, the exercisers and Ethernet subsystem are removed from the design, the PF/VF configuration is reduced to either 2PF/0VF or 1PF/1VF, and the AFU PR area is expanded to make use of the available area previously used by the removed components.
There are two pre-provided PCIe configurations that can be used to create minimal FIM.
- 2PF: this PCIe configuration has two physical functions, with the APF/BPF on PF0, and the AFU PR region on PF1.
- 1PF/1VF: This PCIe configuration has a single physical function, with the APF/BPF on PF0, and the AFU PR region on PF0-VF0.
Block diagrams for the PCIe 2x8 and PCIe 1x16 configurations are shown below.
-
PCIe 1x16 Minimal Fim
-
PCIe 2x8 Minimal Fim
The FIM source repository contains OFSS file that can be used to configure the PCIe IP for the minimal FIM.
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2pf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link_1pf_1vf.ofss
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#451-create-a-minimal-fim","title":"4.5.1 Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment for hardware testing. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
The OFS FIM repo provides a PR assignments TCL file which optimizes the PR region for the minimal FIM. Copy the minimal PR assignments TCL file into the pr_assignments.tcl file location for use in the FIM build process.
-
Rename the current pr_assignments.tcl file to pr_assignments_base.tcl for future use
mv $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments_base.tcl\n
-
Copy the pr_assignments_slim.tcl file to pr_assignments.tcl to be used in the current build
cp $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments_slim.tcl $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl\n
-
Compile the FIM with Null HEs compile option, the No HSSI compile option, and the desired PCIe PF/VF configuration.
cd $OFS_ROOTDIR\n
-
PCIe 1x16 2PF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2pf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim\n
-
PCIe 1x16 1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim\n
-
PCIe 2x8 1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2link_1pf_1vf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim\n
-
Review the $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/output_files/ofs_top.fit.rpt utilization report to see the utilization statistics for the Minimal fim. Refer to Appendix A: Resource Utilization Tables Table A-4 for the expected utilization for this Minimal FIM.
-
Copy the resulting $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment, if different than your development environment.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Use lspci to verify that the PCIe changes have been implemented.
sudo lspci -vvv -s b1:00.0 | grep VF\n
Example output for a 1PF/1VF image:
Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 1, stride: 1, Device ID: bcce\nVF Migration: offset: 00000000, BIR: 0\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#46-migrating-to-a-different-agilex-device-number","title":"4.6 Migrating to a Different Agilex Device Number","text":"The following instructions enable a user to change the device part number of the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). Please be aware that this release works with Agilex\u00ae 7 FPGA devices that have F-tile for PCIe and Ethernet. Other tiles will take further work.
You may wish to change the device part number for the following reasons
- Migrate to same device package but with a different density
- Migrate to a different package and with a different or same density
The default device for the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) is AGIB027R29A1E2VR3
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"Perform the following steps to migrate to a different Agilex Device. In this example, we will migrate from the default AGIB027R29A1E2VR3 device to AGIB027R31A1E2VB. The package will change from R29A to R31A.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/iseries-dk_base.ofss file to use AGIB027R31A1E2VB.
[ip]\ntype = ofs\n\n[settings]\nplatform = n6001\nfamily = agilex\nfim = base_x16\npart = AGIB027R31A1E2VB device_id = 6001\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qsf file to use AGIB027R31A1E2VB.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY \"Agilex 7\"\nset_global_assignment -name DEVICE AGIB027R31A1E2VB
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_pr_afu.qsf file to use AGIB027R31A1E2VB.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex 7\nset_global_assignment -name DEVICE AGIB027R31A1E2VB
-
If the device you are migrating to uses the same package and pinout, you do not need to modify the pinout constraints. In this example, because we are migrating from package R29A to R31A, we need to modify the pinout to match the new device. If you would like Quartus to attempt to pin out the design automatically, you may remove all pin assignments. Typically, you will still need to set certain pins manually in order to guide Quartus for a successful compile (e.g. transceiver reference clocks).
-
Commment out all pin assignments in the following files: * $OFS_ROOTDIR/syn/board/iseries-dk/setup/emif_loc.tcl * $OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl
-
Identify the pins in the design that will be constrained. In this example we will manually constrain the QSFP reference clock and the PCIe reference clock to help guide the fitter. The Device Migration Pinout Table shows the pin assignments that will be set, along with the pin number for both the old R24C package and the new R31C package.
Net Name R29A Pin Name R31A Pin Name AGI 027 R29A Pin # AGI 027 R31A Pin # qsfp_ref_clk REFCLK_FGTL12A_Q2_RX_CH4p REFCLK_FGTL12C_Q2_RX_CH4p JD74 AM57 PCIE_REFCLK0 REFCLK_GXRL14C_CH0p REFCLK_GXRL14A_CH0p DR68 BU56 PCIE_REFCLK1 REFCLK_GXRL14C_CH1p REFCLK_GXRL14A_CH1p CU68 BP57 -
Re-pin the pins identified in the previous step in the $OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl file for the new pinout for the AGF 027 R31C package.
set_location_assignment PIN_AM57 -to qsfp_ref_clk\n\nset_location_assignment PIN_BU56 -to PCIE_REFCLK0\nset_location_assignment PIN_BP57 -to PCIE_REFCLK1\n
-
Uncomment the instance assignments related to he QSFP and PCIe reference clocks in the $OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl file.
set_instance_assignment -name IO_STANDARD \"CURRENT MODE LOGIC (CML)\" -to qsfp_ref_clk\n\nset_instance_assignment -name IO_STANDARD HCSL -to PCIE_REFCLK0\nset_instance_assignment -name IO_STANDARD HCSL -to PCIE_REFCLK1\n
-
Compile a flat design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh ${{ env.ISERIS_DK_MODEL }}:flat work_${{ env.ISERIS_DK_MODEL }}\n
-
Verify that the build completes successfuly. If the compilation fails with errors relating to the pinout, review the error messages and modify the pinout accordingly. If there are timing violations, try building with a different seed. Refer to the Change the Compilation Seed section for instructions on changing the build seed.
-
When you are satisfied with the pinout, preserve it by hard-coding the desired pinout back to the followig files:
$OFS_ROOTDIR/syn/board/iseries-dk/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl
-
When you are ready to re-incorporate PR into the design, modify the PR region to be compatible with the new device. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM. This section provides an example walkthrough for modifiying the Memory-SS.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#471-walkthrough-modify-the-memory-sub-system-using-ip-presets-with-ofss","title":"4.7.1 Walkthrough: Modify the Memory Sub-System Using IP Presets With OFSS","text":"This walkthrough will go through the flow of modifying the Memory-SS in the OFS FIM. In this example, we will enable ECC on memory channels 0-3.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script to create a work directory with the Memory-SS configured for the iseries-dk
./ofs-common/scripts/common/syn/build_top.sh --stage setup iseries-dk work_iseries-dk\n
-
Open the Memory Subsystem mem_ss.ip in IP Parameter Editor
qsys-edit $OFS_ROOTDIR/work_iseries-dk/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
The Memory Subsystem IP will open in IP Parameter Editor. Click Dive Into Packaged Subsystem.
-
The Platform Designer mem_ss view opens. All of the EMIFs are shown in the Filter window.
-
Click each EMIF 0 through 3 and perform the following actions.
-
In the Parameters window, click the Memory tab and change the DQ width to 72.
-
In the Parameters window, click the Controller tab. Scroll down and check the box for Enable Error Detection and Correction Logic with ECC.
-
Once Step 6 has been done for each EMIF 0-3, click File -> Save. Close the Platform Designer window.
-
In the IP Parameter Editor Presets window, click New to create an IP Presets file.
-
In the New Preset window, set the Name for the preset to iseries-dk.
-
Click the ... button to select the location for the Preset file.
-
In the Save As window, change the save location to $OFS_ROOTDIR/ipss/mem/qip/presets and change the File Name to mem_presets.qprs to overwrite the existing presets file. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close the IP Parameter Editor. You do not need to generate or save the IP.
-
Edit the $OFS_ROOTDIR/syn/board/iseries-dk/setup/emif_loc.tcl file to add pin assignments for the new signals supporting ECC on Channels 0-3.
# CH0 DQS8\nset_location_assignment PIN_BC29 -to ddr4_mem[0].dbi_n[8]\nset_location_assignment PIN_AT30 -to ddr4_mem[0].dqs_n[8]\nset_location_assignment PIN_AV29 -to ddr4_mem[0].dqs[8]\nset_location_assignment PIN_AT28 -to ddr4_mem[0].dq[64]\nset_location_assignment PIN_AV27 -to ddr4_mem[0].dq[66]\nset_location_assignment PIN_AT32 -to ddr4_mem[0].dq[65]\nset_location_assignment PIN_BC31 -to ddr4_mem[0].dq[67]\nset_location_assignment PIN_BF32 -to ddr4_mem[0].dq[68]\nset_location_assignment PIN_AV31 -to ddr4_mem[0].dq[69]\nset_location_assignment PIN_BC27 -to ddr4_mem[0].dq[70]\nset_location_assignment PIN_BF28 -to ddr4_mem[0].dq[71]\n# CH1 DQS8\nset_location_assignment PIN_BC59 -to ddr4_mem[1].dbi_n[8]\nset_location_assignment PIN_AT60 -to ddr4_mem[1].dqs_n[8]\nset_location_assignment PIN_AV59 -to ddr4_mem[1].dqs[8]\nset_location_assignment PIN_BC61 -to ddr4_mem[1].dq[64]\nset_location_assignment PIN_AT62 -to ddr4_mem[1].dq[65]\nset_location_assignment PIN_AV61 -to ddr4_mem[1].dq[66]\nset_location_assignment PIN_BC57 -to ddr4_mem[1].dq[67]\nset_location_assignment PIN_BF62 -to ddr4_mem[1].dq[68]\nset_location_assignment PIN_AT58 -to ddr4_mem[1].dq[69]\nset_location_assignment PIN_BF58 -to ddr4_mem[1].dq[70]\nset_location_assignment PIN_AV57 -to ddr4_mem[1].dq[71]\n# CH2 DQS8\nset_location_assignment PIN_MD30 -to ddr4_mem[2].dbi_n[8]\nset_location_assignment PIN_MK31 -to ddr4_mem[2].dqs_n[8]\nset_location_assignment PIN_MH30 -to ddr4_mem[2].dqs[8]\nset_location_assignment PIN_MD32 -to ddr4_mem[2].dq[64]\nset_location_assignment PIN_MH32 -to ddr4_mem[2].dq[65]\nset_location_assignment PIN_MK33 -to ddr4_mem[2].dq[66]\nset_location_assignment PIN_MC33 -to ddr4_mem[2].dq[67]\nset_location_assignment PIN_MD28 -to ddr4_mem[2].dq[68]\nset_location_assignment PIN_MH28 -to ddr4_mem[2].dq[69]\nset_location_assignment PIN_MK29 -to ddr4_mem[2].dq[70]\nset_location_assignment PIN_MC29 -to ddr4_mem[2].dq[71]\n# CH3 DQS8\nset_location_assignment PIN_MD42 -to ddr4_mem[3].dbi_n[8]\nset_location_assignment PIN_MK43 -to ddr4_mem[3].dqs_n[8]\nset_location_assignment PIN_MH42 -to ddr4_mem[3].dqs[8]\nset_location_assignment PIN_MH40 -to ddr4_mem[3].dq[64]\nset_location_assignment PIN_MD40 -to ddr4_mem[3].dq[65]\nset_location_assignment PIN_MC41 -to ddr4_mem[3].dq[66]\nset_location_assignment PIN_MK41 -to ddr4_mem[3].dq[67]\nset_location_assignment PIN_MH44 -to ddr4_mem[3].dq[68]\nset_location_assignment PIN_MK45 -to ddr4_mem[3].dq[69]\nset_location_assignment PIN_MD44 -to ddr4_mem[3].dq[70]\nset_location_assignment PIN_MC45 -to ddr4_mem[3].dq[71]\n
-
Compile the design.
./ofs-common/scripts/common/syn/build_top.sh -p iseries-dk work_iseries-dk_mem_ecc_preset\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#472-walkthrough-add-or-remove-the-memory-sub-system","title":"4.7.2 Walkthrough: Add or remove the Memory Sub-System","text":"The Memory Sub-System can be added or removed to the FIM design using the INCLUDE_DDR4 macro defined in the ofs_top.qsf file. The Memory-SS is enabled by default in the iseries-dk.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the ofs_top.qsf file to either enable or disable the INCLUDE_DDR4 macro. Comment out this macro assignemnt if you wish to remove the Memory-SS.
-
Compile the design. Refer to the Compile OFS FIM section for step-by-step instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System.
Note: The default HSSI-SS configuration for the iseries-dk is 2x4x25GbE.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#481-walkthrough-modify-the-ethernet-sub-system-to-1x400gbe","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System to 1x400GbE","text":"OFS provides a preconfigured ofss file so the build script produces a FIM with a 1x400GbE Ethernet subsystem set for 400 GAUI-8. You can build this system with the following:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Build the I-Series FIM with 1x400GbE FIM:
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_1x400_ftile.ofss iseries-dk work_iseries-dk_400\n
The resulting FIM with 1x400GbE has the following MAC settings:
Table: 1x400GbE MAC Settings
MAC Setting Value Client Interface Segmented FEC mode IEEE 802.3 RS(544.514) (CL 134) Auto Negotiation and link training Disabled Maximum Frame Size 1518 You can change the MAC settings by opening the Ethernet Subsystem IP in IP Parameter Editor, update the setting and then save the update as a new preset. You will then edit the ofss_config/hssi/hssi_1x400_ftile.ofss to use the new preset.
The following steps describe the steps to change the 400 GbE MAC frame size to 9600 bytes. Note, the 2x200 and 8x25 GbE MAC implementations can be changed using this process.
-
Invoke IP Parameter editor to make changes to the Ethernet Subsystem IP.
cd $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss\nqsys-edit hssi_ss.ip --quartus-project=$OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
-
In IP Parameter editor, load the 400g-fseries-dk preset by selecting and then click Apply
-
In the Device 0 Configuration tab, go to the F-Tile IP Configuration tab and scroll down to P8 MAC Options - P8 Basic and change the TX and RX maximum framesize to 9600.
-
Click New in the Presets panel and in the New Preset pop up window, Name the new preset 400g-fseries-dk-9600 and in File enter $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets/400g-fseries-dk-9600.qprs and click Save
-
Close IP Parameter Editor and do not save changes to hssi_ss.ip. The new preset captured the changes and this new preset will be used in the following updates to re-generate the Ethernet IP subsystem with the updated frame size.
-
Create a new ofss file for the new preset.
cp $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_1x400_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_1x400_ftile_9600.ofss\n
-
Edit $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_1x400_ftile_9600.ofss to use the new 400g-fseries-dk-9600 preset as listed below:
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 1\ndata_rate = 400GAUI-8\npreset = 400g-fseries-dk-9600\n
-
Build the FIM with 9600 byte frame size by using the new hssi_1x400_ftile_9600.ofss file
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_1x400_ftile_9600.ofss iseries-dk work_iseries-dk_400_9600\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the iseries-dk is done by programming a SOF image to the FPGA the embedded USvia JTAG using Quartus Programer.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"The Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) has an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG.
Perform the following steps to establish a JTAG connection with the iseries-dk.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
Steps:
-
Refer to the following figure showing the location of the on-board Intel FPGA Download Cable II micro USB connector.
-
Verify all switches are set to default as defined in Agilex\u00ae 7 FPGA I-Series Development Kit User Guide.
-
Connect a Micro-USB to USB-A cable between the front panel J8 micro USB port and either the deployment server or an external computer that has Quartus Prime Pro Programming tools installed.
-
Use the jtagconfig tool to check that the JTAG chain contains the AGIB027R29A1E2VR3 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
1) AGI FPGA Development Kit [1-13]\n034BB0DD AGIB027R29A(.|R2|R3)/..\n 020D10DD VTAP10\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the iseries-dk. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) is connected via JTAG. The Quartus programmer version must be the same as the version of Quartus used to build the design.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device 84:00.0 unplug\n
-
Switch to the machine with JTAG connection to the iseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
Note: the Quartus programmer version must be the same as the version of Quartus used to build the design.
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the iseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGIB027R29A1E2VR3 device. The JTAG chain should show the device.
-
Right click the AGIB027R29A1E2VR3 row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGIB027R29A1E2VR3 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device 84:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#appendix","title":"Appendix","text":""},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#appendix-a-resource-utilization-tables","title":"Appendix A: Resource Utilization Tables","text":"Table A-1 Default FIM Resource Utilization with PR (PCIe 1x16)
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 261647.7 28.66 821 6.19 PCIE_RST_CTRL.rst_ctrl 465.6 0.05 0 0.0 afu_top 177715.5 19.47 393 2.96 auto_fab_0 1278.7 0.14 8 0.06 bpf 732.3 0.08 0 0.0 fme_top 625.1 0.07 6 0.05 hssi_wrapper 15867.4 1.74 80 0.6 local_mem_wrapper 10712.7 1.17 60 0.45 ofs_top_auto_tiles 7619.0 0.83 10 0.08 pcie_wrapper 44687.6 4.9 256 1.93 pmci_dummy_csr 682.6 0.07 0 0.0 qsfp_0 629.3 0.07 4 0.03 qsfp_1 629.0 0.07 4 0.03 sys_pll 0.5 0.0 0 0.0 Table A-2 Default FIM Resource Utilization with PR (PCIe 2x8)
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 239719.6 26.26 840 6.33 PCIE_RST_CTRL.rst_ctrl 458.3 0.05 0 0.0 PCIE_RST_CTRL.rst_ctrl 79.0 0.01 0 0.0 afu_top 157249.2 17.23 309 2.33 auto_fab_0 1288.1 0.14 8 0.06 bpf 703.5 0.08 0 0.0 fme_top 626.6 0.07 6 0.05 hssi_wrapper 16164.1 1.77 80 0.6 local_mem_wrapper 10901.9 1.19 60 0.45 ofs_top_auto_tiles 7563.1 0.83 10 0.08 pcie_wrapper 42755.4 4.68 359 2.7 pmci_dummy_csr 676.5 0.07 0 0.0 qsfp_0 627.2 0.07 4 0.03 qsfp_1 624.0 0.07 4 0.03 sys_pll 0.5 0.0 0 0.0 Table A-3 Minimal FIM Resource Utilization (PCIe 2x8, 2PF)
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 122157.7 13.38 598 4.51 PCIE_RST_CTRL.rst_ctrl 366.5 0.04 0 0.0 PCIE_RST_CTRL.rst_ctrl 65.4 0.01 0 0.0 afu_top 62450.5 6.84 190 1.43 auto_fab_0 1273.8 0.14 8 0.06 bpf 835.0 0.09 0 0.0 fme_top 614.6 0.07 6 0.05 hssi_dummy_csr 704.7 0.08 0 0.0 local_mem_wrapper 11229.5 1.23 60 0.45 pcie_wrapper 42541.4 4.66 334 2.52 pmci_dummy_csr 688.1 0.08 0 0.0 qsfp0_dummy_csr 697.4 0.08 0 0.0 qsfp1_dummy_csr 689.8 0.08 0 0.0 sys_pll 0.5 0.0 0 0.0"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/","title":"Getting Started Guide: Open FPGA Stack for Intel Agilex 7 FPGAs Targeting the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)","text":"Last updated: November 19, 2024
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in evaluating the 2024.2-1 version of the PCIe Attach release targeting the I-Series Development Kit. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Load and verify firmware targeting the FIM and AFU regions of the Agilex FPGA
- Verify full stack functionality offered by the PCIe Attach OFS solution
- Learn where to find additional information on other PCIe Attach ingredients
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell targeting Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). This platform is a Development Kit intended to be used as a starting point for evaluation and development.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#13-references-and-versions","title":"1.3 References and Versions","text":""},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-2-software-and-component-version-summary-for-ofs-pcie-attach-targeting-the-i-series-development-kit","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach targeting the I-Series Development Kit","text":"The OFS 2024.2-1 PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are used to manually build the Shell and the AFU portion of any potential workloads. Use this section as a general reference for the versions which compose this release. Specific instructions on building the FIM or AFU are discussed in their respective documents.
Component Version Quartus https://www.intel.com/content/www/us/en/software-kit/794624/intel-quartus-prime-pro-edition-design-software-version-24-1-for-linux.html, patches: 0.18, 0.26 Host Operating System https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.10/x86_64/product-software OneAPI-ASP https://github.com/OFS/oneapi-asp/releases/tag/ofs-2024.2-1, patches: 0.02 OFS Platform AFU BBB https://github.com/OFS/ofs-platform-afu-bbb/releases/tag/ofs-2024.2-1 OFS FIM Common Resources https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.2-1 AFU Examples https://github.com/OFS/examples-afu/releases/tag/ofs-2024.2-1 OPAE-SIM https://github.com/OPAE/opae-sim"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-3-programmable-firmware-version-summary-for-ofs-pcie-attach-targeting-the-i-series-development-kit","title":"Table 3: Programmable Firmware Version Summary for OFS PCIe Attach Targeting the I-Series Development Kit","text":"OFS releases include pre-built binaries for the FPGA, OPAE SDK and Linux DFL which can be programmed out-of-box (OOB) and include known identifiers shown below. Installation of artifacts provided with this release will be discussed in their relevant sections.
Component Version Link FIM (shell) Pr Interface ID: b7855572-6ca9-58b8-bd11-44e1f1ab329f https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Host OPAE SDK https://github.com/OPAE/opae-sdk, tag: 2.13.0-3 https://github.com/OFS/opae-sdk/releases/tag/2.13.0-3 Host Linux Backport DFL Drivers https://github.com/OPAE/linux-dfl, tag: intel-1.11.0-2 https://github.com/OFS/linux-dfl-backport/releases/tag/intel-1.11.0-2"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-4-hardware-bkc-for-ofs-pcie-attach","title":"Table 4: Hardware BKC for OFS PCIe Attach","text":"The following table highlights the hardware which composes the Best Known Configuration (BKC) for the OFS 2024.2-1 PCIe Attach release. The Intel FPGA Download Cable II is not required when using the I Series Dev Kit, as the device has an on-board blaster.
Component Link Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/agilex/agi027.html (optional) Intel FPGA Download Cable II https://www.intel.com/content/www/us/en/products/sku/215664/intel-fpga-download-cable-ii/specifications.html"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#14-board-installation-and-server-settings","title":"1.4 Board Installation and Server Settings","text":"Instructions detailing the board installation guidelines for an I-Series Dev Kit including server BIOS settings and regulatory information can be found in the [Board Installation Guide: OFS for Agilex\u00ae 7 PCIe Attach Development Kits]. This document also covers the installation of a JTAG cable, which is required for shell programming.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#15-reference-documents","title":"1.5 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.1-1/.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#16-i-series-development-kit-jtag-driver-setup","title":"1.6 I-Series Development Kit JTAG Driver Setup","text":"A specific JTAG driver needs to be installed on the host OS. Follow the instructions under the driver setup for Red Hat 5+ on Intel\u00ae FPGA Download Cable (formerly USB-Blaster) Driver for Linux*.
View the JTAG Chain after installing the proper driver and Quartus Programmer.
cd ~/intelFPGA_pro/quartus/bin\n./jtagconfig -D\n1) AGI FPGA Development Kit [1-13]\n(JTAG Server Version 23.3.0 Build 104 09/20/2023 SC Pro Edition)\n034BB0DD AGIB027R29A(.|R2|R3)/.. (IR=10)\n020D10DD VTAP10 (IR=10)\nDesign hash 27AA3E0B7CE0A5B9F366\n + Node 08586E00 (110:11) #0\n+ Node 0C006E00 JTAG UART #0\n+ Node 19104600 Nios II #0\n+ Node 30006E02 Signal Tap #2\n+ Node 30006E01 Signal Tap #1\n+ Node 30006E00 Signal Tap #0\nCaptured DR after reset = (0069761BB020D10DD) [65]\nCaptured IR after reset = (000D55) [21]\nCaptured Bypass after reset = (2) [3]\nCaptured Bypass chain = (0) [3]\nJTAG clock speed auto-adjustment is enabled. To disable, set JtagClockAutoAdjust parameter to 0\nJTAG clock speed 24 MHz\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#17-upgrading-the-i-series-development-kit-fim-via-jtag","title":"1.7 Upgrading the I-Series Development Kit FIM via JTAG","text":"Intel provides a pre-built FIM that can be used out-of-box for platform bring-up. This shell design is available on the OFS 2024.2-1 Release Page. After programming the shell and installing both the OPAE SDK and Backport Linux DFL kernel drivers as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach, you can confirm the correct FIM has been configured by checking the output of fpgainfo fme against the following table:
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-5-fim-version","title":"Table 5: FIM Version","text":"Identifier Value Pr Interface ID b7855572-6ca9-58b8-bd11-44e1f1ab329f Bitstream ID 360571656856467345 -
Download and unpack the artifacts from the 2024.2-1 release page. The file ofs_top_hps.sof is the base OFS FIM file. This file is loaded into the FPGA using the development kit built in USB Blaster. Please be aware this FPGA is not loaded into non-volatile storage, therefore if the server is power cycled, you will need to reload the FPGA .sof file.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/iseries-dk-images_ofs-2024-2-1.tar.gz\ntar xf iseries-dk-images.tar.gz\ncd iseries-dk-images/\n
-
Remove the card from the PCIe bus to prevent a surprise link down. This step is not necessary if no image is currently loaded.
sudo pci_device <PCIe BDF> unplug\n
-
Start the Quartus Prime Programmer GUI interface, quartus_pgmw &, located in the bin directory of your Quartus installation. Select \"Hardware Setup\", double click the AGI FPGA Development Kit hardware item and change the hardware frequency to 16MHz.
-
On the home screen select \"Auto Detect\" and accept the default device.
-
Left click on the line for device AGIB027R29A (the Agilex device) and hit \"Change File\". Load ofs_top.sof from the artifacts directory and check \"Program / Configure\". Hit start.
-
Re-add the card to the PCIe bus. This step is not necessary if no image was loaded beforehand.
sudo pci_device <BDF> plug\n
-
If this is the first time you've loaded an image into the board, you will need to restart (warm boot) the server (not power cycle / cold boot).
-
Verify the PR Interface ID for your image matches expectation. When loading a FIM from the pre-compiled binary included in the artifacts archive, this ID will match the one listed in Table 4.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
OFS is a hardware and software infrastructure that provides an efficient approach to developing a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM), or shell of the FPGA provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The OFS architecture for Intel Agilex 7 FPGA provides modularity, configurability, and scalability. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem - a hierarchical design that targets the P-tile PCIe hard IP and is configured to support bifurcated Gen 5 speeds
- Ethernet Subsystem - provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
- Memory Subsystem - 2 x 8 GB DDR4 DIMMs, supporting 2666 MHz speeds, 64-bit width (no ECC)
- Reset Controller
- FPGA Management Engine - Provides a way to manage the platform and enable acceleration functions on the platform.
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration. Each feature of the FME exposes itself to the kernel-level OFS drivers on the host through a Device Feature Header (DFH) register that is placed at the beginning of Control Status Register (CSR) space. Only one PCIe link can access the FME register space in a multi-host channel design architecture at a time.
Note: For more information on the FIM and its external connections, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required to support partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Like the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your entire AFU resides in a partial reconfiguration region of the FPGA.
- The AFU is part of the static region and is compiled as a flat design.
- Your AFU contains both static and PR regions.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components: ST2MM module, Virtio LB stub, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), Memory Host Exerciser (HE-MEM), Traffic Generator to memory (HE-MEM-TG), Port Gasket (PRG) and HPS Copy Engine.
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Basic HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
The AFU has the option to consume native packets from the host or interface channels or to instantiate a shim provided by the Platform Interface Manager (PIM) to translate between protocols.
Note: For more information on the Platform Interface Manager and AFU development and testing, refer to the Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#22-ofs-software-overview","title":"2.2 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the [Software Reference Manual: Open FPGA Stack], on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS Backport DFL driver software provides the bottom-most API to FPGA platforms for this release. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on the wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The Backport DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
You can also build and install the software stack yourself from source as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You may choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
Instructions on building and installing the OPAE SDK from source can be found in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#41-opae-tools-overview","title":"4.1 OPAE Tools Overview","text":"The following section offers a brief introduction including expected output values for the utilities included with OPAE. A full explanation of each command with a description of its syntax is available in the opae-sdk GitHub repo.
A list of all tools included in the OPAE SDK release can be found on the OPAE FPGA Tools tab of ofs.github.io.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#411-board-management-with-fpgainfo","title":"4.1.1 Board Management with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files. As this release targets a development kit platform, it does not have access to an on-board BMC. Subcommands that target specific BMC functionality (such as reading temeletry) are not supported for this release.
Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Note: Your Bitstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo. As the I-Series Development Kit does not contain a traditional BMC as used by other OFS products, those lines in fpgainfo's output will not return valid objects. The subcommand fpgainfo bmc will likewise fail to report telemetry data.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : N/A\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#412-updating-with-fpgasupdate","title":"4.1.2 Updating with fpgasupdate","text":"The fpgasupdate tool is used to program AFU workloads into an open slot in a FIM. The fpgasupdate tool only accepts images that have been formatted using PACsign.
As the I-Series Development Kit does not contain a traditional BMC, you do not have access to a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL. Only the programming of a GBS workload is supported for this release.
The process of programming a SOF with a new FIM version is shown in section 1.5 Upgrading the I-Series Development Kit FIM via JTAG
sudo fpgasupdate ofs_pr_afu.gbs <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:42:31.58] [INFO ] updating from file ofs_pr_afu.gbs with size 19928064 [2022-04-14 16:42:31.60] [INFO ] waiting for idle [2022-04-14 16:42:31.60] [INFO ] preparing image file [2022-04-14 16:42:38.61] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] [2022-04-14 16:42:54.63] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] [2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:49:11.03] [INFO ] Secure update OK [2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#413-verify-fme-interrupts-with-hello_events","title":"4.1.3 Verify FME Interrupts with hello_events","text":"The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors.
Sample output from sudo hello_events.
sudo hello_events\nWaiting for interrupts now...\ninjecting error\nFME Interrupt occurred\nSuccessfully tested Register/Unregister for FME events!\nclearing error\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#414-host-exerciser-modules","title":"4.1.4 Host Exerciser Modules","text":"The reference FIM and unchanged FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. There are three HEMs present in the Intel OFS Reference FIM - HE-LPBK, HE-MEM, and HE-HSSI. These exercisers are tied to three different VFs that must be enabled before they can be used. Execution of these exercisers requires you bind specific VF endpoint to vfio-pci. The host-side software looks for these endpoints to grab the correct FPGA resource.
Refer to the Intel Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for a full description of these modules.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-6-module-pfvf-mappings","title":"Table 6: Module PF/VF Mappings","text":"Module PF/VF ST2MM PF0 HE-MEM PF0-VF0 HE-HSSI PF0-VF1 HE-MEM_TG PF0-VF2 HE-LB Stub PF1-VF0 HE-LB PF2 VirtIO LB Stub PF3 HPS Copy Engine PF4"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#4141-he-mem-he-lb","title":"4.1.4.1 HE-MEM / HE-LB","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: - Latency (AFU to Host memory read) - MMIO latency (Write+Read) - MMIO BW (64B MMIO writes) - BW (Read/Write, Read only, Wr only)
Host Exerciser Loopback Memory (HE-MEM) AFU is used to exercise use of FPGA connected DDR, data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host.
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. When using the I-Series Development Kit SmartNIC Platform, optimal performance requires the exercisers be run at 400 MHz.
Execution of these exercisers requires you to bind specific VF endpoint to vfio-pci. The following commands will bind the correct endpoint for a device with B/D/F 0000:b1:00.0 and run through a basic loopback test.
Note: While running the opae.io init command listed below, the command has failed if no output is present after completion. Double check that Intel VT-D and IOMMU have been enabled in the kernel as discussed in section 3.0 OFS DFL Kernel Drivers.
sudo pci_device 0000:b1:00.0 vf 3\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci Binding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci iommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188 Assigning /dev/vfio/188 to user Changing permissions for /dev/vfio/188 to rw-rw----\n\n$ sudo host_exerciser --clock-mhz 400 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 128 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2166\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 12.103 GB/s\n Test lpbk(1): PASS\n
The following example will run a loopback throughput test using one cache line per request.
sudo pci_device 0000:b1:00.0 vf 3\nsudo opae.io init -d 0000:b1:00.2 user:user\n\nsudo host_exerciser --clock-mhz 400 --mode trput --cls cl_1 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 128 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 256\nHost Exerciser numWrites: 257\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2127\nTotal number of Reads sent: 256\nTotal number of Writes sent: 256\nBandwidth: 12.325 GB/s\n Test lpbk(1): PASS\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#4142-traffic-generator-afu-test-application","title":"4.1.4.2 Traffic Generator AFU Test Application","text":"Beginning in OPAE version 2.0.11-1+ the TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank. The supported arguments for test configuration are:
- Number of test loops: --loops
- Number of read transfers per test loop: -r,--read
- Number of write transfers per test loop: -w,--write
- Burst size of each transfer: -b,--bls
- Address stride between each transfer: --stride
- Target memory TG: -m,--mem-channel
Below are some example commands for how to execute the test application. To run the preconfigured write/read traffic test on channel 0:
mem_tg tg_test\n[2024-06-27 21:07:11.753] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 0:\nTG PASS\nMem Clock Cycles: 117\nDEBUG: wcnt_ 1\nDEBUG: rcnt_ 1\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1\nDEBUG: num_ticks 117\nWrite BW: 0.164103 GB/s\nRead BW: 0.164103 GB/s\n\nThread on channel 0 exited with status 0\n[2024-06-27 21:07:11.754] [tg_test] [info] Test tg_test(1): PASS\n
Target channel 1 with a 1MB single-word write only test for 1000 iterations
mem_tg --loops 1000 -r 0 -w 2000 -m 1 tg_test\n[2024-06-27 21:07:28.601] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 2116468\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 0\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1000\nDEBUG: num_ticks 2116468\nWrite BW: 18.1434 GB/s\nRead BW: 0 GB/s\n\nThread on channel 1 exited with status 0\n[2024-06-27 21:07:28.608] [tg_test] [info] Test tg_test(1): PASS\n
Target channel 2 with 4MB write/read test of max burst length for 10 iterations
mem_tg --loops 10 -r 8 -w 8 --bls 255 -m 2 tg_test\n[2024-06-27 21:07:41.537] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 2:\nTG PASS\nMem Clock Cycles: 43398\nDEBUG: wcnt_ 8\nDEBUG: rcnt_ 8\nDEBUG: bcnt_ 255\nDEBUG: loop_ 10\nDEBUG: num_ticks 43398\nWrite BW: 9.0253 GB/s\nRead BW: 9.0253 GB/s\n\nThread on channel 2 exited with status 0\n[2024-06-27 21:07:41.539] [tg_test] [info] Test tg_test(1): PASS\n
sudo mem_tg --loops 1000 -r 2000 -w 2000 --stride 2 --bls 2 -m 1 tg_test\n[2024-06-27 21:07:54.841] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 8508637\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 2000\nDEBUG: bcnt_ 2\nDEBUG: loop_ 1000\nDEBUG: num_ticks 8508637\nWrite BW: 9.02612 GB/s\nRead BW: 9.02612 GB/s\n\nThread on channel 1 exited with status 0\n[2024-06-27 21:07:54.867] [tg_test] [info] Test tg_test(1): PASS\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#414-he-hssi","title":"4.1.4 HE-HSSI","text":"HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic.
The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode- specific options, and displays the ethernet statistics by invoking ethtool --statistics INTERFACE.
sudo fpgainfo phy b1:00.0\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102025B1C68BA\nBitstream Version : 5.0.1\nPr Interface Id : e5ee3be6-1b34-5c65-923b-84cf652e6b93\nQSFP0 : Not Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
The following example walks through the process of binding the VF corresponding with the HE-HSSI exerciser to vfio-pci, sending traffic, and verifying that traffic was received.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-7-accelerator-pfvf-and-guid-mappings","title":"Table 7: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID I Series Dev Kit base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to user\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to user\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
-
Check Ethernet PHY settings with fpgainfo.
sudo fpgainfo phy -B 0xb1 Intel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102025B1C68BA\nBitstream Version : 5.0.1\nPr Interface Id : e5ee3be6-1b34-5c65-923b-84cf652e6b93\nQSFP0 : Not Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
-
Set loopback mode.
sudo hssiloopback --loopback enable --pcie-address 0000:b1:00.0 args Namespace(loopback='enable', ncsi_ch_sel=None, pcie_address='0000:84:00.0', port=0)\nsbdf: 0000:b1:00.0\nFPGA dev: {'segment': 0, 'bus': 132, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:b1:00.0'}\nargs.hssi_grps[('dfl_dev.3', ['/sys/bus/pci/devices/0000:b1:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.3/uio/uio0'], '0000:b1:00.0', 21)]\nfpga uio dev:dfl_dev.3\n\n--------HSSI INFO START-------\nDFH :0x3000000010003015\nHSSI ID :0x15\nDFHv :0.5\nguidl :0x99a078ad18418b9d\nguidh :0x4118a7cbd9db4a9b\nHSSI version :2.0\nFirmware Version :16\nHSSI num ports :8\nPort0 :25GbE\nPort1 :25GbE\nPort2 :25GbE\nPort3 :25GbE\nPort4 :25GbE\nPort5 :25GbE\nPort6 :25GbE\nPort7 :25GbE\n--------HSSI INFO END-------\n\nhssi loopback enabled to port0\n[DCPsupport@AN-R760-1 ~]$ sudo fpgainfo phy 0xb1\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:b1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102025B1C68BA\nBitstream Version : 5.0.1\nPr Interface Id : 79a90b4b-b308-55d0-961e-a882ef571b2c\nQSFP0 : Not Connected\nQSFP1 : Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE UP\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
-
Send traffic through the 10G AFU.
10G loopback test\nTx/Rx port: 99\nTx port: 0\nRx port: 0\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth:\n\nNo eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.\n0x40000 ETH_AFU_DFH: 0x1000010000001000\n0x40008 ETH_AFU_ID_L: 0xbb370242ac130002\n0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea\n0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000\n0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000\n0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000\n0x40048 AFU_SCRATCHPAD: 0x0000000045324511\n\n0x3c00 number_packets: 0x00000064\n0x3c01 random_length: 0x00000000\n0x3c02 random_payload: 0x00000000\n0x3c03 start: 0x00000000\n0x3c04 stop: 0x00000000\n0x3c05 source_addr0: 0x44332211\n0x3c06 source_addr1: 0x00006655\n0x3c07 dest_addr0: 0xaa998877\n0x3c08 dest_addr1: 0x0000ccbb\n0x3c09 packet_tx_count: 0x00000064\n0x3c0a rnd_seed0: 0x5eed0000\n0x3c0b rnd_seed1: 0x5eed0001\n0x3c0c rnd_seed2: 0x00025eed\n0x3c0d pkt_length: 0x00000040\n0x3cf4 tx_sta_tstamp: 0x02c1a766\n0x3cf5 tx_end_tstamp: 0x02c1abd6\n0x3d00 num_pkt: 0xffffffff\n0x3d01 pkt_good: 0x00000063\n0x3d02 pkt_bad: 0x00000000\n0x3d07 avst_rx_err: 0x00000000\n0x3d0b rx_sta_tstamp: 0x02c1a85d\n0x3d0c rx_end_tstamp: 0x02c1acd2\n0x3e00 mac_loop: 0x00000000\n\nHSSI performance:\n Selected clock frequency : 402.832 MHz\n Latency minimum : 613.159 ns\n Latency maximum : 625.571 ns\n Achieved Tx throughput : 15.8863 GB/s\n Achieved Rx throughput : 15.8167 GB/s\n
The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. hssi_loopback tests both external and internal loopbacks.
The hssistats tool provides the MAC statistics.
"},{"location":"hw/n6001/","title":"Index","text":"Repository folder for Agilex OFS PCIe Attach collateral targeting Intel FPGA SmartNIC N6001-PL.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/","title":"Shell Developer Guide for Open FPGA Stack: Intel\u00ae FPGA SmartNIC N6000-PL / Intel\u00ae FPGA SmartNIC N6001-PL PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a guide for OFS Agilex PCIe Attach developers targeting the Intel\u00ae FPGA SmartNIC N6000-PL and Intel\u00ae FPGA SmartNIC N6001-PL. The following topics are covered in this guide:
- Compiling the OFS Agilex PCIe Attach FIM design
- Simulating the OFS Agilex PCIe Attach design
- Customizing the OFS Agilex PCIe Attach FIM design
- Configuring the FPGA with an OFS Agilex PCIe Attach FIM design
The FIM Development Walkthroughs Table lists all of the walkthroughs provided in this guide. These walkthroughs provide step-by-step instructions for performing different FIM Development tasks.
Table: FIM Development Walkthroughs
Walkthrough Name Category Install Quartus Prime Pro Software Setup Clone FIM Repository Setup Set Development Environment Variables Setup Set Up Development Environment Setup Compile OFS FIM Compilation Manually Generate OFS Out-Of-Tree PR FIM Compilation Change the Compilation Seed Compilation Run Individual Unit Level Simulation Simulation Run Regression Unit Level Simulation Simulation Add a new module to the OFS FIM Customization Modify and run unit tests for a FIM that has a new module Customization Modify and run UVM tests for a FIM that has a new module Customization Hardware test a FIM that has a new module Customization Debug the FIM with Signal Tap Customization Compile the FIM in preparation for designing your AFU Customization Resize the Partial Reconfiguration Region Customization Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Customization Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Customization Create a Minimal FIM Customization Migrate to a Different Agilex Device Number Customization Modify the Memory Sub-System Using IP Presets With OFSS Customization Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Customization Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Customization Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Customization Modify the Ethernet Sub-System Without HSSI OFSS Customization Remove the HPS Customization Set up JTAG Configuration Program the FPGA via JTAG Configuration Program the FPGA via RSU Configuration"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#111-knowledge-pre-requisites","title":"1.1.1 Knowledge Pre-Requisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex\u00ae 7 PCIe Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL).
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition Software.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition Software, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Software Signal Tap Logic Analyzer tool software.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex PCIe Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex PCIe Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new acceleration card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1211-top-level","title":"1.2.1.1 Top Level","text":"The top level block diagram for the OFS Agilex PCIe Attach reference design is shown below.
Figure: OFS Agilex PCIe Attach FIM Top-Level Diagram
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex PCIe Attach design are listed in the Release Capabilities tables below. They describe the capabilities of the n6001 and n6000 hardware as well as the capabilities of the default OFS Agilex PCIe Attach reference designs targeting the n6001 and the n6000.
Table: Intel\u00ae FPGA SmartNIC N6001-PL OFS Release Capabilities
Interface Intel\u00ae FPGA SmartNIC N6001-PL Hardware Capabilities n6001 OFS Agilex PCIe Attach Reference Design Implementation Host Interface PCIe Gen4x16 PCIe Gen4x16 Network Interface 2 x QSFP-28/56 cages 2x4x25GbE | 2x1x100GbE | 2x4x10GbE External Memory 5xDDR4 DIMMs sockets - 40-bits (1 available for HPS) 4xDDR4 - 2400MHz - 4GB (1Gb x 32) - 32-bits - No ECC1xDDR4 - 2400MHz - 1GB (256Mb x 32 with 256 Mb x8 ECC) - 40-bits - With ECC - For HPS Table: Intel\u00ae FPGA SmartNIC N6000-PL OFS Release Capabilities
Interface Intel\u00ae FPGA SmartNIC N6000-PL Hardware Capabilities n6000 OFS Agilex PCIe Attach Reference Design Implementation Host Interface To Agilex 7: PCIe Gen4x8 To E810: PCIe Gen4x8 To Agilex 7: PCIe Gen4x8 To E810: PCIe Gen4x8 Network Interface 2 x QSFP-28/56 cages 2x2x100GbE External Memory 5xDDR4 DIMMs sockets - 40-bits (1 available for HPS) Not enabled by default, but can support: 4xDDR4 - 2400MHz - 4GB (1Gb x 32) - 32-bits - No ECC1xDDR4 - 2400MHz - 1GB (256Mb x 32 with 256 Mb x8 ECC) - 40-bits - With ECC - For HPS"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex PCIe Attach n6001 FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem AXI Streaming IP for PCI Express User Guide 790711 Memory Subsystem Memory Subsystem Intel FPGA IP User Guide for Intel Agilex OFS 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workload in the OFS Agilex PCIe Attach n6001/n6000 FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI n6001: Used to exercise and characterize HSSI interfaces. n6000: Used to pass through data between the E810 to the QSFPs. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex PCIe Attach n6001 FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the host software and AFU and board peripherals. The APF Address Map Table describes the address mapping of the APF, followed by the BPF Address Map Table which describes the address mapping of the BPF.
Table: APF Address Map
Address Size (Bytes) Feature 0x00000\u20130x3FFFF 256K Board Peripherals (See BPF Address Map table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket:4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x80000 \u2013 0x80FFF 4K AFU Error Reporting Table: BPF Address Map
Address Size (Bytes) Feature 0x00000 - 0x0FFFF 64K FME 0x10000 - 0x10FFF 4K PCIe 0x11000 - 0x11FFF 4K Reserved 0x12000 - 0x12FFF 4K QSFP0 0x13000 - 0x13FFF 4K QSFP1 0x14000 - 0x14FFF 4K HSSI 0x15000 - 0x15FFF 4K EMIF 0x20000 - 0x3FFFF 128K PMCI Controller"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customization Examples Table lists the general user flows for OFS Agilex PCIe Attach n6001/n6000 FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customization Examples
Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Modify and run UVM tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Memory Sub-System Using IP Presets With OFSS Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Modify the Ethernet Sub-System Without HSSI OFSS Remove the HPS"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA acceleration card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Please see the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up the environment for deployment machines.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 N/A Quartus\u00ae Prime Pro Edition Software Quartus Prime Pro Version 24.1 for Linux + Patches 0.18, 0.26 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A FIM Source Files ofs-2024.2-1 Section 1.3.2.1"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"The source files for the OFS Agilex PCIe Attach FIM are provided in the following repository: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1.
Some essential directories in the repository are described as follows:
ofs-agx7-pcie-attach\n| syn // Contains files related to synthesis\n| | board // Contains synthesis files for several cards, including the n6001 and n6000\n| | | n6001 // Contains synthesis files for n6001 and n6000\n| | | | setup // Contains setup files, including pin constraints and location constraints\n| | | | syn_top // Contains Quartus project files\n| verification // Contains files for UVM testing\n| ipss // Contains files for IP Sub-Systems\n| | qsfp // Contains source files for QSFP Sub-System\n| | hssi // Contains source files for HSSI Sub-System\n| | pmci // Contains source files for PMCI Sub-System (not used in F-Tile FIM)\n| | pcie // Contains source files for PCIe Sub-System\n| | mem // Contains source files for Memory Sub-System\n| sim // Contains simulation files\n| | unit_test // Contains files for all unit tests\n| | | scripts // Contains script to run regression unit tests\n| license // Contains Quartus patch\n| ofs-common // Contains files which are common across OFS platforms\n| | verification // Contains common UVM files\n| | scripts // Contains common scripts\n| | | common\n| | | | syn // Contains common scripts for synthesis, including build script\n| | | | sim // Contains common scripts for simulation\n| | tools // Contains common tools files\n| | | mk_csr_module // Contains common files for CSR modules\n| | | fabric_generation // Contains common files for APF/BPF fabric generation\n| | | ofss_config // Contains common files for OFSS configuration tool\n| | | | ip_params // Contains default IP parameters for certain Sub-Systems when using OFSS\n| | src // Contains common source files, including host exercisers\n| tools //\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| | | hssi // Contains OFSS files for Ethernet-SS configuraiton\n| | | memory // Contains OFSS files for Memory-SS configuration\n| | | pcie // Contains OFSS files for PCIe-SS configuration\n| | | iopll // Contains OFSS files for IOPLL configuration\n| src // Contains source files for Agilex PCIe Attach FIM\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | includes // Contains source file header files\n| | top // Contains top-level source files, including design top module\n| | afu_top // Contains top-level source files for AFU\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 PCIe Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n
-
Check out the correct tag of the repository
cd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.2-1)\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-agx7-pcie-attach\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.N6001_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.N6001_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 24.1 for Linux with Agilex\u00ae 7 FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 24.1 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
Clone the ofs-agx7-pcie-attach repository. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.18, 0.26. All required patches are provided in the Assets of the OFS FIM Release: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
-
Extract and unzip the patch-agx7-2024-2-1.tar.gz file.
tar -xvzf patch-agx7-2024-2-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-24.1-0.18-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
- Change the Compilation Seed
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This section describes the theory behind FIM compilation.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2121-top-level-ofss-file","title":"2.1.2.1 Top Level OFSS File","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2122-base-ofss-file","title":"2.1.2.2 Base OFSS File","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2123-pcie-ofss-file","title":"2.1.2.3 PCIe OFSS File","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2124-iopll-ofss-file","title":"2.1.2.4 IOPLL OFSS File","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2125-memory-ofss-file","title":"2.1.2.5 Memory OFSS File","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is:
- For N6001:
$OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/n6001/syn_top/output_files - For N6000:
$OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/n6000/syn_top/output_files
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes the images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Description ofs_top[_hps].bin This is an intermediate, raw binary file. This intermediate raw binary file is produced by taking the Quartus generated .sof file, and converting it to *.pof using quartus_pfg, then converting the *.pof to *.hexout using quartus_cpf, and finally converting the *.hexout to *.bin using objcopy. Depending on whether the FPGA design contains an HPS block, a different file will be generated. **ofs_top.bin* - Raw binary image of the FPGA generated if there is no HPS present in the design. ofs_top_hps.bin - Raw binary image of the FPGA generated if there is an HPS present in the design. ofs_top_page1.bin This is the binary of the Factory Image and is the input to PACSign utility to generate ofs_top_page1_unsigned.bin binary image file. This image will carry binary content for the HPS if it is included in the SOF image. ofs_top_page0_factory.bin This is an input file to PACSign to generate ofs_top_page0_unsigned_factory.bin. ofs_top_page0_unsigned_factory.bin This is the unsigned PACSign output generated for the Factory Image. ofs_top_page1_user1.bin This is an input file to PACSign to generate ofs_top_page1_unsigned_user1.bin. This file is created by taking the ofs_top_[hps].bin file and assigning the User1 or appending factory block information. ofs_top_page1_unsigned_user1.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User1 Image. This file is used to load the FPGA flash User1 Image using the fpgasupdate tool. ofs_top_page2_user2.bin This is an input file to PACSign to generate ofs_top_page2_unsigned_user2.bin. This file is created by taking the ofs_top_[hps].bin file and assigning the User2 or appending factory block information. ofs_top_page2_unsigned_user2.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User2 Image. This file is used to load the FPGA flash User2 Image using the fpgasupdate tool. ofs_top_hps.sof If your design contains an Agilex\u00ae 7 FPGA Hard Processor System, then the build assembly process combines the FPGA ofs_top.sof programming file with u-boot-spl-dtb.hex to produce this file."},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <tgt dir> [-f] <build target> <work dir name>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <tgt dir> String Specifies the location of the relocatable PR directory tree to be created. [-f] - If exists, the script will abort unless -f is specified. <build target> n6001 | n6000 | fseries-dk | iseries-dk Specifies the name of the board target. <work dir name> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_hps.sof\n\u2514\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#224-he_null-fim","title":"2.2.4 HE_NULL FIM","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex PCIe Attach FIM for n6001 or n6000:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. The following command would build the default n6001 design:
./ofs-common/scripts/common/syn/build_top.sh n6001 work_n6001\n
The build command options allow for many modifications to the shell design at build time. The following tool is provided to help you conveniently get the build command for a specific shell configuration:
OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command. Note: If no OFSS file is specified, the build script will use the <target>.ofss file by default.
- Once the build script is complete, the build summary should report that the build is complete and passes timing. For example:
***********************************\n***\n*** OFS_PROJECT: n6001\n*** OFS_BOARD: n6001\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#226-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM. This can be automatically done for you if you run the build script with the -p option. This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the n6001 OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
-
N6001
./ofs-common/scripts/common/syn/build_top.sh n6001 work_n6001\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh n6000 work_n6000\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
-
N6001
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_n6001/pr_build_template n6001 work_n6001\n
-
N6000
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_n6000/pr_build_template n6000 work_n6000\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/board/<board_name>/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
For example, for n6001:
vim $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf\n
set_global_assignment -name SEED 1\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#3-fim-simulation","title":"3. FIM Simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6001 | n6000 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional Refer to the Run Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files automatically; refer to the Run Regression Unit Level Simulation section for step-by-step instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#321-walkthrough-run-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Run Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the target design.
-
N6001
./gen_sim_files.sh n6000\n
-
N6000
./gen_sim_files.sh n6000\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Example output:
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"You may use the regression script regress_run.py to run some or all of the unit tests available with a single command. The regression script is in the following location:
$OFS_ROOTDIR/sim/unit_test/scripts/regress_run.py\n
The usage of the regression script is as follows:
regress_run.py [-h] [-l] [-n <num_procs>] [-k <test_package>] [-s <simulator>] [-g] [--ofss <ip_config>] [-b <board_name>] [-e]\n
The Regression Unit Test Script Options table describes the options for the regress_run.py script.
Table: Regression Unit Test Script Options
Field Options Description -h | --help N/A Show the help message and exit -l | --local N/A Run regression locally -n | --n_procs Integer Maximum number of processes/tests to run in parallel when run locally. This has no effect on farm runs. -k | --pack all | fme | he | hssi | list | mem | pmci Test package to run during regression. The \"list\" option will look for a text file named \"list.txt\" in the \"unit_test\" directory for a text list of tests to run (top directory names). The default test package is all. -s | --sim vcs | vcsmx | msim Specifies the simulator used for the regression tests. The default simulator is vcs -g | --gen_sim_files N/A Generate simulation files. This only needs to be done once per repo update. This is the equivalent of running the gen_sim_files.sh script. -o | --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. -b | --board_name n6001 | n6000 | fseries-dk | iseries-dk Specifies the board target -e | --email_list String Specifies email list to send results to multiple recipients The log for each unit test that is run by the regression script is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#331-walkthrough-run-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Run Regression Unit Level Simulation","text":"Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a test list file to only run the unit level simulations that are supported for the target boards.
touch $OFS_ROOTDIR/sim/unit_test/list.txt\n
Copy the following list into the new file. You may remove tests from this list as desired.
./bfm_test/set_params.sh\n./csr_test/set_params.sh\n./dfh_walker/set_params.sh\n./flr/set_params.sh\n./fme_csr_access/set_params.sh\n./fme_csr_directed/set_params.sh\n./he_lb_test/set_params.sh\n./he_mem_lb_test/set_params.sh\n./he_null_test/set_params.sh\n./hssi_csr_test/set_params.sh\n./hssi_kpi_test/set_params.sh\n./hssi_test/set_params.sh\n./indirect_csr/set_params.sh\n./mem_ss_csr_test/set_params.sh\n./mem_ss_rst_test/set_params.sh\n./mem_tg_test/set_params.sh\n./pcie_ats_basic_test/set_params.sh\n./pcie_csr_test/set_params.sh\n./pcie_ss_axis_components/set_params.sh\n./pf_vf_access_test/set_params.sh\n./pmci_csr_test/set_params.sh\n./pmci_mailbox_test/set_params.sh\n./pmci_rd_default_value_test/set_params.sh\n./pmci_ro_mailbox_test/set_params.sh\n./pmci_vdm_b2b_drop_err_scenario_test/set_params.sh\n./pmci_vdm_len_err_scenario_test/set_params.sh\n./pmci_vdm_mctp_mmio_b2b_test/set_params.sh\n./pmci_vdm_multipkt_error_scenario_test/set_params.sh\n./pmci_vdm_multipkt_tlp_err_test/set_params.sh\n./pmci_vdm_tlp_error_scenario_test/set_params.sh\n./pmci_vdm_tx_rx_all_random_lpbk_test/set_params.sh\n./pmci_vdm_tx_rx_all_toggle_test/set_params.sh\n./pmci_vdm_tx_rx_lpbk_test/set_params.sh\n./port_gasket_test/set_params.sh\n./qsfp_test/set_params.sh\n./remote_stp_test/set_params.sh\n./uart_csr/set_params.sh\n
-
Navigate to the unit test scripts directory.
cd $OFS_ROOTDIR/sim/unit_test/scripts\n
-
Run regression test with the your desired options. For example, to simulate with the options to generate simulation files, run locally, use 8 processes, run all tests, use VCS simulator, and target the desired board:
-
N6001
python regress_run.py -g -l -n 8 -k list -s vcs -b n6001\n
-
N6000
python regress_run.py -g -l -n 8 -k list -s vcs -b n6000\n
Note: You may run all available tests by using -k all instead of creating and using -k list, however not all tests are supported depending on the target board.
-
Once all tests are complete, check that the tests have passed.
Example output:
Passing Unit Tests:37/37 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n bfm_test:............................. PASS -- Time Elapsed:0:01:09.697671\n csr_test:............................. PASS -- Time Elapsed:0:01:25.629880\n dfh_walker:........................... PASS -- Time Elapsed:0:01:10.426995\n flr:.................................. PASS -- Time Elapsed:0:01:09.624240\n fme_csr_access:....................... PASS -- Time Elapsed:0:00:37.512457\n fme_csr_directed:..................... PASS -- Time Elapsed:0:00:45.458114\n he_lb_test:........................... PASS -- Time Elapsed:0:02:07.375008\n he_mem_lb_test:....................... PASS -- Time Elapsed:0:38:52.066694\n he_null_test:......................... PASS -- Time Elapsed:0:01:04.939156\n hssi_csr_test:........................ PASS -- Time Elapsed:0:06:23.029516\n hssi_kpi_test:........................ PASS -- Time Elapsed:2:29:26.521521\n hssi_test:............................ PASS -- Time Elapsed:2:26:59.511113\n indirect_csr:......................... PASS -- Time Elapsed:0:00:53.041156\n mem_ss_csr_test:...................... PASS -- Time Elapsed:0:18:43.020948\n mem_ss_rst_test:...................... PASS -- Time Elapsed:0:37:35.095748\n mem_tg_test:.......................... PASS -- Time Elapsed:0:25:07.194488\n pcie_ats_basic_test:.................. PASS -- Time Elapsed:0:01:08.777231\n pcie_csr_test:........................ PASS -- Time Elapsed:0:01:10.284642\n pcie_ss_axis_components:.............. PASS -- Time Elapsed:0:02:06.266695\n pf_vf_access_test:.................... PASS -- Time Elapsed:0:01:08.948374\n pmci_csr_test:........................ PASS -- Time Elapsed:0:01:16.863665\n pmci_mailbox_test:.................... PASS -- Time Elapsed:0:02:40.877768\n pmci_rd_default_value_test:........... PASS -- Time Elapsed:0:01:13.951152\n pmci_ro_mailbox_test:................. PASS -- Time Elapsed:0:08:40.948176\n pmci_vdm_b2b_drop_err_scenario_test:.. PASS -- Time Elapsed:0:08:53.753526\n pmci_vdm_len_err_scenario_test:....... PASS -- Time Elapsed:0:14:18.816711\n pmci_vdm_mctp_mmio_b2b_test:.......... PASS -- Time Elapsed:0:01:55.465110\n pmci_vdm_multipkt_error_scenario_test: PASS -- Time Elapsed:1:37:57.808918\n pmci_vdm_multipkt_tlp_err_test:....... PASS -- Time Elapsed:0:20:29.416693\n pmci_vdm_tlp_error_scenario_test:..... PASS -- Time Elapsed:0:19:57.978144\n pmci_vdm_tx_rx_all_random_lpbk_test:.. PASS -- Time Elapsed:0:10:54.756589\n pmci_vdm_tx_rx_all_toggle_test:....... PASS -- Time Elapsed:0:07:54.360796\n pmci_vdm_tx_rx_lpbk_test:............. PASS -- Time Elapsed:0:08:13.492743\n port_gasket_test:..................... PASS -- Time Elapsed:0:01:09.992123\n qsfp_test:............................ PASS -- Time Elapsed:0:05:48.597337\n remote_stp_test:...................... PASS -- Time Elapsed:0:01:12.008778\n uart_csr:............................. PASS -- Time Elapsed:0:01:33.400947\nFailing Unit Tests: 0/37 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n Number of Unit test results captured: 37\nNumber of Unit test results passing.: 37\nNumber of Unit test results failing.: 0\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4-fim-customization","title":"4. FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
Table: OFS FIM Customization Examples
Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Modify and run UVM tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Memory Sub-System Using IP Presets With OFSS Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Modify the Ethernet Sub-System Without HSSI OFSS Remove the HPS"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a information for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example will add a hello_fim module to the design. The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the Host. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the n6001 card. The process for these are described in this section.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can be mastered by the Host. The BPF is an interconnect generated by Platform Designer. The Hello FIM BPF Interface Diagram figure shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
Figure: Hello FIM BPF Interface Diagram
The BPF fabric is defined in $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt.
We will add the Hello FIM module to an un-used address space in the MMIO region. The Hello FIM MMIO Address Layout table below shows the MMIO region for the Host with the Hello FIM module added at base address 0x16000.
Table: Hello FIM MMIO Address Layout
Offset Feature CSR set 0x00000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 E-Tile Ethernet Interface 0x15000 EMIF 0x16000 Hello FIM 0x20000 PMCI Controller 0x40000 ST2MM (Streaming to Memory-Mapped) 0x60000 VUART 0x70000 PR Control & Status (Port Gasket) 0x71000 Port CSRs (Port Gasket) 0x72000 User Clock (Port Gasket) 0x74000 Remote SignalTap (Port Gasket) 0x80000 AFU Errors (AFU Interface Handler)"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the Hello FIM CSR table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Table: Hello FIM CSR
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Make hello_fim source directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create hello_fim_top.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_top.sv\n
Copy the following code into hello_fim_top.sv:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : September 2023\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create hello_fim_com.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_com.sv\n
Copy the following code to hello_fim_com.sv:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Create hello_fim_design_files.tcl file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_design_files.tcl\n
Copy the following code into hello_fim_design_files.tcl
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify ofs_top.qsf for the target board to include Hello FIM module
-
N6001
vim $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf\n
-
N6000
vim $OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top.qsf\n
Add the following line:
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify ofs_top_sources.tcl for the target board to include Hello FIM design files
-
N6001
vim $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top_sources.tcl\n
-
N6000
vim $OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top_sources.tcl\n
Add the following line:
############################################\n# Design Files\n############################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_design_files.tcl\n
-
Modify fabric_design_files.tcl to include BPF Hello FIM Slave IP.
vim $OFS_ROOTDIR/src/pd_qsys/fabric/fabric_design_files.tcl\n
Add the following line:
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE $::env(BUILD_ROOT_REL)/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify fabric_width_pkg.sv to add Hello FIM slave information and update EMIF slave next offset.
-
Open fabric_width_pkg.sv
vim $OFS_ROOTDIR/src/includes/fabric_width_pkg.sv\n
-
Add/modify the following lines:
localparam bpf_hello_fim_slv_baseaddress = 'h16000; // New\nlocalparam bpf_hello_fim_slv_address_width = 12; // New\nlocalparam bpf_emif_slv_next_dfh_offset = 'h1000; // Old value: 'hB000\nlocalparam bpf_hello_fim_slv_next_dfh_offset = 'hA000; // New\nlocalparam bpf_hello_fim_slv_eol = 'b0; // New\n
-
Modify top.sv to support the Hello FIM module.
-
Open top.sv
vim $OFS_ROOTDIR/src/top/top.sv\n
-
Add bpf_hello_fim_slv_if to AXI interfaces
// AXI4-lite interfaces\n...\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width), .ARADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width)) bpf_hello_fim_slv_if();\n
-
Add Hello FIM instantiation
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (fabric_width_pkg::bpf_hello_fim_slv_address_width),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`else\ndummy_csr #(\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif\n
-
Add interfaces for Hello FIM slv to bpf instantiation
bpf bpf (\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\n);\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt to add the hello_fim module as a slave to the apf.
# NAME FABRIC BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 18 fme,pcie,pmci,qsfp0,qsfp1,emif,hssi,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute helper script to generate BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\n\nsh gen_fabrics.sh\n
-
Once the script completes, the following new IP is created: $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip.
-
[OPTIONAL] You may verify the BPF changes have been made correctly by opening bpf.qsys to analyze the BPF.
-
Navigate to the $OFS_ROOTDIR/src/pd_qsys/fabric directory.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\n
-
Open the bpf.qsys file.
-
N6001
qsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qpf\n
-
N6000
qsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top.qpf\n
-
Find the bpf_hello_fim_slv instance:
-
Compile the Hello FIM design:
-
Return to the OFS root directory.
cd $OFS_ROOTDIR\n
-
Call the build_top.sh script for the target board.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -p n6001 work_n6001_hello_fim\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -p n6000 work_n6000_hello_fim\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/dfh_walker/test_csr_defs.sv
-
Add HELLO_FIM_IDX entry to t_dfh_idx enumeration.
...\ntypedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX, // New\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nVUART_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n...\n
-
Add HELLO_FIM_DFH to get_dfh_names function.
...\nfunction automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\n...\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\"; // New\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\n...\nreturn dfh_names;\n...\n
-
Add expected DFH value for Hello FIM to the get_dfh_values function.
...\nfunction automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\n...\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_xxxxxx_1012;\ndfh_values[PMCI_DFH_IDX][39:16] = fabric_width_pkg::bpf_pmci_slv_next_dfh_offset;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_xxxxxx_0100; // New\ndfh_values[HELLO_FIM_DFH_IDX][39:16] = fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset; // New\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_xxxxxx_0014;\ndfh_values[ST2MM_DFH_IDX][39:16] = fabric_width_pkg::apf_st2mm_slv_next_dfh_offset;\n...\nreturn dfh_values;\n...\n
-
Generate simulation files:
-
Navigate to the sim directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the gen_sim_files.sh script.
-
N6001
./gen_sim_files.sh n6001\n
-
N6000
./gen_sim_files.sh n6000\n
-
Run DFH Walker Simulation
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\nsh run_sim.sh TEST=dfh_walker\n
-
Verify that the test passes, and that the output shows the Hello FIM in the DFH sequence.
********************************************\n Running TEST(0) : test_dfh_walking\n********************************************\n...\n\nREAD64: address=0x00015000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000010001009\n\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nREAD64: address=0x00016000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x30000000a0000100\n\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000000a0000100)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000200001012\n\nPMCI_DFH\n Address (0x20000)\nDFH value (0x3000000200001012)\n...\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356791250000\nV C S S i m u l a t i o n R e p o r t\nTime: 356791250000 fs\nCPU Time: 61.560 seconds; Data structure size: 47.4Mb\nTue Aug 15 16:29:45 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#414-walkthrough-modify-and-run-uvm-tests-for-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Modify and run UVM tests for a FIM that has a new module","text":"Perform the following steps to modify the UVM simulation files to support the Hello FIM design.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Modify $OFS_ROOTDIR/verification/tests/sequences/dfh_walking_seq.svh
-
Modify the dfh_offset_array to insert the Hello FIM at dfh_offset_array[8].
dfh_offset_array = new[17];\ndfh_offset_array[0] = tb_cfg0.PF0_BAR0; // FME_DFH 0x8000_0000\ndfh_offset_array[1] = dfh_offset_array[0] + 64'h0_1000; // THERM_MNGM_DFH 0x8000_1000\ndfh_offset_array[2] = dfh_offset_array[1] + 64'h0_2000; // GLBL_PERF_DFH 0x8000_3000\ndfh_offset_array[3] = dfh_offset_array[2] + 64'h0_1000; // GLBL_ERROR_DFH 0x8000_4000\ndfh_offset_array[4] = dfh_offset_array[3] + 64'h0_E000; // QSFP0_DFH 0x8001_2000\ndfh_offset_array[5] = dfh_offset_array[4] + 64'h0_1000; // QSFP1_DFH 0x8001_3000\ndfh_offset_array[6] = dfh_offset_array[5] + 64'h0_1000; // HSSI_DFH 0x8001_4000\ndfh_offset_array[7] = dfh_offset_array[6] + 64'h0_1000; // EMIF_DFH 0x8001_5000\ndfh_offset_array[8] = dfh_offset_array[7] + 64'h0_1000; // HELLO_FIM_DFH 0x8001_6000\ndfh_offset_array[9] = dfh_offset_array[8] + 64'h0_A000; // PMCI_DFH 0x8002_0000\ndfh_offset_array[10] = dfh_offset_array[9] + 64'h2_0000; // ST2MM_DFH 0x8004_0000\ndfh_offset_array[11] = dfh_offset_array[10] + 64'h2_0000; // VUART_DFH 0x8006_0000\ndfh_offset_array[12] = dfh_offset_array[11] + 64'h1_0000; // PG_PR_DFH_IDX 0x8007_0000\ndfh_offset_array[13] = dfh_offset_array[12] + 64'h0_1000; // PG_PORT_DFH_IDX 0x8007_1000\ndfh_offset_array[14] = dfh_offset_array[13] + 64'h0_1000; // PG_USER_CLK_DFH_IDX 0x8007_2000\ndfh_offset_array[15] = dfh_offset_array[14] + 64'h0_1000; // PG_REMOTE_STP_DFH_IDX 0x8007_3000\ndfh_offset_array[16] = dfh_offset_array[15] + 64'h0_D000; // PG_AFU_ERR_DFH_IDX 0x8008_0000\n
-
Modify `$OFS_ROOTDIR/verification/tests/sequences/mmio_seq.svh``
-
Add test code related to the Hello FIM. This code will verify the scratchpad register at 0x16030 and read only the register at 0x16038.
// HELLO_FIM_Scratchpad 64 bit access\n`uvm_info(get_name(), $psprintf(\"////Accessing PF0 HELLO_FIM_Scratchpad Register %0h+'h16030////\", tb_cfg0.PF0_BAR0), UVM_LOW)\nassert(std::randomize(wdata));\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h30;\nmmio_write64(.addr_(addr), .data_(wdata));\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h38;\nwdata = 64'h6626_0701_5000_0034;\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\n
Note: uvm_info and uvm_error statements will put a message into log file.
-
Modify $OFS_ROOTDIR/verification/scripts/Makefile_VCS.mk
-
Add INCLUDE_HELLO_FIM define option to enable Hello FIM on UVM
VLOG_OPT += +define+INCLUDE_HELLO_FIM\n
-
Re-generate the UVM files
-
Navigate to the verification scripts directory
cd $VERDIR/scripts\n
-
Clean the output of previous builds
gmake -f Makefile_VCS.mk clean\n
-
Compile the IP files
-
N6001
gmake -f Makefile_VCS.mk cmplib_adp\n
-
N6000
gmake -f Makefile_VCS.mk cmplib_adp n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when compiling IP files for UVM simulation. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Build the RTL and Test Benches
-
N6001
gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1\n
-
N6000
gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1 n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when building the RTL and Test Benches for UVM simulation. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Run the UVM DFH Walker Simulation
-
N6001
gmake -f Makefile_VCS.mk run TESTNAME=dfh_walking_test DUMP=1 DEBUG=1\n
-
N6000
gmake -f Makefile_VCS.mk run TESTNAME=dfh_walking_test DUMP=1 DEBUG=1 n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when running UVM simulations. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Verify the DFH Walker test results
-
The output logs are stored in the $VERDIR/sim/dfh_walking_test directory. The main files to note are described in Table 5-3:
Table 5-3 UVM Output Logs
File Name Description runsim.log A log file of UVM trans.log A log file of transactions on PCIe bus inter.vpd A waveform for VCS -
Run the following command to quickly verify that the Hello FIM module was successfully accessed. In the example below, the message DFH offset Match! Exp = 80016000 Act = 80016000 shows that the Hello FIM module was successfully accessed.
cd $VERDIR/sim/dfh_walking_test\ncat runsim.log | grep \"DFH offset\"\n
Expected output:
UVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 111950000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp = 80000000 Act = 80000000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 112586000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80001000 Act = 80001000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113222000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80003000 Act = 80003000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113858000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80004000 Act = 80004000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 114494000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80012000 Act = 80012000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115147000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80013000 Act = 80013000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115801000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80014000 Act = 80014000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 116628000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80015000 Act = 80015000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117283000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80016000 Act = 80016000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117928000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80080000 Act = 80080000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 118594000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80100000 Act = 80100000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119248000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80130000 Act = 80130000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119854000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80131000 Act = 80131000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 120460000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80132000 Act = 80132000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121065000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80133000 Act = 80133000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121672000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80140000 Act = 80140000\n
-
Run the UVM MMIO Simulation
-
N6001
gmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1 DEBUG=1\n
-
N6000
gmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1 DEBUG=1 n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when running UVM simulations. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Verify the MMIO test results. Run the following commands to show the result of the scratchpad register and Hello FIM ID register. You can see the \"Data match\" message indicating that the registers are successfuly verified.
cd $VERDIR/sim/mmio_test\ncat runsim.log | grep \"Data\" | grep 1603\n
Expected output:
UVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/mmio_seq.svh(68) @ 115466000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016030, data = 880312f9558c00e1\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/mmio_seq.svh(76) @ 116112000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016038, data = 6626070150000034\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#415-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.5 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
[OPTIONAL] In the work directory where the FIM was compiled, determine the PR Interface ID of your design. You can use this value at the end of the walkthrough to verify that the design has been configured to the FPGA.
-
Navigate to the syn_top directory of the target board in the work directory.
cd $OFS_ROOTDIR/<work_directory>/syn/board/<target_board>/syn_top/\n
As an example:
cd $OFS_ROOTDIR/work_n6001/syn/board/n6001/syn_top/\n
-
View the contents of fme-ifc-id.txt.
cat fme-ifc-id.txt\n
Example output:
a791757d-38a6-5159-a7fc-e1a61157a07b\n
-
Switch to your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Run fpgainfo to determine the PCIe B:D.F of your board, and to verify the PR Interface ID matches the ID you found in Step 1.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Initialize opae.io
sudo opae.io init -d <B:D.F>\n
For example:
sudo opae.io init -d 98:00.0\n
-
Run DFH walker. Note the value read back from offset 0x16000 indicates the DFH ID is 0x100 which matches the Hello FIM module.
sudo opae.io walk -d <B:D.F>\n
For example:
sudo opae.io walk -d 98:00.0\n
Example output:
...\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000000a0000100\n dfh: id = 0x100, rev = 0x0, next = 0xa000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000200001012\n dfh: id = 0x12, rev = 0x1, next = 0x20000, eol = 0x0, reserved = 0x0, feature_type = 0x3\n...\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d 98:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d 98:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d 98:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d 0000:15:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d 15:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d 15:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#416-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.6 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Perform the following steps in your development environment:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Synthesize the design using the -e build script option. You may skip this step if you are using a pre-synthesized design.
- Navigate to the OFS root directory
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script for the target board with the -e option.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -e n6001 work_hello_fim_with_stp\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -e n6000 work_hello_fim_with_stp\n
-
Open the design in Quartus. The Compilation Dashboard should show that the Analysis & Elaboration step has completed.
-
N6001
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/n6001/syn_top/ofs_top.qpf\n
-
N6000
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/n6000/syn_top/ofs_top.qpf\n
Note: If a full compile has been done in the work directory you are using, you may need to switch from the ofs_pr_afu view to the ofs_top view.
-
Open Tools -> Signal Tap Logic Analyzer
-
Select the Default template and click Create
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note that the clock selected should correspond to the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the block of interest and open the design instance for review.
-
In the Signal Configuration -> Clock box of the Signal Tap Logic Analyzer window, click the \"...\" button
-
In the Node Finder tool that opens, type clock into the Named field, and select hello_fim_top_inst in the Look in field, then click Search. Select the hello_fim_top_inst|clk signal from the Matching Nodes box and click the \">\" button to move it to the Nodes Found box. Click OK to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM.
-
In the Named field, enter csr_lite_if*. In the Look In field, select hello_fim_top_inst.
-
Select the nets that appear in the Matching Nodes box. Click the > button to add them to the Nodes Found box.
-
Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto_signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, stp_for_hello_fim.
-
Save the newly created Signal Tap file, in the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/n6001/syn_top/ directory, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will automatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/<board_target>/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE stp_for_hello_fim.stp\nset_global_assignment -name SIGNALTAP_FILE stp_for_hello_fim.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -p -k n6001 work_hello_fim_with_stp\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -p -k n6000 work_hello_fim_with_stp\n
-
Ensure that the compile completes successfully and meets timing.
As an example:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: n6001\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the n6001 or n6000. Refer to Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the n6001/n6000 via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/<board_target>/syn_top/output_files/ofs_top.sof image to the n6001 or n6000 FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open the Quartus Signal Tap GUI
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap Logic Analyzer window, select File -> Open, and choose the stp_for_hello_fim.stp file.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable for the n6001/n6000. In the Device: selection box select the Agilex device.
-
If the Agilex Device is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
Create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'stp_for_hello_fim' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, walk the DFH list or peek/poke the Hello FIM registers.
opae.io init -d 0000:98:00.0\nopae.io walk -d 0000:98:00.0\nopae.io release -d 0000:98:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#421-changing-the-afu-in-the-fim-port-gasket","title":"4.2.1 Changing the AFU in the FIM Port Gasket","text":"In the FIM build, the standard AFUs in the port gasket can be replaced with PIM-based AFUs, wrapped by the ofs_plat_afu() top-level module. Before invoking build_top.sh, set the environment variable AFU_WITH_PIM to the path of a sources text file -- the same sources file from examples such as sources.txt in hello_world:
export AFU_WITH_PIM=<path to>/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt\n
When set during the build_top.sh setup stage, the FIM build is configured to load the specified AFU. PR will continue to work, if enabled. The only difference with AFU_WITH_PIM is the change to the AFUs present at power-on. Keep in mind that the default exercisers were chosen to attach to all devices and force reasonable routing decisions during the FIM build across the PR boundary. AFU_WITH_PIM is best used for FIMs that disable PR.
Configuration of the user clock from an AFU's JSON file is ignored in a FIM build.
The AFU_WITH_PIM setting matters only during the setup stage of build_top.sh. It is not consumed by later stages.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#422-compiling-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2 Compiling the FIM in preparation for designing your AFU","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" blocks. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination and order can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4221-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
-
Navigate to the OFS root directory
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the null host exerciser options.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -p n6001:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_n6001\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -p n6000:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_n6000\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to adjust the PR region to fit the additional logic into the static region. Similarly, if you reduce logic in the FIM region, then you can adjust the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions as reported in the following two files
$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/<TARGET_BOARD>/syn_top/output_files/ofs_top.fit.place.rpt\n$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/<TARGET_BOARD>/syn_top/output_files/ofs_top.fit.rpt\n
An example report of the resources usage by partitions defined for the FIM is shown below.
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to customize the resources allocated to the AFU in the PR regions:
-
The OFS flow provides the TCL file $OFS_ROOTDIR/syn/board/<TARGET_BOARD>/setup/pr_assignments.tcl which defines the PR partition where the AFU is allocated.
Example contents of this file for the n6000 are shown as follows:
#####################################################\n# Main PR Partition -- green_region\n#####################################################\nset_instance_assignment -name PARTITION green_region -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name CORE_ONLY_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name RESERVE_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name PARTIAL_RECONFIGURATION_PARTITION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n\nset_instance_assignment -name PLACE_REGION \"X76 Y30 X300 Y195\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name ROUTE_REGION \"X0 Y0 X344 Y212\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n
-
Use Quartus Chip Planner to identify the locations of the resources available within the Agilex\u00ae 7 FPGA chip for placement and routing your AFU. You need to identify a pair of coordinates, the origin (X0, Y0) and top right corner (X1, Y1) of the new or existing rectangles to modify as shown in the following image of the n6000 Chip Planner view.
The coordinates of the top right corner of the lock regions are computed indirectly based on the Width and Height, as follows.
X1 = X0 + Width \nY1 = Y0 + Height\n
-
Make changes to the partial reconfiguraton region in the $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments.tcl file. You can modify the size and location of existing lock regions or add new ones and assign them to the AFU PR partition.
-
Build your FIM and create the PR relocatable build tree using the build_top.sh script. Refer to the Compile OFS FIM section for step-by-step compilation instructions.
-
Analyze the resource utilization report per partition produced after recompiling the FIM.
-
Perform further modification to the PR regions until the results are satisfactory. Make sure timing constraints are met.
For more information on how to optimize the floor plan of your Partial Reconfiguration design refer to the following online documentation.
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3 - Floorplan the Design
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe sub-system IP and PF/VF MUX can be modified either using the OFSS flow or the IP Presets flow. The OFSS flow supports a subset of all available PCIe Sub-system settings, while the IP Preset flow can make any available PCIe Sub-system settings change. With PCIe-SS modifcations related to the PFs and VFs, the PF/VF MUX logic is automatically configured based on the PCIe-SS configuration when using OFSS.
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section.
The sections below describe each modification.
- PCIe Configuration Using OFSS
- PCIe Sub-System configuration Using IP Presets
- [PCIe Sub-System Target IP]
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS PCIe Attach FIM for the n6001 can support up to 8 PFs and 2000 VFs distributed accross all PFs.
For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 (if at least 1 VF present) | 2 (if no VFs present) Max # of PFs 8 Min # of VFs 1 on PF0 (if only 1 PF present) | 0 (if 2PFs present) Max # of VFs 2000 distributed across all PFs Note that as the number of VFs goes up, timing closure can become more difficult.
The scripts provided in ${OFS_ROOTDIR}/ofs-common/tools/ofss_config allows you to easily reconfigure the number of PFs and VFs, bar addresses, vendor/device ID values and more. The PCIe Subsystem IP parameters that can be modified can be seen by reviewing ${OFS_ROOTDIR}/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py
New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe-SS configuration registers contain the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00001771 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00001771"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4431-walkthrough-modify-the-pcie-sub-system-and-pfvf-mux-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS","text":"Perform the following steps to modify the PF/VF MUX configuration.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
View the default PCIe OFSS file contents. For example, the n6001 default PCIe OFSS configuration is defined in the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n
-
Create a new PCIe OFSS file from the existing pcie_host.ofss file
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_pfvf_mod.ofss\n
-
Configure the new pcie_pfvf_mod.ofss with your desired settings. In this example we will add PF5 with 2 VFs.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n[pf5]\nnum_vfs = 2\n
-
Compile the FIM with the new PCIe OFSS file. In this example we are targeting the n6001 board.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_pfvf_mod.ofss n6001 work_n6001_pfvf_mod\n
-
Copy the resulting .bin user 1 image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the .bin image to the n6001 FPGA. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Verify the number of VFs on the newly added PF8. In this example, we defined 2 VFs on PF5 in Step 5.
sudo lspci -vvv -s 98:00.5 | grep VF\n
Example output:
Initial VFs: 2, Total VFs: 2, Number of VFs: 0, Function Dependency Link: 05\nVF offset: 4, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
-
Verify communication with the newly added PF5. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
1. Initialize the driver on PF5
```bash\nsudo opae.io init -d 98:00.5\n```\n
2. Read the GUID for the PF5 CSR stub.
```bash\nsudo opae.io -d 98:00.5 -r 0 peek 0x8\nsudo opae.io -d 98:00.5 -r 0 peek 0x10\n```\n\nExample output:\n\n```bash\n0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n```\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#444-pcie-sub-system-configuration-using-ip-presets","title":"4.4.4 PCIe Sub-System configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings as a starting point.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4441-walkthrough-modify-pcie-sub-system-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Sub-System Configuration Using IP Presets","text":"Perform the following steps to use an IP preset file to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID on PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets. This walkthrough targets the n6001, but similar steps can be applied to the n6000.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment to test the design. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script using your desired OFSS configration to create a working directory for the target board. In this example we will target the n6001.
./ofs-common/scripts/common/syn/build_top.sh --stage setup n6001 work_n6001\n
-
Open the PCIe-SS in the work directory using Quartus Parameter Editor. If you performed Step 3, open the PCIe-SS IP from the work directory; otherwise, open the PCIe-SS IP from the source files.
qsys-edit $OFS_ROOTDIR/work_n6001/ipss/pcie/qip/pcie_ss.ip\n
-
Modify the settings as desired. In this example we are changing Revision ID to 0x2. In the IP Parameter Editor, scroll down and expand the PCIe Interfaces Ports Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab and make these changes.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, n6001-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 6.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = n6001-rev2\n
-
Compile the design with the modified new pcie_host_mod_preset.ofss file.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_mod_preset.ofss n6001 work_n6001_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_n6001_pcie_mod/syn/board/n6001/syn_top/output_files/ofs_top_hps.sof image to your deployment environmenment for JTAG programming, or copy a bin file (e.g. ofs_top_page1_unsigned_user1.bin) for RSU programming.
Note: OPAE FPGA management commands require recognition of the FPGA PCIe Device ID for control. If there is a problem between OPAE management recognition of FPGA PCIe values, then control of the card will be lost. For this reason, you are strongly encouraged to program the FPGA via JTAG to load the test FPGA image. If there is a problem with the SOF image working with your host software that is updated for the new PCIe settings, then you can load a known good SOF file to recover. Once you sure that both the software and FPGA work properly, you can load the FPGA into FPGA flash using the OPAE command fpgasupdate.
-
Switch to your deployment environment.
-
Program the image to the n6001 FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step JTAG programming instructions, or the Program the FPGA via RSU Section for step-by-step RSU programming instructions.
-
Determing the PCIe B:D.F of your board. You may use the OPAE command fpgainfo fme to determine this.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : bdea3c8cdf144c7c227b416cac1358f7\nUser1 Image Info : bbe54f1804c88bbd5a555b072c9f5d80\nUser2 Image Info : bdea3c8cdf144c7c227b416cac1358f7\n
-
Use lspci to verify that the PCIe changes have been implemented.
lspci -nvmms 98:00.0\n
Example output:
Slot: 98:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 1771\nPhySlot: 1\nRev: 02\nNUMANode: 1\nIOMMUGroup: 180\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#445-pcie-sub-system-target-ip","title":"**4.4.5 PCIe Sub-System Target IP **","text":"As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the Intel FPGA IP Subsystem for PCI Express which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. You must also reduce the IOPLL frequency to a maximum of 470 MHz.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4451-walkthrough-change-the-pcie-sub-system-target-ip","title":"4.4.5.1 Walkthrough: Change the PCIe Sub-System Target IP","text":"Perform the following steps to change the target PCIe IP from AXI Streaming Intel FPGA IP for PCI Express to Intel FPGA IP Subsystem for PCI Express.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Add the following line to the [settings] section of the PCIe OFSS file you are using. In this example, it will be added to the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file, which is the default PCIe OFSS file used for n6001.
ip_component = pcie_ss\n
Note: To change back to using the AXI Streaming Intel FPGA IP for PCI Express, simply remove this line.
-
Build the FIM using the 470 MHz IOPLL and the PCIe OFSS file that was modified in step 3.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/iopll/iopll_470MHz.ofss,tools/ofss_config/pcie/pcie_host.ofss n6001 work_n6001\n
-
When the build completes, the output files can be found in $OFS_ROOTDIR/work_n6001/syn/board/n6001/output_files.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#45-minimal-fim","title":"4.5 Minimal FIM","text":"In a minimal FIM, the exercisers and Ethernet subsystem are removed from the design, the PF/VF configuration is reduced to either 2PF/0VF or 1PF/1VF, and the AFU PR area is expanded to make use of the available area previously used by the removed components.
There are two pre-provided PCIe configurations that can be used to create minimal FIM:
- 2PF: this PCIe configuration has two physical functions, with the APF/BPF on PF0, and the AFU PR region on PF1.
- 1PF/1VF: This PCIe configuration has a single physical function, with the APF/BPF on PF0, and the AFU PR region on PF0-VF0.
The FIM source repository contains OFSS file that can be used to configure the PCIe IP for the minimal FIM.
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2pf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#451-walkthrough-create-a-minimal-fim","title":"4.5.1 Walkthrough: Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM. A minimal FIM is one that has the host exercisers and ethernet subsystem removed. This frees up resources that can be used.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment for hardware testing. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
[N6001 ONLY] The OFS FIM repo provides a PR assignments TCL file which optimizes the PR region for the minimal FIM. Copy the minimal PR assignments TCL file into the pr_assignments.tcl file location for use in the FIM build process.
-
Rename the current pr_assignments.tcl file to pr_assignments_base.tcl for future use
mv $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments.tcl $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments_base.tcl\n
-
Copy the pr_assignments_slim.tcl file to pr_assignments.tcl to be used in the current build
cp $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments_slim.tcl $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments.tcl\n
-
Compile the FIM with Null HEs compile option, the No HSSI compile option, and the desired PCIe PF/VF configuration.
cd $OFS_ROOTDIR\n
-
2PF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2pf.ofss n6001:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_n6001_minimal_fim\n
-
1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss n6001:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_n6001_minimal_fim\n
-
Review the $OFS_ROOTDIR/work_n6001_minimal_fim/syn/board/n6001/syn_top/output_files/ofs_top.fit.rpt utilization report to see the utilization statistics for the Minimal fim. Refer to Appendix A: Resource Utilization Tables Table A-4 for the expected utilization for this Minimal FIM.
-
Copy the resulting $OFS_ROOTDIR/work_n6001_minimal_fim/syn/board/n6001/syn_top/output_files/ofs_top_page1_unsigned_user1.bin image to your deployment environmenment.
-
Switch to your deployment environment, if different than your development environment.
-
Program the .bin image to the n6001 FPGA. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Verify the minimal FIM design on the board.
-
For 2PF Minimal FIM:
- Use
opae.io ls to verify that there are two PFs
sudo opae.io ls\n
Example output:
[0000:98:00.1] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
-
Use fpgainfo port to verify the ports.
sudo fpgainfo port\n
Example output:
//****** PORT ******//\nInterface : DFL\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nInterface : UIO\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 00000000-0000-0000-0000-000000000000\n
-
Bind the VFIO driver on PF1.
sudo opae.io init -d 98:00.1\n
Example output:
Unbinding (0x8086,0xbcce) at 0000:98:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:98:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:98:00.1 is 15\n
-
Use fpgainfo port to verify the ports. There should now be a port entry for PF1. The Accelerator GUID for PF1 should be as shown below, which is the GUID for the HE-MEM.
//****** PORT ******//\n Interface : DFL\n Object Id : 0xEE00000\n PCIe s:b:d.f : 0000:98:00.0\n Vendor Id : 0x8086\n Device Id : 0xBCCE\n SubVendor Id : 0x8086\n SubDevice Id : 0x1771\n Socket Id : 0x00\n //****** PORT ******//\n Interface : VFIO\n Object Id : 0x2098000000000000\n PCIe s:b:d.f : 0000:98:00.1\n Vendor Id : 0x8086\n Device Id : 0xBCCE\n SubVendor Id : 0x8086\n SubDevice Id : 0x1771\n Socket Id : 0x01\n Accelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n //****** PORT ******//\n Interface : UIO\n Object Id : 0xED00000\n PCIe s:b:d.f : 0000:98:00.0\n Vendor Id : 0x8086\n Device Id : 0xBCCE\n SubVendor Id : 0x8086\n SubDevice Id : 0x1771\n Socket Id : 0x01\n Accelerator GUID : 00000000-0000-0000-0000-000000000000\n
-
For 1PF/1VF Minimal FIM:
- Use
opae.io ls to verify that there is one PF.
sudo opae.io ls\n
Example output:
[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
-
Use lspci to verify that there is 1 VF on PF0.
sudo lspci -vvv -s 98:00.0 | grep VF\n
Example output:
Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 1, stride: 1, Device ID: bcce\nVF Migration: offset: 00000000, BIR: 0\n
-
Use fpgainfo port to verify the ports.
sudo fpgainfo port\n
Example output:
//****** PORT ******//\nInterface : DFL\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nInterface : UIO\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 00000000-0000-0000-0000-000000000000\n
-
Initialize one virtual function
sudo pci_device 0000:98:00.0 vf 1\n
-
Use opae.io ls to verify the VF has been initialized.
sudo opae.io ls\n
Example output:
[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n[0000:98:00.1] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
-
Bind the VFIO driver on VF0
sudo opae.io init -d 0000:98:00.1\n
-
Use fpgainfo port to verify the ports. There should now be a port entry for VF0. The Accelerator GUID for VF0 should be as shown below, which is the GUID for the HE-MEM.
sudo fpgainfo port\n
Example output:
//****** PORT ******//\nInterface : DFL\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nInterface : VFIO\nObject Id : 0x2098000000000000\nPCIe s:b:d.f : 0000:98:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nInterface : UIO\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 00000000-0000-0000-0000-000000000000\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#46-migrate-to-a-different-agilex-device-number","title":"4.6 Migrate to a Different Agilex Device Number","text":"The following instructions enable a user to change the device part number of the Intel\u00ae FPGA SmartNIC N6001-PL. Please be aware that this release works with Agilex\u00ae 7 FPGA devices that have P tile for PCIe and E tile for Ethernet. Other tiles will take further work.
You may wish to change the device part number for the following reasons
- Migrate to same device package but with a different density
- Migrate to a different package and with a different or same density
The default device for the Intel\u00ae FPGA SmartNIC N6001-PL is AGFB014R24A2E2V
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"Perform the following steps to migrate your design to target a different Agilex device using the OFSS build flow. In this example we will change the device from the default AGFB014R24A2E2V to a new device AGFB022R25A2E2V. This walkthrough will target the n6001, but similar steps can be used for the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/n6001_base.ofss file to use AGFB022R25A2E2V. This is only necessary if you are using the OFSS flow.
[ip]\ntype = ofs\n\n[settings]\nplatform = n6001\nfamily = agilex\nfim = base_x16\npart = AGFB022R25A2E2V\ndevice_id = 6001\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFB022R25A2E2V\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_pr_afu.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFB022R25A2E2V\n
-
Modify the DEVICE field in te $OFS_ROOTDIR/ipss/pmci/pmci_ss.qsf file.
set_global_assignment -name DEVICE AGFB022R25A2E2V\n
-
If you are changing to a device with a different package, you must change the pin assignments in the location files. If you would like Quartus to attempt to pin out the design automatically, you may remove all pin assignments instead. Typically you will be required to set certain pins manually in order to guide Quartus for a successful compile (e.g. transceiver reference clocks).
-
Comment out all of the pin constraints in the following files and attempt to let Quartus pin out the design as much as possibe:
$OFS_ROOTDIR/syn/board/n6001/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/hps_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/pmci_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl
-
Identify the pins you wish to assign prior to compiling. In this example, we will re-pin some of the reference clocks to help guide the fitter. Refer to the Pin-Out Files for Altera FPGAs for the pin list of your device. In this example, the Migration Re-Pin Mapping table below shows the pins we will re-pin in the constraints files.
Table: Migration Re-Pin Mapping
Pin Name FIM Signal Name AGF 014 R24A Pin # AGF 022 R25A Pin # REFCLK_GXER9A_CH0p cr3_cpri_reflclk_clk[0] PIN_AT13 PIN_CE18 REFCLK_GXER9A_CH0n \"cr3_cpri_reflclk_clk[0](n)\" PIN_AP13 PIN_CA18 REFCLK_GXER9A_CH1p cr3_cpri_refclk_clk[1] PIN_AR14 PIN_CC19 REFCLK_GXER9A_CH1n \"cr3_cpri_refclk_clk[1](n)\" PIN_AN14 PIN_BW19 REFCLK_GXER9A_CH2p cr3_cpri_refclk_clk[2] PIN_AJ12 PIN_BL17 REFCLK_GXER9A_CH2n \"cr3_cpri_refclk_clk[2](n)\" PIN_AH11 PIN_BJ15 REFCLK_GXER9A_CH3p qsfp_ref_clk PIN_AK13 PIN_BN18 REFCLK_GXER9A_CH3n \"qsfp_ref_clk(n)\" PIN_AH13 PIN_BJ18 REFCLK_GXER9A_CH4p cr3_cpri_reflclk_clk_184_32m PIN_AJ14 PIN_BL19 REFCLK_GXER9A_CH4n \"cr3_cpri_reflclk_clk_184_32m(n)\" PIN_AL14 PIN_BR19 REFCLK_GXER9A_CH5p cr3_cpri_reflclk_clk_153_6m PIN_AR16 PIN_CC21 REFCLK_GXER9A_CH5n \"cr3_cpri_reflclk_clk_153_6m(n)\" PIN_AN16 PIN_BW21 REFCLK_GXPL10A_CH0n \"PCIE_REFCLK0(n)\" PIN_AH49 PIN_DD56 REFCLK_GXPL10A_CH0p PCIE_REFCLK0 PIN_AJ48 PIN_DF57 REFCLK_GXPL10A_CH2n \"PCIE_REFCLK1(n)\" PIN_AD49 PIN_CT56 REFCLK_GXPL10A_CH2p PCIE_REFCLK1 PIN_AE48 PIN_CV57 -
Re-pin the pins identified in the previous step in the $OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl file for the new pinout.
set_location_assignment PIN_BN18 -to qsfp_ref_clk\nset_location_assignment PIN_BJ18 -to \"qsfp_ref_clk(n)\"\nset_location_assignment PIN_CC19 -to cr3_cpri_refclk_clk[1]\nset_location_assignment PIN_BW19 -to \"cr3_cpri_refclk_clk[1](n)\"\nset_location_assignment PIN_BL17 -to cr3_cpri_refclk_clk[2]\nset_location_assignment PIN_BJ15 -to \"cr3_cpri_refclk_clk[2](n)\"\nset_location_assignment PIN_CE18 -to cr3_cpri_reflclk_clk[0]\nset_location_assignment PIN_CA18 -to \"cr3_cpri_reflclk_clk[0](n)\"\nset_location_assignment PIN_BL19 -to cr3_cpri_reflclk_clk_184_32m\nset_location_assignment PIN_BR19 -to \"cr3_cpri_reflclk_clk_184_32m(n)\"\nset_location_assignment PIN_CC21 -to cr3_cpri_reflclk_clk_153_6m\nset_location_assignment PIN_BW21 -to \"cr3_cpri_reflclk_clk_153_6m(n)\"\nset_location_assignment PIN_DD56 -to \"PCIE_REFCLK0(n)\"\nset_location_assignment PIN_DF57 -to PCIE_REFCLK0\nset_location_assignment PIN_CT56 -to \"PCIE_REFCLK1(n)\"\nset_location_assignment PIN_CV57 -to PCIE_REFCLK1\n
-
Un-comment the instance assignemnts for the transceiver reference clocks defined in $OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl.
set_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=156250000\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=TRUE\" -to qsfp_ref_clk\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=184320000\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=153600000\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=245760000\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=enable_3p3v_tol\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=184320000\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=enable_3p3v_tol\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=153600000\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_reflclk_clk_153_6m\n
-
Compile a flat design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh n6001:flat work_n6001\n
-
Verify that the build completes successfuly. If the compilation fails with errors relating to the pinout, review the error messages and modify the pinout accordingly. If there are timing violations, try building with a different seed. Refer to the Change the Compilation Seed section for instructions on changing the build seed.
***********************************\n***\n*** OFS_PROJECT: n6001\n*** OFS_BOARD: n6001\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 3\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
When you are satisfied with the pinout, preserve it by hard-coding the desired pinout back to the followig files:
$OFS_ROOTDIR/syn/board/n6001/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/hps_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/pmci_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl ```
-
When you are ready to re-incorporate PR into the design, modify the PR region to be compatible with the new device. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM. This section provides an example walkthrough for modifiying the Memory-SS.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#471-walkthrough-modify-the-memory-sub-system-using-ip-presets-with-ofss","title":"4.7.1 Walkthrough: Modify the Memory Sub-System Using IP Presets With OFSS","text":"This walkthrough will go through the flow of modifying the Memory-SS in the OFS FIM. In this example, we will enable ECC on memory channels 0-3. Note that routes for the ECC pins on Channels 0 and 1 are not physiclly present on standard n6001/n6000 board hardware; the purpose of this walkthrough is only to show an example of how to make modifications to the IP. This walkthrough targets the n6001 board, but similar steps can be used for the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Memory Subsystem mem_ss.ip in IP Parameter Editor
qsys-edit $OFS_ROOTDIR/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
The Memory Subsystem IP will open in IP Parameter Editor. Click Dive Into Packaged Subsystem.
-
The Platform Designer mem_ss view opens. All of the EMIFs are shown in the Filter window.
-
Click each EMIF 0 through 3 and perform the following actions.
-
In the Parameters window, click the Memory tab and change the DQ width to 40.
-
In the Parameters window, click the Controller tab.
-
Scroll down and check the box for Enable Error Detection and Correction Logic with ECC.
-
Once Step 6 has been done for each EMIF 0-3, click File -> Save. Close the Platform Designer window.
-
In the IP Parameter Editor Presets window, click New to create an IP Presets file.
-
In the New Preset window, set the Name for the preset. In this case we will name it n6001.
-
Click the ... button to select the location for the Preset file.
-
In the Save As window, change the save location to $OFS_ROOTDIR/ipss/mem/qip/presets and change the File Name to n6001.qprs. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close the IP Parameter Editor. You do not need to generate or save the IP.
-
Edit the $OFS_ROOTDIR/syn/board/n6001/setup/emif_loc.tcl file to add pin assignments for the new signals supporting ECC on Channels 0-3. Note that routes for the ECC pins on Channels 0 and 1 are not physiclly present on a standard n6001 board.
# CH0 DQS4 (ECC)\nset_location_assignment PIN_CG48 -to ddr4_mem[0].dbi_n[4]\nset_location_assignment PIN_CF47 -to ddr4_mem[0].dqs_n[4]\nset_location_assignment PIN_CH47 -to ddr4_mem[0].dqs[4]\nset_location_assignment PIN_CE50 -to ddr4_mem[0].dq[32]\nset_location_assignment PIN_CG50 -to ddr4_mem[0].dq[33]\nset_location_assignment PIN_CF49 -to ddr4_mem[0].dq[34]\nset_location_assignment PIN_CH49 -to ddr4_mem[0].dq[35]\nset_location_assignment PIN_CE46 -to ddr4_mem[0].dq[36]\nset_location_assignment PIN_CG46 -to ddr4_mem[0].dq[37]\nset_location_assignment PIN_CF45 -to ddr4_mem[0].dq[38]\nset_location_assignment PIN_CH45 -to ddr4_mem[0].dq[39]\n# CH1 DQS4 (ECC)\nset_location_assignment PIN_DC34 -to ddr4_mem[1].dbi_n[4]\nset_location_assignment PIN_CY33 -to ddr4_mem[1].dqs_n[4]\nset_location_assignment PIN_DB33 -to ddr4_mem[1].dqs[4]\nset_location_assignment PIN_DA36 -to ddr4_mem[1].dq[32]\nset_location_assignment PIN_DC36 -to ddr4_mem[1].dq[33]\nset_location_assignment PIN_CY35 -to ddr4_mem[1].dq[34]\nset_location_assignment PIN_DB35 -to ddr4_mem[1].dq[35]\nset_location_assignment PIN_DA32 -to ddr4_mem[1].dq[36]\nset_location_assignment PIN_DC32 -to ddr4_mem[1].dq[37]\nset_location_assignment PIN_CY31 -to ddr4_mem[1].dq[38]\nset_location_assignment PIN_DB31 -to ddr4_mem[1].dq[39]\n# CH2 DQS4 (ECC)\nset_location_assignment PIN_G36 -to ddr4_mem[2].dbi_n[4]\nset_location_assignment PIN_H35 -to ddr4_mem[2].dqs_n[4]\nset_location_assignment PIN_F35 -to ddr4_mem[2].dqs[4]\nset_location_assignment PIN_G38 -to ddr4_mem[2].dq[32]\nset_location_assignment PIN_J38 -to ddr4_mem[2].dq[33]\nset_location_assignment PIN_H33 -to ddr4_mem[2].dq[34]\nset_location_assignment PIN_J34 -to ddr4_mem[2].dq[35]\nset_location_assignment PIN_F33 -to ddr4_mem[2].dq[36]\nset_location_assignment PIN_H37 -to ddr4_mem[2].dq[37]\nset_location_assignment PIN_F37 -to ddr4_mem[2].dq[38]\nset_location_assignment PIN_G34 -to ddr4_mem[2].dq[39]\n# CH3 DQS4 (ECC)\nset_location_assignment PIN_L50 -to ddr4_mem[3].dbi_n[4]\nset_location_assignment PIN_P49 -to ddr4_mem[3].dqs_n[4]\nset_location_assignment PIN_M49 -to ddr4_mem[3].dqs[4]\nset_location_assignment PIN_M51 -to ddr4_mem[3].dq[32]\nset_location_assignment PIN_N48 -to ddr4_mem[3].dq[33]\nset_location_assignment PIN_M47 -to ddr4_mem[3].dq[34]\nset_location_assignment PIN_L48 -to ddr4_mem[3].dq[35]\nset_location_assignment PIN_P47 -to ddr4_mem[3].dq[36]\nset_location_assignment PIN_P51 -to ddr4_mem[3].dq[37]\nset_location_assignment PIN_N52 -to ddr4_mem[3].dq[38]\nset_location_assignment PIN_L52 -to ddr4_mem[3].dq[39]\n
-
Compile the design. In this example we are targeting the n6001.
./ofs-common/scripts/common/syn/build_top.sh -p n6001 work_n6001_mem_ecc_preset\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#472-walkthrough-add-or-remove-the-memory-sub-system","title":"4.7.2 Walkthrough: Add or remove the Memory Sub-System","text":"The Memory Sub-System can be added or removed to the FIM design using the INCLUDE_DDR4 macro defined in the ofs_top.qsf file. The Memory-SS is enabled by default in the n6001, and is disabled by default in the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the ofs_top.qsf file to either enable or disable the INCLUDE_DDR4 macro. Comment out this macro assignemnt if you wish to remove the Memory-SS.
Note: The default Memory-SS has connections to the HPS. When enabling the Memory-SS, you must either ensure that the INCLUDE_HPS and INCLUDE_UART macros are also enabled, or you must remove the connections from the Memory-SS to the HPS. Refer to the Remove the HPS section for step-by-step instructions on removing the HPS from the design.
-
Compile the design. Refer to the Compile OFS FIM section for step-by-step instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System. There are three flows you may use to make modifications.
- Modify the Ethernet Sub-System with OFSS supported changes only. These modifications are supported natively by the build script, and are made at run-time of the build script. This flow is useful for users who only need to leverage natively supported HSSI OFSS settings.
- Modify the Ethernet Sub-System with OFSS supported changes, then make additional custom modifications not covered by OFSS. These modifications will be captured in a presets file which can be used in future compiles. This flow is useful for users who whish to leverage pre-made HSSI OFSS settings, but make additional modifications not natively supported by HSSI OFSS.
- Modify the Ethernet Sub-System without HSSI OFSS. These modification will be made directly in the source files.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#481-walkthrough-modify-the-ethernet-sub-system-channels-with-pre-made-hssi-ofss","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS","text":"This walkthrough describes how to use OFSS to configure the Ethernet-SS. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This walkthrough is useful for users who only need to leverage the pre-made, natively supported HSSI OFSS settings.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM using the desired HSSI configuration.
cd $OFS_ROOTDIR\n
-
N6001 - 2x4x25GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_8x25.ofss n6001 work_n6001\n
-
N6001 - 2x4x10GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_8x10.ofss.ofss n6001 work_n6001\n
-
N6001 - 2x1x100GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_2x100.ofss n6001 work_n6001\n
-
N6000 - 4x1x100GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_4x100.ofss n6000 work_n6000\n
-
The resulting FIM will contain the Ethernet-SS configuration specified in Step 3. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#482-walkthrough-add-channels-to-the-ethernet-sub-system-channels-with-custom-hssi-ofss","title":"4.8.2 Walkthrough: Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS","text":"This walkthrough describes how to create an use a custom OFSS file to add channels to the Ethernet-SS and compile a design with a 3x4x10GbE Ethernet-SS configuration. This walkthrough is useful for users who wish to leverage the natively supported HSSI OFSS settings.
Note: The n6000 uses 16 channels in the HSSI-SS by default: 8 to the QSFPs, and 8 to the E810 device. Thus, this walkthrough applies only to the n6001.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a new HSSI OFSS file $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_12x10.ofss with the following contents.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 12\ndata_rate = 10GbE\n
-
Identify the which channels will be added. You may use the E-Tile Channel Placement Tool to aid in your design. In this example we will add the 4 new 10GbE channels to Channels 8-11.
-
Based on your channel selection, identify which pins will be used. Refer to the Pin-Out Files for Altera FPGAs determine the required pins for your device. In this example we are targeting the AGFB014R24A2E2V device. Set the pin assignments in the $OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl file.
set_location_assignment PIN_AV7 -to qsfp_serial[2].rx_p[0]\nset_location_assignment PIN_AW10 -to qsfp_serial[2].rx_p[1]\nset_location_assignment PIN_BB7 -to qsfp_serial[2].rx_p[2]\nset_location_assignment PIN_BC10 -to qsfp_serial[2].rx_p[3]\nset_location_assignment PIN_AV1 -to qsfp_serial[2].tx_p[0]\nset_location_assignment PIN_AW4 -to qsfp_serial[2].tx_p[1]\nset_location_assignment PIN_BB1 -to qsfp_serial[2].tx_p[2]\nset_location_assignment PIN_BC4 -to qsfp_serial[2].tx_p[3]\n
-
Change the number of QSFP ports from 2 to 3 in the $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv file.
localparam NUM_QSFP_PORTS_USED = 3; // Number of QSFP cages on board used by target hssi configuration\n
-
Edit $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/hssi_wrapper.sv so that the QSFP LED signals use NUM_QSFP_PORTS_USED defined in the previous step.
// Speed and activity LEDS\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_green, // Link up in Nx25G or 2x56G or 1x100G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_yellow, // Link up in Nx10G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_green, // Link up and activity seen\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_red // LOS, TX Fault etc\n
-
Compile the design using the OFSS file created previously. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_12x10.ofss n6001:flat work_n6001_12x10\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#483-walkthrough-modify-the-ethernet-sub-system-with-pre-made-hssi-ofss-plus-additional-modifications","title":"4.8.3 Walkthrough: Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications","text":"This walkthrough describes how to use OFSS to first modify the Ethernet-SS, then make additional modifications on top. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This flow is useful for users who whish to leverage pre-made OFSS settings, but make additional modifications not natively supported by OFSS. This walkthorugh targets the n6001, however, similiar steps can be performed on the n6000.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script with the desired HSSI OFSS file to create a work directory whith the desired HSSI configuration. For example, to create a work directory for n6001 with HSSI configuration 8x25 GbE.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss tools/ofss_config/hssi/hssi_8x25.ofss n6001 work_n6001\n
-
Open the Ethernet-SS IP in Quartus Parameter Editor. The IP settings will match te configuration of the OFSS file defined in Step 3. Make any additional modifications in the Parameter Editor.
qsys-edit $OFS_ROOTDIR/work_n6001/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the New... button in the Presets pane of IP Parameter Editor.
-
In the New Preset window, create a unique Name. In this example the name is n6001-hssi-presets.
-
Click the ... button to select where to save the preset file. Give it a name, and save it to $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close out of all Quartus GUIs. You do not need to save or compile the IP.
-
Create a new HSSI OFSS file in the $OFS_ROOTDIR/tools/ofss_config/hssi directory named hssi_preset_n6001.ofss with the following contents. Note that the num_channels and data_rate settings will be overwritten by the contents of the preset file. The preset setting must match the name you selected in Step 7.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 8\ndata_rate = 25GbE\npreset = n6001-hssi-presets\n
-
Compile the design using the new HSSI OFSS file. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_preset_n6001.ofss n6001:flat work_n6001_hssi_preset\n
-
The resulting FIM will contain the Ethernet-SS configuration specified by the presets file. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#484-walkthrough-modify-the-ethernet-sub-system-without-hssi-ofss","title":"4.8.4 Walkthrough: Modify the Ethernet Sub-System Without HSSI OFSS","text":"This walkthrough describes how to modify the Ethernet-SS wihout using OFSS. This flow will edit the Ethernet-SS IP source directly. This walkthrough is useful for users who wish to make all Ethernet-SS modifications manually, without leveraging HSSI OFSS.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Ethernet-SS IP in Quartus Parameter Editor. Make your modifications in the Parameter Editor.
qsys-edit $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the Generate HDL. Save the design if prompted.
-
Compile the design with the nodefault option to ensure that the source files are used during compilation. You may specify OFSS files for any component other than the HSSI if desired.
-
If you are not using any other OFSS files in your compilation flow (i.e. only the source files configuration will be used), use the following command to compile. It is recommended to compile a flat design before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss nodefault n6001:flat work_n6001\n
- If you are using OFSS files for other IP in the design, list them after
nodefault, but ensure that an HSSI OFSS file is not specified. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss nodefault, <pcie_ofss_file>,<memory_ofss_file>,<iopll_ofss_file>,<base_ofss_file> n6001:flat work_n6001\n
-
The resulting FIM will contain the Ethernet-SS configuration contained in the hssi_ss.ip source file.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#49-modifying-the-hps","title":"4.9 Modifying the HPS","text":"This section describes ways to modify the HPS.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#491-walkthrough-remove-the-hps","title":"4.9.1 Walkthrough: Remove the HPS","text":"Perform the following steps to remove the HPS from the FIM design.
Note: The n6000 default FIM has the HPS and Memory-SS disabled by default. This walkthrough targets the n6001.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a Memory Sub-system IP presets file with the connection to the HPS removed.
-
Open the the Memory Sub-System IP
qsys-edit $OFS_ROOTDIR/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
In the IP Parameter Editor window that opens, remove the entries corresponding to the HPS (row #4) in the Memory Interfaces and Application Interfaces sections. To do this, click the row to be removed, then click the minus (-) button.
-
In the Presets pane, click New... to create a new IP presets file.
-
In the New Preset window set the preset name to n6001.
-
Click the ... button to select the save location of the IP presets file. In the Save As window, set the Look In field to the memory IP presets directory $OFS_ROOTDIR/ipss/mem/qip/presets. Set the File Name field to overwrite the existing mem_presets.qprs file. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Edit $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf to comment out the INCLUDE_HPS and INCLUDE_UART macro definitions.
#set_global_assignment -name VERILOG_MACRO \"INCLUDE_HPS\"\n#set_global_assignment -name VERILOG_MACRO \"INCLUDE_UART\"\n
-
Build the FIM.
./ofs-common/scripts/common/syn/build_top.sh -p n6001 work_n6001_no_hps\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#491-walkthrough-add-the-hps","title":"4.9.1 Walkthrough: Add the HPS","text":"Note: The n6001 default FIM has the HPS and Memory-SS enabled by default. This walkthrough targets the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit $OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top.qsf to un-comment the INCLUDE_DDR4, INCLUDE_HPS, and INCLUDE_UART macro definitions.
set_global_assignment -name VERILOG_MACRO \"INCLUDE_DDR4\"\n\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HPS\"\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_UART\"\n
-
Build the FIM.
./ofs-common/scripts/common/syn/build_top.sh -p work_n6000_w_hps\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the n6001 and n6001 can be done by Remote System Update (RSU) using OPAE commands, or by programming a SOF image to the FPGA via JTAG using Quartus Programer.
Programming via RSU will program the flash device on the board for non-volatile image updates. Programming via JTAG will configure the FPGA for volatile image updates.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"Perform the following steps to set up a JTAG connection to the Intel\u00ae FPGA SmartNIC N6001-PL/Intel\u00ae FPGA SmartNIC N6001-PL.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
- This walkthrough requires an Intel FPGA Download Cable II.
Steps:
-
Set the board switches to dynamically select either the Agilex\u00ae 7 FPGA or MAX\u00ae 10 device on the JTAG chain.
- Set SW1.1=ON as shown in the next image. The switches are located at the back of the Intel\u00ae FPGA SmartNIC N6001-PL.
-
The Intel\u00ae FPGA SmartNIC N6001-PL has a 10 pin JTAG header on the top side of the board. Connect an Intel FPGA Download II Cable to the JTAG header of the Intel\u00ae FPGA SmartNIC N6001-PL as shown in picture below. This picture shows the Intel\u00ae FPGA SmartNIC N6001-PL card installed in the middle bay, top slot of a SuperMicro\u00ae SYS-220HE-FTNR server where the lower slot does not have card installed allowing the Intel\u00ae FPGA Download II cables to pass through removed the slot access.
Note: If using the Intel FGPA download Cable on Linux, add the udev rule as described in Intel FPGA Download Cable Driver for Linux.
-
Set the JTAG chain to select the Agilex\u00ae 7 FPGA as the target by writing to the JTAG enable register in the BMC (Register 0x378). This is done via PMCI registers 0x2040C and 0x20400.
Note: The commands below are targeted to a board with PCIe B:D.F of 98:00.0. Use the correct PCIe B:D.F of your card.
sudo opae.io init -d 0000:98:00.0 $USER\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x2040c 0x100000000\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:98:00.0\n
Note: To later re-direct the JTAG back to the MAX 10 device, execute the following commands.
sudo opae.io init -d 0000:b1:00.0 $USER\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x2040c 0x000000000\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:b1:00.0\n
Optionally, rather than dynamically commanding Agilex\u00ae 7 FPGA/MAX10 selection with the PMCI register settings, you can fix the selection with the following switch settings shown in the table below:
SW1.1 SW2 JTAG Target OFF OFF Agilex\u00ae 7 FPGA OFF ON MAX\u00ae 10 FPGA ON X Agilex\u00ae 7 FPGA if BMC register 0x378=0x1 ON X MAX\u00ae 10 FPGA if BMC register 0x378=0x0 -
Use the jtagconfig tool to check that the JTAG chain contains the AGFB014R24A2E2V device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL/Intel\u00ae FPGA SmartNIC N6000-PL with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the n6001. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae FPGA SmartNIC N6001-PL is connected via JTAG. The version of the programmer must be the same as the Quartus version used to build the FIM.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the n6001, if different than your deployment machine.
-
Open the Quartus programmer GUI
Note: the Quartus programmer version must be the same as the version of Quartus used to build the design.
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the n6001.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB014R24A2E2V device. The JTAG chain should show the divice.
-
Right click the AGFB014R24A2E2V row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGFB014R24A2E2V row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI. You can answer 'No' if a dialog pops up asking to save the 'Chain.cdf' file
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#53-remote-system-update","title":"5.3 Remote System Update","text":"The OPAE fpgasupdate tool can be used to update the Max10 Board Management Controller (BMC) image and firmware (FW), root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate tool only accepts images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then you must also sign the image using the correct keys. Refer to the Security User Guide: Open FPGA Stack for information on created signed images and on programming and managing the root entry hash.
The Intel\u00ae FPGA SmartNIC N6001-PL ships with a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL on all cards. The platform ships with a single FIM image that can be programmed into either user1 or user2, depending in the image selected.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#531-walkthrough-program-the-fpga-via-rsu","title":"5.3.1 Walkthrough: Program the FPGA via RSU","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL with a BIN image via RSU.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough requires a
BIN image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a BIN image. The image used for programming must be formatted with PACsign before programming. This is done automatically by the build script.
Steps:
-
Start in your deployment environment.
-
Determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Use the OPAE fpgasupdate command to program a PACsign signed image to flash. The flash slot which will be programmed is determined by the PACsign header.
sudo fpgasupdate <IMAGE> <PCIe B:D.F>\n
-
Example: update User Image 1 in flash
sudo fpgasupdate ofs_top_page1_unsigned_user1.bin 98:00.0\n
-
Example: update User Image 2 in flash
sudo fpgasupdate ofs_top_page2_unsigned_user2.bin 98:00.0\n
-
Example: update Factory Image in flash
sudo fpgasupdate ofs_top_page0_unsigned_factory.bin 98:00.0\n
-
Use the OPAE rsu command to reconfigure the FPGA with the new image. You may select which image to configure from (User 1, User 2, Factory).
sudo rsu fpga --page <PAGE> <PCIe B:D.F>\n
-
Example: configure FPGA with User 1 Image
sudo rsu fpga --page user1 98:00.0\n
-
Example: configure FPGA with User 2 Image
sudo rsu fpga --page user2 98:00.0\n
-
Example: configure FPGA with Factory Image
sudo rsu fpga --page factory 98:00.0\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#appendix","title":"Appendix","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#appendix-a-resource-utilization-tables","title":"Appendix A: Resource Utilization Tables","text":"Table A-1 Default Out-of-Tree FIM Resource Utilization
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 207715.9 42.63 689 9.69 PCIE_RST_CTRL.rst_ctrl 476.7 0.1 0 0.0 afu_top 145973.9 29.96 331 4.66 auto_fab_0 2839.6 0.58 12 0.17 bpf 1286.6 0.26 0 0.0 fme_top 615.7 0.13 6 0.08 hps_ss 0.0 0.0 0 0.0 hssi_wrapper 22143.5 4.55 131 1.84 local_mem_wrapper 8456.9 1.74 60 0.84 pcie_wrapper 18449.2 3.79 114 1.6 pmci_wrapper 6225.3 1.28 27 0.38 qsfp_0 622.5 0.13 4 0.06 qsfp_1 622.7 0.13 4 0.06 sys_pll 0.5 0.0 0 0.0 Table A-2 Minimal FIM Resource Utilization
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 90688.0 18.61 410 5.77 PCIE_RST_CTRL.rst_ctrl 384.0 0.08 0 0.0 afu_top 49696.4 10.2 197 2.77 auto_fab_0 1742.0 0.36 8 0.11 bpf 1380.2 0.28 0 0.0 fme_top 667.2 0.14 6 0.08 hps_ss 0.0 0.0 0 0.0 hssi_dummy_csr 688.1 0.14 0 0.0 local_mem_wrapper 9020.6 1.85 60 0.84 pcie_wrapper 18899.2 3.88 112 1.58 pmci_wrapper 6838.5 1.4 27 0.38 qsfp0_dummy_csr 688.5 0.14 0 0.0 qsfp1_dummy_csr 682.4 0.14 0 0.0 sys_pll 0.4 0.0 0 0.0"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/","title":"Hard Processor System Software Developer Guide: OFS for Intel Agilex FPGAs Targeting Intel\u00ae N6000/1-PL FPGA SmartNIC Platform","text":"Quartus Prime Pro Version: 23.1
Last updated: Last updated: November 19, 2024
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#glossary","title":"Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel Max10 or Intel Cyclone10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implemented on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Management Component Transport Protocol MCTP A standardized model for communication with management controllers. Defines the transport protocol carrying PLDM messages through the BMC. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE-SDK The OPAE-SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Level Data Model PLDM A specification for reporting telemetry data to the host, such as board temperature, voltage, and current. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#file-types","title":"File Types","text":"Extension Description ITS File (*.its) The image source file which describes the contents of the image and defines various properties used during boot. Actual contents to be included in the image (kernel, ramdisk, etc.) are specified in the image source file as paths to the appropriate data files. ITB File (*.itb) Produced as output from mkimage, using an image source file. Contains all the referenced data (kernel, ramdisk, SSBL, etc.) and other information needed by UBoot to handle the image properly. This image is transferred to the target and booted. DTB File (*.dtb) The Device Tree Blob is loaded into memory by U-Boot during the boot process, and a pointer to it is shared with the kernel. This file describes the system's hardware layout to the kernel. FIT Image (*.fit) Format used for uImage payloads developed by U-Boot. On aarch64 the kernel must be in image format and needs a device tree to boot. SPL (*.spl) The Secondary Program Loader is a small binary which is embedded in a FIM SOF and loaded into board DDR4 RAM when the FPGA is initially configured. This file is responsible for loading U-Boot into system RAM."},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#10-introduction","title":"1.0 Introduction","text":"The Open FPGA Stack (OFS) is a modular collection of hardware platform components, open source upstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS Provides a framework of FPGA synthesizable code, a simulation environment and synthesis/simulation scripts. The updated OFS architecture for Intel Agilex FPGA devices improves upon the modularity, configurability and scalability of the first release of the OFS architecture while maintaining compatibility with the original design. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem
- HSSI Subsystem
- Memory Subsystem
- Hard Processor System (HPS)
- Reset Controller
- FPGA Management Engine (FME)
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- SPI Interface to BMC controller
The Intel\u00ae N6000-PL and N6001-PL FPGA SmartNIC Platforms are acceleration cards that use the OFS infrastructure. The key difference between these two platforms is:
- Intel\u00ae N6000-PL SmartNIC Platform has a bifurcated PCIe bus with Gen4x8 interfacing to the the Intel Agilex FPGA and Gen4x8 interfacing to an Intel E810 SmartNIC. This platform is targeted specifically for VRAN, UPF and vCSR applications. The FPGA designs targeting these vertical market applications were generated using the OFS infrastructure.
- Intel\u00ae N6001-PL SmartNIC Platform has a Gen4x16 interface directly to the Intel Agilex FPGa and is not populated with an Intel E810 SmartNIC. This platform is the reference platform for the OFS reference designs for Intel Agilex FPGA.
Note: throughout this document \"Intel N6000/1-PL FPGA SmartNIC Platform\" denotes both cards. This document describes the software package that runs on the Hard Processor System (HPS) which is a key component within both platforms.
The Intel N6000/1-PL FPGA SmartNIC Platform has a customized build script that can be used to both set up a development environment and build the essential pieces of the HPS software image. This script, meta-bake.py, has its own dedicated Section 3.1 Building U-Boot which can be used to quickly get started with the HPS build flow. It is recommended you use this build script to construct the first and second stage bootloader files, as it will handle all of the setup and patching required to build out your complete Yocto image. You can familiarize yourself with the contents of this package in its public GitHub repository located at https://github.com/OPAE/meta-opae-fpga/tree/main/tools/meta-bake. All other information included for individual components is included for learning purposes only and is not meant as a recipe for image creation.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#11-reference-documents","title":"1.1 Reference Documents","text":"This document pulls much of its information from related Agilex FPGA documentation hosted on intel.com. Reading these references is not required for initial platform bring up, but will serve to further your knowledge of the FPGA SoC boot and configuration process.
Table 1. Reference Documents
Document Title Intel\u00ae Agilex\u2122 Hard Processor System Technical Reference Manual Intel\u00ae Agilex\u2122 SoC FPGA Boot User Guide Intel\u00ae Agilex\u2122 Configuration User Guide"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#12-reference-images","title":"1.2 Reference Images","text":"Intel has provided a set of two pre-compiled ITB images that can be used for exploration and evaluation of the HPS bring-up flow. These images contain the complete SSBL package specific to the board and can be copied to the N6000/1-PL SmartNIC Platform with an HPS enabled FIM loaded. Refer to Section 4.1 Example Boot for an example on how to use the built-in copy engine IP in tandem with the host-side cpeng software to transfer an SSBL.
The package is found on the official OFS 2024.2-1 Release on GitHub. Two ITB artifacts are included at the bottom of the page under Assets - one with the Vendor Authorized Boot (VAB) certificate included, and one without. Which you choose to load depends on whether the currently loaded FIM requires VAB authentication. Section 4.3 Example Boot contains instructions on the boot flow using these files for platform bring up.
The default username for these two images is root and the password is empty. A good place to start after loading the ITB is to set up SSH for file transfers and the remote console, as seen in 8.0 Connecting remotely to the HPS using ssh.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#20-architecture-overview","title":"2.0 Architecture Overview","text":"The OFS architecture is classified into:
- 1. Host Interface Adapters (PCIe)
- 2. Low Performance Peripherals
- 2.1. Slow speed peripherals (example: JTAG, I2C, SMBus, and so on)
- 2.2. Management peripherals (example: FPGA FME)
- 3. High Performance Peripherals
- 3.1. Memory peripherals
- 3.2. Acceleration Function Units (AFUs)
- 3.3. HPS Peripheral
- 4. Fabrics
- 4.1. Peripheral Fabric (multi drop)
- 4.2. AFU Streaming fabric (point to point)
The HPS is connected to the AFU and implements all the board specific flows that customers require to begin the application development using the HPS such as host communication, firmware load and update, integration with OFS, and memory. The HPS implements a basic Hello World software application and is intended as a starting point for customers to begin development with HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#21-hps-peripherals","title":"2.1 HPS Peripherals","text":"Figure 1 Intel Agilex FPGA HPS Peripherals
The Intel Agilex\u2122 SoC integrates a full-featured Arm\u00ae Cortex-A53\u00ae MPCore Processor.
The Cortex-A53 MPCore supports high-performance applications and provides the capability for secure processing and virtualization.
Each CPU in the processor has the following features:
- Support for 32-bit and 64-bit instruction sets.
- To pipeline with symmetric dual issue of most instructions.
- Arm NEON\u00ae Single Instruction Multiple Data (SIMD) co-processor with a Floating-Point Unit (FPU)
- Single and double-precision IEEE-754 floating point math support
- Integer and polynomial math support.
- Symmetric Multiprocessing (SMP) and Asymmetric Multiprocessing (AMP) modes.
- Armv8 Cryptography Extension.
- Level 1 (L1) cache:
- 32 KB two-way set associative instruction cache.
- Single Error Detect (SED) and parity checking support for L1 instruction cache.
- 32 KB four-way set associative data cache.
- Error checking and correction (ECC), Single Error Correct, Double Error Detect (SECDED) protection for L1 data cache.
- Memory Management Unit (MMU) that communicates with the System MMU (SMMU).
- Generic timer.
- Governor module that controls clock and reset.
- Debug modules:
- Performance Monitor Unit.
- Embedded Trace Macrocell (ETMv4).
- Arm CoreSight\u00ae cross trigger interface, the four CPUs share a 1 MB L2 cache with ECC, SECDED protection.
A Snoop Control Unit (SCU) maintains coherency between the CPUs and communicates with the system Cache Coherency Unit (CCU). At a system level, the Cortex-A53 MPCore interfaces to a Generic Interrupt Controller (GIC), CCU, and System Memory Management Unit (SMMU).
Beyond the Arm Cortex-A53 MPCore Processor, the HPS integrates a variety of useful peripherals for use in your design, such as Ethernet, USB, Memory Controller, on-chip RAM, SPI, UART and more. Refer to the Intel\u00ae Agilex\u2122 Hard Processor System Technical Reference Manual for more information.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#22-zarlink-device","title":"2.2 Zarlink Device","text":"The Microchip\u00ae Zarlink device ZL30793 is used for time synchronization. It acts as the protocol engine that drives IEEE 1588-2008 PTP protocol. The Zarlink device is connected to the HPS side and the programming interface is SPI. The FPGA bitstream containing the HPS has the First Stage Bootloader (FSBL) only. This enable commands to be given from a terminal program connected through UART.
The software in HPS can access the Clock generator through SPI to enable write and read operations controlled by the terminal program. It can also read the status of the hold over and Loss of Lock signals and control the LED.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#23-copy-engine","title":"2.3 Copy Engine","text":"A host with OPAE SDK and Linux DFL installed will provide the hps OPAE command with related options to transfer images from host to the HPS image. The module in the OFS FIM and HPS software that performs this transfer is called the Copy Engine (CPE), which is included by default within the HPS image.
Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for platform and software installation instructions.
The CPE software is patched into Linux on the HPS in Yocto through the meta-intel-fpga-refdes layer. This service is daemonized and requires systemd in order to operate. This service will communicate with the HPS IP integrated in the FIM in order to coordinate and monitor file transfers from the host CPE software to DDR connected the HPS. The CPE HPS-side software takes advantage of the built-in I/O lightweight kernel module to communicate with the FIM's HPS IP. It can restart the transfer if the initial transfer of the image is not successful. The CPE can also serve as reference on how to integrate your own systemd service in the Linux build running on the HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#24-boot-flow","title":"2.4 Boot Flow","text":"The boot flow for the Agilex OFS design for the Intel N6000/1-PL FPGA SmartNIC Platform is an FPGA-first boot flow, meaning the Intel Agilex Secure Device Manager (SDM) configures the FPGA first and then boots the HPS. Alternatively, you can boot the HPS first and then configure the FPGA core as part of the Second-Stage Boot Loader (SSBL) or after the Operating System (OS) boots. HPS-first boot is not covered in this document, but for more information please refer to the Intel\u00ae Agilex\u2122 SoC FPGA Boot User Guide.
For the FPGA-first boot flow supported by the Intel Agilex OFS FIM, the First Stage Bootloader is part of FPGA bitstream. The available Secure Device Manager (SDM) in the FPGA initially configures the FPGA core and periphery in this mode. The first stage bootloader is produced as a part of a U-Boot build and cna be manually inserted into a Quartus generated SOF file as shown in step 7 of Section 9.2 Configuring the HPS.
After completion, the HPS boots. All the I/O, including the HPS-allocated I/O, are configured, and brought out of tri-state. If the HPS is not booted:
- The HPS is held in reset
- HPS-dedicated I/O are held in reset
- HPS-allocated I/O are driven with reset values from the HPS.
- If the FPGA is configured before the HPS boots, then the boot flow looks as shown in the example figure below.
Figure 2. Typical FPGA First Configuration Steps
The flow includes the Time from Power-on-Reset (TPOR) to boot completion (TBoot_Complete).
Table 2. FPGA Configuration First Stages
Time Boot Stage Device State TPOR to T1 POR Power-on reset T1 to T2 SDM: Boot ROM 1. SDM samples the MSEL pins to determine the configuration scheme and boot source. 2. SDM establishes the device security level based on eFuse values. 3. SDM initializes the device by reading the configuration firmware (initial part of the bitstream) from the boot source. 4. SDM authenticates and decrypts the configuration firmware (this process occurs as necessary throughout the configuration). 5. SDM starts executing the configuration firmware. T2 to T3 SDM: Configuration Firmware 1. SDM I/O are enabled. 2. SDM configures the FPGA I/O and core (full configuration) and enables the rest of your configured SDM I/O. 3. SDM loads the FSBL from the bitstream into HPS on-chip RAM. 4. SDM enables HPS SDRAM I/O and optionally enables HPS debug. 5. FPGA is in user mode. 6. HPS is released from reset. CPU1-CPU3 are in a wait-for-interrupt (WFI) state. T3 to T4 First-Stage Boot Loader (FSBL) 1. HPS verifies the FPGA is in user mode. 2. The FSBL initializes the HPS, including the SDRAM. 3. The user application through the host must request the copy engine using the OPAE command hps to transfer the itb image (SSBL +Linux) to the HPS DRAM. 4. HPS peripheral I/O pin mux and buffers are configured. Clocks, resets, and bridges are also configured. 5. HPS I/O peripherals are available. T4 to T5 Second-Stage Boot Loader (SSBL) 1. HPS bootstrap completes. 2. OS is loaded into SDRAM. T5 to TBoot\\Complete Operating System (OS) The OS boots and applications are scheduled for runtime launch. When using the Pre-boot Execution Environment (PXE) boot mechanism, you must use an option ROM. OFS FIM does not have PXE boot implemented in the HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#25-authorization","title":"2.5 Authorization","text":"The HPS FSBL is part of the static region (SR) FPGA bitstream. Intel provides the capability to sign the FPGA bitstream binaries so that they can be authenticated when remotely updated and when configuring the FPGA. Signing of the SR bitstream is a two-stage process where you must sign with:
1. `quartus_sign` tool\n2. OPAE `PACSign` tool\n
Signing with PACSign ensures the security of the BMC RSU update process to the flash, and requires a compatible binary file. Quartus signing provides ensures security when the bitstream is configured through the SDM into the Intel Agilex FPGA device using Vendor Authorized Boot.
Vendor Authorized Bootloader (VAB) considers the SDM as a trusted root entity such that when firmware is authenticated and booted and is running on the SDM with dedicated crypto HW IP blocks, it is considered a trusted entity. As such it is trusted to perform the authentication and authorization steps for subsequent bitstreams.
Each subsequent loaded object after the SDM boot firmware does not need to re-implement the authentication and authorization functions. The authentication and authorization functions are centralized. Arm Trusted Firmware (ATF) is used to make a trusted execution environment (TEE) in the HPS. The source code for both Arm Trusted firmware and the First Stage Boot Loader (FSBL) is provided in the GitHub.
The SSBL + Linux is a part of an itb file and may also be signed with Quartus_sign and PACSign for authentication. This process is demonstrated in Section 9.2 Configuring the HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#30-environment-setup","title":"3.0 Environment Setup","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#31-building-u-boot","title":"3.1 Building U-Boot","text":"When creating a development environment using meta-bake.py both U-Boot and the patches required to work with the Intel N6000/1-PL FPGA SmartNIC Platform are located at /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig. To review the required patches applied to U-Boot, navigate to /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/git/patches. From there, using git commands such as git status and git branch will show changes to the build environment.
Currently the meta-bake build flow requires a specific environment and software dependencies. Refer to section 6.1 Running meta-bake.py for more information. Download the script from meta-opae-fpga.
Invoke the meta-bake.py build script to build your entire image, including U-Boot.
$ cd /meta-opae-fpga/tools/meta-bake\n$ ./meta-bake.py --conf n6000/layers.yaml builddir\n
This build process is highly system dependent and can take upwards of 1 hour to complete. Make sure you have at least 200 GB of free space on the system before running a build.
To build U-Boot manually after execution of meta-bake.py navigate to /meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-srcfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig and run make. After running meta-bake.py, you can rebuild U-Boot to incorporate any changes you have made. Navigate to the U-Boot directory at /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig and run the following commands to rebuild.
$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\n$ tar xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\n$ rm gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\n$ export CROSS_COMPILE=`pwd`/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-\n$ export ARCH=arm64\n$ make -j `nproc`\n
This recompile will result in a new ITB SSBL which may be loaded on an Intel FPGA SmartNIC N6000/1 platform. Several components of the ITB image are present under the U-Boot directory but are not rebuilt as a part of this flow. These files will need to be replaced before rebuilding U-Boot for changes to take affect.
U-Boot comes with its own dumpimage tool, which can be used to identify an image and extract and identify its contents. This tool is built by default under /u-boot-socfpga/tools, and in the meta-bake.py environment setup under /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig/tools. This tool can also be used to extract specific components of the ITB file.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#32-yocto","title":"3.2 Yocto","text":"Yocto is an open source toolkit used to create Linux distributions and commonly used for creating Linux images and bootloaders for embedded environments. A Yocto build environment is made up of one or more layers, with each layer consisting of recipes that get processed to build/install components in a layer. The workhorse of a Yocto build is the program called bitbake. This program processes the recipes to compile and build packages and images for the target platform. For SoC platforms, like the HPS, the ARM cross compiler is required.
The build script used for the Agilex SoC GSRD, create-linux-distro-release, is a bash script that automates the build of Linux images of different types (gsrd, nand, etc.) that are compatible with a target FPGA platform (agilex, stratix10, etc.). This script has been ported to Python 3 and modified to build an environment for the Intel FPGA SmartNIC N6000/1 platform, named meta-bake.py. This script pulls in the necessary patches and additional changes needed to support the platform.
In general, meta-bake.py pulls Yocto layers/recipes from public repositories, configures a Yocto build environment, and builds an image for a supported FPGA platform. The Yocto layer is always the first to be built, and includes the bitbake utility. The following table lists remote repositories hosting Yocto meta data source used by meta-bake.py and create-linux-distro as well as source code used for building binaries that make up the Linux image (kernel and rootfs).
Note: Not all repositories can be viewed in a web browser. All can be cloned using git.
Repository Description https://git.yoctoproject.org/git/poky Base build tools and meta data layers https://git.openembedded.org/meta-openembedded Layers for OE-core packages https://git.yoctoproject.org/git/meta-intel-fpga Meta data layers for Intel FPGA SoC platforms https://github.com/altera-opensource/meta-intel-fpga-refdes BSP layer for Intel SoC FPGA GSRD https://github.com/altera-opensource/linux-socfpga Linux kernel source repository for socfpga https://github.com/altera-opensource/u-boot-socfpga U-Boot bootloader source repository for socfpga https://github.com/altera-opensource/arm-trusted-firmware Source for ATF Recipes in the meta-intel-fpga-refdes layer mostly inherit from and extend recipes in other layers. The following table lists the new or modified recipes (in meta-intel-fpga-refdes) necessary to support an Intel FPGA SmartNIC N6000/1 HPS boot image.
Component Recipe Description Linux Kernel recipes-kernel/linux/linux-socfpga-lts_5.10.bbappend Recipe to append the GSRC SoC FPGA device tree to the Yocto build U-Boot recipes-bsp/u-boot/u-boot-socfpga_v2021.07.bbappend Recipe to fetch and build socfpga U-Boot. Modified to support N6000/1 in U-Boot. This also creates a shell script, *mkuboot-fit.sh. copy-engine recipes-bsp/copy-engine/copy-engine-0.1.bb New recipe to build copy-engine daemon in rootfs. N6000/1 Image recipes-images/poky/n6000-image-minimal.bb New recipe to create the N6000/1 image with copy-engine and linuxptp packages installed. mkuboot-fit.sh is meant to be called after a Yocto build to create the U-Boot FIT image for N6000/1, and is called automatically by meta-bake.py. This is a workaround for the Yocto build order which builds the bootloader (U-Boot) before building the Linux image rootfs. Because the rootfs is part of the U-Boot FIT image, the rootfs must be built before building the bootloader. The result of calling this script is copying the rootfs (as a .cpio file) to the U-Boot source file tree and calling make in the U-Boot build tree. When called again with the rootfs present, the resulting image will contain the rootfs. This is performed automatically as a part of the meta-bake.py build flow.
See here for more information regarding Yocto. Several reference designs found in rocketboards.org use Yocto for building the Linux image and/or bootloader. For the N6000/1 image and boot flow, the Yocto build script for the Agilex SoC Golden System Reference Design has been adapted to automate building the boot loader, Linux Image, and filesystem needed to support N6000/1 devices.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#321-customizing-the-yocto-image","title":"3.2.1 Customizing the Yocto Image","text":"The following is a list of customizations made for building Yocto to run on the Intel FPGA SmartNIC N6000/1-PL platform.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3211-extending-the-u-boot-recipe","title":"3.2.1.1 Extending the U-Boot recipe","text":"A recipe extension file (recipes-bsb/u-boot/u-boot-socfpga_v2021.07.bbappend) has been added to the meta-intel-fpga-refdes layer which accomplishes the following:
- Adds patches using Yocto's patching mechanism
- Introduces a new U-Boot config, socfpga_agilex_n6000_defconfig, and associates it with a keyword,
agilex-n6000, that can be referenced in Yocto configuration files. These patches are necessary until those changes are merged into the public u-boot-socfpga repository. This config works for both Smartnic Platforms. - Creates mkuboot-fit.sh script file with variables for U-Boot source and build directories that will get expanded to the actual paths that Yocto uses for fetching/building U-Boot. Along with this recipe file, relevant patch files have been added. Once the changes are in the U-Boot repository, the patches and any references to them must be removed.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3212-patching-the-linux-kernel","title":"3.2.1.2 Patching The Linux Kernel","text":"The kernel extension recipe, meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts_5.10.bbappend, in the meta-intel-fpga-refdes layer, has been modified to add a patch file using Yocto's patching mechanism. This patch file adds the device tree for N6000/1 and is only necessary until this change is merged into the public linux-socfpga repository.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3213-adding-custom-user-space-software","title":"3.2.1.3 Adding Custom User Space Software","text":"A new recipe, meta-intel-fpga-refdes/recipes-bsp/copy-engine-0.1.bb and relevant source files, have been added to the meta-intel-fpga-refdes layer. This recipe includes instructions for building the copy-engine program as well as installing it as a systemd service. Yocto will build this into an RPM package that gets installed into any image that includes it in the IMAGE_INSTALL variable. This recipe may be used as a guide for installing additional user space software.
You may also create a new Hello World application and add it to the Yocto build as shown below.
- Generate a BSD 3-Clause License and create an MD5 hash of it.
Copyright (c) 2023, User's Name All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Company Name nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
After license creation you will need to create an MD5 Hash of the clause. On Linux you can either pipe the raw text into echo \"text\" | md5sum, or create a new file and point to it md5sum licensefile.txt.
- Create a BB recipe file, following the same directory structure as other Yocto recipes.
$ cd /meta-opae-fpga/tools/meta-bake/build/meta-intel-fpga-refdes\n$ mkdir -p recipe-example/helloworld && cd recipe-example/helloworld\n
Create recipe file helloworld.bb in directory helloworld.
SUMMARY = \"Example hello world\" DESCRIPTION = \"helloworld in HPS\" AUTHOR = \"Your Name <your.email@address.com>\" LICENSE = \"BSD-3-Clause\" LIC_FILES_CHKSUM = \"file://${COMMON_LICENSE_DIR}/BSD-3-Clause;md5=<Your MD5 Hash>\" inherit systemd pkgconfig SRC_URI = \"file://helloworld.c\" S = \"${WORKDIR}\" do_compile() { ${CC} ${CFLAGS} ${LDFLAGS} helloworld.c -o helloworld }\ndo_install() { install -d ${D}${bindir} install -m 0755 helloworld ${D}${bindir} }
- Create source file
helloworld.c in the same helloworld directory.
#include <stdio.h> void main void() { Printf(\u201c\\nHello World\\n\u201d) }
- Re-run
./meta-bake.py --conf n6000/layers.yaml <Build Directory>. This will a new programmable SSBL that contains your Hello World program. Program the resulting ITB file as shown in Section 4.3 Example Boot and verify the application has been included in your build.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3214-adding-kernel-driver-software","title":"3.2.1.4 Adding Kernel Driver Software","text":"New recipes for custom kenel modules can be created at /build/meta-intel-fpga-refdes/recipes-kernel/linux/, and instructed to include custom module code. These can be patched in, included as a part of a new branch, or included by default if upstreamed. For more information visit the YoctoProject's Linux Kernel Development Manual. An example file from N6000/1 that can be used as an example is /build/meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts_5.10.bbappend.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3215-creating-an-image-type","title":"3.2.1.5 Creating an Image Type","text":"A new recipe, meta-intel-fpga-refdes/recipes-images/poky/n6000-image-minimal.bb, has been added that includes directives to install the copy-engine package (built in this layer) as well as the linuxptp package (available in other layers). In addition to including these packages, this recipe includes a rootfs post processing command that removes the Linux kernel image files from the rootfs. This is done because the Linux kernel is part of the U-Boot FIT image and therefore not used from the rootfs. Removing this redundant file reduces the final U-Boot FIT image by about 30Kb. This recipe may be modified or used as a guide to add additional user space software.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3216-testing-and-debugging","title":"3.2.1.6 Testing and Debugging","text":"As mentioned previously, the script will erase source files every time it is executed. This means that any changes made locally will be lost when the script is run again after making these changes. The example below shows how to test local changes without executing the script again.
$ cd build\n$ source poky/oe-init-build-env agilex-gsrd-rootfs/\n$ bitbake n6000-image-minimal\n$ ./agilex-n6000-rootfs/tmp/deploy/images/agilex/mkuboot-fit.sh\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#40-booting-the-hps","title":"4.0 Booting the HPS","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#41-ofs-fim-boot-overview","title":"4.1 OFS FIM Boot Overview","text":"This implementation of an FPGA First boot flow requires that the FSBL poll on a given register before continuing to boot the HPS. Once this register indicates it is (copy engine) ready, the FSBL loads a monolithic U-Boot FIT image at a given offset 0x02000000.
This image is made up of the following components:
- U-Boot bootloader also referred to as second stage bootloader
- Linux Kernel image
- Root filesystem (rootfs) consisting of kernel modules as well as user space software.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#42-booting-ofs-fim-for-intel-agilex-fpga","title":"4.2 Booting OFS FIM for Intel Agilex FPGA","text":"As mentioned before, the Intel N6000/1-PL FPGA SmartNIC boot flow is an FPGA-first boot flow which requires that the Intel Agilex FPGA to be configured with the necessary components (SPL/FSBL, copyengine) in order for the HPS to boot.
SD/eMMC is not supported for FSBL for HPS first
- First Stage Bootloader (FSBL): u-boot-spl-dtb.hex is embedded into FPGA image
- Monolithic FIT Image Downloaded from Host, u-boot.itb contains the following components
1) Second Stage Bootloader (SSBL): U-Boot + U-Boot Device Tree
2) Linux kernel, Image, + Linux Device Tree
3) Linux RAM based Root File System
- First Stage Bootloader (FSBL) is U-boot-spl
- U-boot-spl is built when U-Boot is built
- Artifact is u-boot-spl-dtb.hex
- The user has to check into build location : ofs-n6001/syn/setup/vab_sw/u-boot-spl-dtb.hex
- Then run the command
- quartus/pfg -c -o hps/path=u-boot-spl-dtb.hex orig.sof orig/fsbl.sof
Things to consider while developing your ITB image:
- The size of the u-boot.itb matters.\n- FIT is downloaded to [0x2000000](https://github.com/altera-opensource/u-boot-socfpga/blob/541b6afcb183ddb350ad367c9b63cc6db94c1f6e/configs/socfpga_agilex_n6010_defconfig#L4)\n- Linux Device Tree and RootFS are unpacked to high memory\n- Linux is unpacked to an address specified in the FIT, [0xb600000](https://github.com/altera-opensource/u-boot-socfpga/blob/541b6afcb183ddb350ad367c9b63cc6db94c1f6e/arch/arm/dts/socfpga_agilex_n6010-u-boot.dtsi#L4)\n- If size of u-boot.itb is greater than 0xb600000 \u2013 0x2000000, then FIT will be corrupted mid extraction, resulting in unpredictable kernel crashes.\n
This example assumes the following preconditions have been met prior to booting HPS:
1) A SOF file synthesized with the SPL (u-boot-spl-dtb.hex).
2) Copy engine IP with relevant registers accessible to host and HPS.
Once the host FPGA boots with the required bitstream, the SPL in the HPS begins polling a register in the copy engine. One way to get an indication that the HPS is ready to continue past the SPL is to use a terminal emulator on a host with a serial cable connected to the FPGA's UART port. To transfer the U-Boot FIT image, use the hps cpeng subcommand from the host. Note, the hps program can be installed as part of installing the OPAE SDK and Linux DFL suite of packages.
hps command details are located in Section 5.0 HPS Command Usage.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#43-example-boot","title":"4.3 Example Boot","text":"This example assumes the following preconditions have been met prior to booting HPS:
- A SOF file synthesized with the SPL (u-boot-spl-dtb.hex).
- Copy engine IP with relevant registers accessible to host and HPS.
Once the host FPGA boots with the required bitstream, the SPL in the HPS will begin polling a register in the copy engine. One way to get an indication that the HPS is ready to continue past the SPL is to use a terminal emulator on a host with a serial cable connected to the FPGA's UART port.
To transfer the U-Boot FIT image, use the hps program with cpeng subcommand from the host. Note, the hps program is installed as part of installing the OPAE SDK suite of packages. See here for information on running the hps program. The following example assumes your N6000/1 board is at PCIe BDF 0000:b1:00.0.
# Bind vfio-pci driver to Copy Engine PCIe Physical Function\n$ sudo opae.io init -d b1:00.4 root\n# Load the HPS SSBL\n$ hps cpeng -f u-boot.itb\n[2021-09-25 01:59:25.538] [cpeng] [info] starting copy of file:u-boot.itb, size: 116725656, chunk size: 4096\n[2021-09-25 01:59:29.935] [cpeng] [info] last chunk 1944, aligned 2048\n[2021-09-25 01:59:29.935] [cpeng] [info] transferred file in 28498 chunk(s)\n[2021-09-25 01:59:29.935] [cpeng] [info] waiting for ssbl verify...\n[2021-09-25 01:59:33.848] [cpeng] [info] ssbl verified\n[2021-09-25 01:59:33.848] [cpeng] [info] waiting for kernel verify...\n[2021-09-25 01:59:39.626] [cpeng] [info] kernel verified\n
This will transfer the U-Boot FIT image via the copy engine IP to the HPS DDR and then signal completion of the transfer to the copy engine. Once the copy engine completes the actual transfer, it will write to the register the HPS SPL is polling on allowing the SPL to load the U-Boot bootloader which will in turn boot into the Linux image embedded in the U-Boot FIT image. If a terminal emulator is connected to the UART as described above, a user can observe U-Boot and Linux running on the HPS.
- Validate the HPS SSBL has been loaded by checking for its heartbeat.
$ hps heartbeat\n[2021-09-25 01:59:42.722] [heartbeat] [info] heartbeat value: 0x30015\n[2021-09-25 01:59:43.722] [heartbeat] [info] heartbeat value: 0x40015\n[2021-09-25 01:59:44.722] [heartbeat] [info] heartbeat value: 0x50015\n[2021-09-25 01:59:45.723] [heartbeat] [info] heartbeat value: 0x60015\n[2021-09-25 01:59:46.723] [heartbeat] [info] heartbeat value: 0x70015\n
- Login to HPS as user, root, with no password over serial connection. This process is covered in 8.0 Connecting remotely to the HPS using
ssh.
agilex login: root\nroot@agilex:~# ls\nroot@agilex:~# ls /\nbin dev home lib mnt root sbin sys usr\nboot etc init media proc run srv tmp var\nroot@agilex:~#\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#50-hps-command-usage","title":"5.0 HPS Command Usage","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#51-synopsis","title":"5.1 Synopsis","text":"hps OPTIONS SUBCOMMAND SUBCOMMAND\\_OPTIONS
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#52-description","title":"5.2 Description","text":"hps is an application to aid in the development, deployment, and debugging of an HPS (hard processor system) on an Intel Agilex device using OFS. The current version of the hps program assumes an AFU (accelerator functional unit) is configured into the FPGA that can be discovered/accessed through an OPAE library and used for communicating with the HPS. When invoked with one of its subcommands, hps will enumerate the AFU for that subcommand before executing it.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#53-options","title":"5.3 Options","text":"-h,--help\n\nPrint this help message and exit\n\n-p,--pci-address address\n\nUse address in the filter for locating the AFU where address must be in\n\nthe following format: [domain]\\bus\\:\\device\\.\\function\\\n\n-l,--log-level \\level\\\n\nstdout logging level. Must be one of:\n\n{trace,debug,info,warning,error,critical,off}\n\nDefault is info.\n\n-s,--shared\n\nopen in shared mode, default is off\n\n-t,--timeout timeout\n\nProgram timeout in milliseconds. Default is 60000 ms."},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#54-subcommands","title":"5.4 Subcommands","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#541-cpeng","title":"5.4.1 cpeng","text":"The copy engine command is used to program copy engine AFU registers to copy an image file from the host into the FPGA DDR. When the HPS boots, the first stage boot loader loads an image from a specific offset in DDR that will be used to transition into the second stage boot loader and subsequently boot into the embedded Linux that is also part of this image.
cpeng options description -h,--help Print this help message and exit -f,--filename filename Path to image file to copy. Default is u-boot.itb -d,--destination offset DDR Offset. Default is 0x2000000. -t,--timeout cpeng timeout Timeout of cpeng command in microseconds. Default is 1 sec (1000000 usec). -r,--data-request-limit size Can be 64, 128, 512, or 1024 and represents the PCIe request size in bytes that the copy engine IP will use. This is encoded to 0, 1, 2, or 3 and written to the copy engine DATA\\REQUEST\\LIMIT register. Default is 512. -c,--chunk size Split the copy into chunks of size size. 0 indicates no chunks. Chunk sizes must be aligned with data request limit. Default is 4096. --soft-reset Issue a soft reset only. --skip-ssbl-verify Do not wait for ssbl verify. --skip-kernel-verify Do now wait for kernel verify."},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#542-heartbeat","title":"5.4.2 heartbeat","text":"This subcommand reads the value in the HPS2HOST register to check for a hearbeat. This compares the value to previous value read and determines the HPS is alive if the value is incrementing. This relies on the hps running the hello-cpeng program in heartbeat mode which will increment the upper 16 bits in the HPS2HOST register. Please see a typical sequence of using the rsu and hps commands as below for a device with BDF 15:00:0
rsu fpga -p user1 15:00.0\nsudo opae.io release -d 15:00.0\nsudo opae.io init -d 15:00.4 root:root\nhps cpeng -f u-boot-userkey-vab.itb\ntimeout 5 hps heartbeat\nsudo opae.io release -d 15:00.0\nhps cpeng -f u-boot-userkey-vab.itb\n
The above command will transfer the U-Boot FIT image via the copy engine IP to the HPS DDR and then signal completion of the transfer to the copy engine. After the copy engine completes the actual transfer, it writes to the register the HPS SPL is polling on allowing the SPL to load the U-Boot bootloader which in turn boots into the Linux image embedded in the U-Boot FIT image. If a terminal emulator is connected to the UART as described above, a user can observe U-Boot and Linux running on the HPS.
First FSBL is loaded and executed by FPGA configuration. Then Board/Server gets powered on. FPGA Configuration is done via JTAG followed by a reboot
The FSBL will send the following output the serial port:
U-Boot SPL 2021.07-00312-g32c0556437 (Sep 17 2021 - 08:42:45 -0700)\nReset state: Cold\nMPU 1200000 kHz\nL4 Main 400000 kHz\nL4 sys free 100000 kHz\nL4 MP 200000 kHz\nL4 SP 100000 kHz\nSDMMC 50000 kHz\nDDR: Warning: DRAM size from device tree (1024 MiB)\nmismatch with hardware (2048 MiB).\nDDR: 1024 MiB\nSDRAM-ECC: Initialized success with 239 ms\nwaiting for host to copy image\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#60-meta-bakepy","title":"6.0 meta-bake.py","text":"A script called meta-bake.py has been added to allow for more control of configuration/customization of recipes and their dependencies. This script separates the data from the logic by requiring that data be expressed in a yaml configuration file. This file contains the following confiration data:
- machine - The FPGA/SoC platform to build for, choices are agilex, stratix10, arria10, cyclone5
- image - The image type to build, choices are gsrd, nand, pcie, pr, qsqpi, sgmii, tse, n6000
- target - The build target to build. This is typically a Yocto image to build.
- fit - Make a monolothic FIT image after the Yocto build. This will use U-Boot source and binaries as well as the rootfs made for the image.
- repos - A list of repositories to pull for Yocto recipes. This information is made up of:
- name - The project name (this is also the directory where source is clone to)
- url - The URL to pull the source from
- branch - The branch to checkout
- add_layers - Can be either True or a list of sub-directories to add as layers in bblayers.conf
- patch - Path to a file to use to patch the source code
- keep - When set to true, this will leave the source tree untouched on subsequent runs
- upstream_versions - Dependencies/versions to use for either Linux kernel, U-Boot, and/or ATF. This information is made up of:
- name - Project name
- version - version to configure recipes that use it
- branch - branch to use, will use git hash in recipe
- url - URL to pull the source from
- disabled - when set to True, this project will be ignored
- local - Used to configure local.conf used by Yocto/bitbake build. This information is made up of:
- remove - List of keys to remove from local.conf
- values - Dictionary of key/value pairs to use to insert into local.conf. Any existing key/value pairs will be overwritten.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#61-running-meta-bakepy","title":"6.1 Running meta-bake.py","text":"First, download the script from meta-opae-fpga.
To create a U-Boot fit and spl image for N6000/1 platforms, you must first meet these setup conditions:
- Host PC with Ubuntu 20.04 LTS
- ARM cross compiler
- set CROSS_COMPILE environment variable to: /bin/aarch64-linux-gnu- - e.x. export CROSS_COMPILE=aarch64-linux-gnu-
- set ARCH to: arm64 - e.x. export ARCH=arm64
- At least 100 Gb of disk space
- Tested on OFS 3.0.0
This script does not require any bare-metal accesses to perform its build and can be run from a VM with no alterations. Ensure that the Ubuntu 20.04 Guest VM you create has enough to space to perform the entire build (recommend at least 200 GiB total space), as does the drive it is stored on. You will need to configure proxy access in the VM if required for your environment. You can use any VM technology; having ssh enabled in the VM will allow you to more easily transfer the completed build files back to the host system but is not required.
Package dependencies to build Yocto on each supported OS can be found on the Yocto Quick Start page.
wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\ntar xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\nrm gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\nexport CROSS_COMPILE=`pwd`/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-\nexport ARCH=arm64\n./meta-bake.py build\n
After running this build, the images you need to boot the HPS are located under build/agilex-n6000-images. Follow the steps in Section 4.3 Example Boot to finish bringing up your board.
This script will do the following:
- Parse layers.yaml for configuration to use for build
- Download recipe repositories (including poky) listed in
repos secion of layers.yaml - Apply refdes-n6000 patch to meta-intel-fpga-refdes source tree
- Configure Yocto build in build directory
- Source build/poky/oe-init-build-env passing in agilex-n6000-rootfs. This will initialize conf files.
- Configure build/agilex-n6000-rootfs/conf/local.conf using values in
local section of layers.yaml * Note: IMAGE_FSTYPES is configured to include cpio - Configure build/agilex-n6000-rootfs/conf/bblayers.conf using layer specification in
repos section of layers.yaml - Run Yocto build for target listed in layers.conf
- Call
bitbake n6000-image-minimal - Get environment variables to locate rootfs cpio file as well as U-Boot source and build directories
- Copy rootfs created by Yocto build for U-Boot
- Copy rootfs cpio file (n6000-image-minimal-agilex*.rootfs.cpio) to U-Boot build directory for selected configuration (socfpga_agilex_n6000_defconfig)
- Call U-Boot build in directory for selected configuration
- Copy FIT image (u-boot.itb) to images directory, build/agilex-n6000-images
- Many important images are copied to build/agilex-n6000-images, which may be useful if using VAB
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#62-required-changes","title":"6.2 Required Changes","text":"The patch file applied on top of the meta-intel-fpga-refdes repository introduces patches to:
- Add patch files so that Yocto can modify Linux kernel to add configuration for creating a device tree binary (DTB) compatible with N6000/1
- Add patch files so that Yocto can modify the bootloader in U-Boot to support booting with the assistance of the copy engine IP
- Modify rootfs to include copy-engine daemon as well as other packages that can be useful for debug
These changes may eventually be merged into upstream repositories for linux-socfpga, u-boot-socfpga, and meta-intel-fpga-refdes. Once all changes make their way into the repositories for the aforementioned projects, it will no longer be necessary to apply patches.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#63-manual-build","title":"6.3 Manual Build","text":"One may use meta-bake.py to only pull down required repositories and configure a Yocto build environment by using the --skip-build command line argument. To initiate a build after this, source poky/oe-init-build-env passing in a directory as the only argument. This will set up the user's environment to be able to run bitbake. To build the Yocto image, run bitbake n6000-image-minimal. This will build all the components necessary to build a FIT image. Once the build is complete, U-Boot make system may be used to make the FIT. The U-Boot build directory for the selected configuration can be found in the Yocto build environment directory at:
$ cd tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig\n
Once in this directory, ensure that the necessary files are present in here in order to assemble the FIT image (u-boot.itb) $ cp ../../../../../../deploy/images/agilex/n6000-image-minimal-agilex.cpio rootfs.cpio\n$ ls Image linux.dtb rootfs.cpio\nImage linux.dtb rootfs.cpio\n$ make\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#64-manual-vab-signing","title":"6.4 Manual VAB Signing","text":" - By default,
meta-bake.py will sign and certify the proper files for use with VAB. Below is an example on how to perform the manual VAB Signing Process.
First, use gzip to compress the follwoing four files before signing: bl31.bin, linux.dtb, rootfs.cpio.gz, and Image. They need to be compressed before you begin the VAB signing process or the image will fail to load through the CPE later on.
gzip <filename>\n
Make sure Quartus already installed and its tools added to environment. Example PATH=$PATH:/home/intelFPGA\\pro/21.3/quartus/bin/
$ cd HPS_VAB\n$ quartus_sign --family=agilex --operation=make_private_pem --curve=secp384r1 --no_passphrase userkey_root_private.pem\n$ quartus_sign --family=agilex --operation=make_public_pem userkey_root_private.pem userkey_root_public.pem\n$ quartus_sign --family=agilex --operation=make_rootuserkey_root_public.pem userkey_root_public.qky\n$ chmod +x fcs_prepare\n$ ./fcs_prepare --hps_cert bl31.bin -v\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_cert_bl31.bin.ccert\n\n# ATF Sign\n$ ./fcs_prepare --finish signed_cert_bl31.bin.ccert --imagefile bl31.bin\n$ mv hps_image_signed.vab signed-bl31.bin\n$ rm unsigned_cert.ccert\n\n# u-boot-nodtb\n$ ./fcs_prepare --hps_cert u-boot-nodtb.bin -v\n\n#signed_u-boot-nodtb.bin.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_u-boot-nodtb.bin.ccert\n\n# u-boot-nodtb.bin Sign\n$ ./fcs_prepare --finish signed_u-boot-nodtb.bin.ccert --imagefile u-boot-nodtb.bin\n$ mv hps_image_signed.vab signed-u-boot-nodtb.bin\n$ rm unsigned\\_cert.ccert\n\n# u-boot.dtb\n$ ./fcs_prepare --hps_cert u-boot.dtb -v\n\n#signed_u-boot.dtb.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_u-boot.dtb.ccert\n\n# u-boot.dtb Sign\n$ ./fcs_prepare --finish signed_u-boot.dtb.ccert --imagefile u-boot.dtb\n$ mv hps_image_signed.vab signed-u-boot.dtb\n$ rm unsigned_cert.ccert\n\n# Image\n$ ./fcs_prepare --hps/cert Image -v\n\n#signed_Image.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_Image.ccert\n\n# Image Sign\n$ ./fcs_prepare --finish signed_Image.ccert --imagefile Image\n$ mv hps_image_signed.vab signed-Image\n$ rm unsigned_cert.ccert\n\n# linux.dtb\n$ ./fcs_prepare --hps_cert linux.dtb -v\n\n#signed_linux.dtb.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_linux.dtb.ccert\n\n# linux.dtb Sign\n$ ./fcs_prepare --finish signed_linux.dtb.ccert --imagefile linux.dtb\n$ mv hps_image_signed.vab signed-linux.dtb\n$ rm unsigned_cert.ccert\n\n# rootfs.cpio\n$ ./fcs_prepare --hps_cert rootfs.cpio -v\n\n#signed_rootfs.cpio.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_rootfs.cpio.ccert\n\n# rootfs.cpio\n$ ./fcs_prepare --finish signed_rootfs.cpio.ccert --imagefile rootfs.cpio\n$ mv hps_image_signed.vab signed-rootfs.cpio\n$ rm unsigned_cert.ccert\n
Copy the following files to u-boot-socfpga folder:
#Copy the image back to uboot folder\n$ cp signed-bl31.bin ../u-boot-socfpga/\n$ cp signed-u-boot-nodtb.bin ../u-boot-socfpga/\n$ cp signed-u-boot.dtb ../u-boot-socfpga/\n$ cp signed-Image ../u-boot-socfpga/\n$ cp signed-linux.dtb ../u-boot-socfpga/\n$ cp signed-root\n$ fs.cpio ../u-boot-socfpga/\n
Recompile the U-Boot $ git clone https://github.com/altera-opensource/u-boot-socfpga\n$ cd u-boot-socfpga\n$ export CROSS\\COMPILE=aarch64-none-linux-gnu-; export ARCH=arm\n$ make socfpga/agilex/n6000/vab/defconfig\n$ make -j 24\n$ cd ..\n
Figure 3.1 N6000/1 Configs
If you not see the defconfig desired, please checkout the correct branch version. Example config shown above is socfpga_v2021.10.
If the memory device tree it mismatches with your hardware (figure below), change the memory device tree at u-boot-socfpga/arch/arm/dts/socfpga_agilex_n6000-u-boot.dtsi
To make it 2GB, change as
memory {\n\n\\* 2GB \\*\n\nreg = <0 0x00000000 0 0x40000000>,<0 0x00000000 0 0x40000000>;\n\n};\n
Figure 3.2 Device tree mismatches example Refer to 6. Host Side Startup
$ sudo opae.io init -d 4b:00.4 root:root\n$ hps cpeng -f u-boot.itb\n$ timeout 5 hps heartbeat\n
The error happen (Figure below) when the Images do not sign with VAB. Figure 3.3 VAB certificate error example
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#70-debugging","title":"7.0 Debugging","text":"Debugging the HPS from the host side is standard HPS debugging. The primary debug tool is UART on the HPS and Arm DS-5 debugger.
A UART connection can be enabled on the board using the following procedure:
-
Connect the HPS Debug Card and HPS UART to the Intel N6000/1-PL FPGA SmartNIC Platform board
-
Open Putty with the following setting
Port:COM4\n\n Baudrate:115200\n\n Data bits : 8\n\n Stop bits : 1\n\n Parity : None\n\n Flow Control : None\n
-
Reboot the Intel N6000/1-PL FPGA SmartNIC Platform board by typing reboot in the shell. You will be able to see the HPS UART traffic in the putty. If any issues are encountered in this step, check the HPS UART connection and the UART driver.
-
Check the PCI bdf ( lspci | grep acc ) or fpgainfo fme at the shell prompt.
-
Run the rsu and fpga\\reconfig scripts with respective arguments to print the logs.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#80-connecting-remotely-to-the-hps-using-ssh","title":"8.0 Connecting remotely to the HPS using ssh","text":"The HPS running on the Intel FPGA N6000/1-PL SmartNIC Platform can be remotely accessed to via the utility ssh, allowing the user to run commands and copy files. SSH must be run over a Point-To-Point Protocol daemon that has been included in the HPS software (as a part of the meta-openembedded layer, in the recipes-daemons/ippool recipe). In this example, the HPS is set up as a PPP Server, and the host OS is set up as a PPP Client. Serial communication between the host and HPS is accomplished via HPS UART1, which communicates through the FIM to the Soft UART on the FPGA, who in turn communicates with the host over PCIe.
The following steps assume the SSBL has not yet been loaded onto the HPS. If it has, a cold boot will reset the system.
- The HPS Copy Engine Module is available for access on PF 4 via the PF/VF Mux on the FPGA. This port needs to be bound to driver
vfio-pci (the following example assumes PCIE BDF 0000:b1:00.0). Substitute your device's BDF address and desired user/group access permissions into the following command.
$ sudo opae.io init -d 0000:b1:00.4 <USER>[:<GROUP>]\nUnbinding (0x8086,0xbcce) at 0000:b1:00.4 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.4 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.4 is 190\nAssigning /dev/vfio/190 to DCPsupport\nChanging permissions for /dev/vfio/190 to rw-rw----\n
- When an HPS enabled SOF or BIN with the FSBL is loaded onto the FPGA, a message will be displayed on the host OS (seen via
dmesg) after boot once the serial port has been registered with the dfl-uart driver. The UART driver is included as a part of the linux-dfl driver package. An example output from dmesg is shown below (search dmesg using dmesg | grep dfl-uart):
[ 7.343014] dfl-uart dfl_dev.7: serial8250_register_8250_port 2\n
The device file that corresponds with serial UART port 2 is /dev/ttyS2 (format is /dev/ttyS<port number>). A serial communication program can be used to view the HPS boot in realtime, then log in and run commands when boot has completed. Minicom is the program that will be used in this example, although others will work. Install Minicom using DNF sudo dnf install minicom.
- Minicom requires configuration changes before it can listen to the serial device. Using the built-in menu accessed by
sudo minicom -s, ensure the information under \"Serial port setup\" matches the following, where the serial device corresponds with the serial port discussed previously:
+-----------------------------------------------------------------------+\n | A - Serial Device : /dev/ttyS2 |\n| |\n| C - Callin Program : |\n| D - Callout Program : |\n| E - Bps/Par/Bits : 115200 8N1 |\n| F - Hardware Flow Control : Yes |\n| G - Software Flow Control : No |\n| |\n| Change which setting? |\n+-----------------------------------------------------------------------+\n
-
Save and exit the configuration menu. Run Minicom using the command sudo minicom and keep the terminal open and connected.
-
Load the SSBL onto the HPS using a second terminal. This requires a built ITB image.
$ hps cpeng -f u-boot.itb\n
- You should see the HPS boot sequence continue through your Minicom terminal. Once boot has completed, log in using the user
root with an empty password.
...\n...\n...\n[ OK ] Finished Load/Save Random Seed.\n[ OK ] Finished OpenSSH Key Generation.\n\nPoky (Yocto Project Reference Distro) 3.3.6 agilex ttyS0\n\nagilex login: root\nroot@agilex:~#\n
- Configure the running Yocto image on the HPS as a PPP server. Run the following command through Minicom on the HPS (connects address 192.168.250.2 on the HPS to 192.168.250.1 on the host):
root@agilex:~# pppd noauth passive 192.168.250.1:192.168.250.2\n[ 410.465450] PPP generic driver version 2.4.2\n...\n
- Exit the Minicom program running on the host using
^A X. Execute the following command on the host to establish a PPP connection as the client (if not installed on the host, run sudo dnf install ppp):
$ sudo pppd ttyS2 115200 crtscts lock local noauth passive debug\n
- A new network interface device registered to ppp should be visible.
$ ip -4 addr\n...\n8: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 3\ninet 192.168.250.2 peer 192.168.250.1/32 scope global ppp0\n valid_lft forever preferred_lft forever\n
With both the client and server communicating, ssh and scp can be used to run commands and transfer files using IPv4 address 192.168.250.1 on the host. An example operation run on the host OS is shown below:
[user@localhost ]: scp file_package.tar.gz root@192.168.250.1\n
Note: If you are developing software for the HPS and altering system settings, it is possible for ssh to prohibit a connection due to a false man-in-the-middle attack warning. The flag <ssh/scp> -o StrictHostKeyChecking=no can be used to ignore the warning.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#90-example-design-enabling-emmc-i2c-and-uart-in-platform-design","title":"9.0 Example Design - Enabling eMMC, I2C and UART in Platform Design","text":"The following section will walk through the process by which eMMC, I2C, and UART can be added to the FIM and the HPS image. The goal of this section is to allow the HPS to configure eMMC memory on boot and uses WNC's FPGA SmartNIC Series as a reference.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#91-configuring-the-fim","title":"9.1 Configuring the FIM","text":" - Configure eMMC, I2C, and UAET in Platform Designer. ACtual pin assignments are determined by the WNC board schematic. In Quartus, navigate to the HPS Processor Subsystem Intel Agilex FPGA IP -> Pin Mux and Peripherals -> Advanced -> Advanced IP Placement.
Check your pin assigments for the eMMC, UART and I2C in the Pin Planner. If these assignments are not present, then they can be found at the following link. Based on the changes shown above, the UART pins are removed on HPS IO3 and IO4 what are mapped on AG6 and AB1.
- Click Apply Selections->Generate HDL
- Check for instantiation in
top.sv. Click Generate -> Show Instatiation Template.
The following image demonstrates eMMC and I2C properly instatiated in top.sv.
- Add the following to the hps_ss modules in
top.sv.
- Compile the design.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#92-configuring-the-hps","title":"9.2 Configuring the HPS","text":" - Enable mmc and DesignWare Memory Card interface flags in U-Boot (.config). After building U-Boot with
meta-bake.py, this file is located at /agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga//build/socfpga_agilex_n6000_defconfig/.config. CONFIG_CMD_MMC=y //Enable mmc command tool in uboot CONFIG_MMC_DW=y // support DesignWare Memory Card Interface\n
- Enable and configure I2C and eMMC in U-Boot (socfpga_agilex_n6000.dts). After building U-Boot with
meta-bake.py, this file is located at /agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/git/arch/arm/dts/socfpga_agilex_n6000.dts. - Enable and configure I2C and eMMC in the Linux device tree (socfpga_agilex_n6000.dts). After building U-Boot with
meta-bake.py, this file is located at /agilex-n6000-rootfs/tmp/work-shared/agilex/kernel-source/arch/arm64/boot/dts/intel/socfpga_agilex_n6000.dts. - Compile Linux by navigating to the directory agilex-n6000-rootfs/tmp/work/agilex-poky-linux/linux-socfpga-lts/5.10.60-lts+gitAUTOINC+c35d63f9c7-r0/linux-agilex-standard-build and running the following:
$ make -j `nproc` Image dtbs
- Add the software utilities
util-linux-mkfs e2fsprogs.mke2fs e2fsprogs in to the Linux RootFS. Thes utilities will be used to create a filsystem (ext4, FAT32, etc.) and partition the eMMC. Make the following changes in meta-intel-fpga-refdes/recipes-images/poky/n6000-image-minimal.bb.
-
As an output from the Linux compilation from step 4 you will produce the files Image and socfpga_agilex_n6000.dtb. Transfer both over to the socfpga_agilex_n6000_defconfig directory. Rename socfpga_agilex_n6000.dtb to linux.dtb. Compile U-Boot by running make in directory agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig/. This compilation will produce both spl/u-boot-spl-dtb.hex and u-boot.itb.
-
Combines these files with your SOF FSBL bootloader (created in Section 9.1 Configuring the FIM).
$ quartus_pfg -c ofs_top.sof ofs_top_hps.sof -o hps_path=u-boot-spl-dtb.hex\n$ quartus_pfg -c ofs_top_hps_pof_flash.pfg //to pof file and flash to qspi\n
Program the FSBL enabled SOF onto the FPGA and warm reboot the server for changes to take affect. After reboot has completed use the following commands to program the SSBL.
$ sudo opae.io release -d d8:00.0\n$ sudo opae.io init -d d8:00.4 root:root\n$ hps cpeng -f u-boot.itb\n
During HPS boot you should see the following message if the eMMC has been properly configured.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#93-emmc-testing-from-the-hps","title":"9.3 eMMC Testing from the HPS","text":"The following memory test was run from the U-Boot shell.
- Erase eMMC using a start block offset 0x400.
$ mmc erase 0x400 20\n
Write test data (at 0x00A00000) into the eMMC offset 0x400.
$ mmc write 0x00A00000 0x400 10\n
Read test data (at 0x00A00040) back from eMMC offset 0x400.
$ mmc read 0x00A00040 0x400 10\n
Data comparison at memory offset 0x00A00000 and 0x00A00040. Data should match.
$ cmp.l 0x00A00000 0x00A00040 10.\n
The following memory test was run from Linux running on the HPS.
- Display the eMMC and its partitions.
$ fdisk -l\n
- Create a primary partition on the eMMC.
- Verify the partition has been created.
- Format the ext3 filesystem in the partition you just created (p1).
- Create the directory
mydata in /mnt. Mount the eMMC p1 partition to /mnt/mydata and verify the filsystem mount was successful.
$ mkdir -p /mnt/mydata\n$ mount /dev/mmcblk0p1 /mnt/mydata\n$ df\n
- Create a new text file and write some data in it - \"Hello World!\". After the device has been written to run
sync, unmount eMMC p1 partition and verify the unmount was successful.
$ sync\n$ umount /dev/mmcblk0p1\n$ df\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#layersyaml-reference","title":"layers.yaml Reference","text":"machine: agilex\nimage: n6000\ntarget: n6000-image-minimal\nfit: true\nuboot-dtb: [u-boot-nodtb.bin, u-boot.dtb]\nlinux-binary: [bl31.bin, linux.dtb, rootfs.cpio, Image]\nroot-public-qky: userkey_root_public.qky\nroot-private-pem: userkey_root_private.pem\nroot-public-pem: userkey_root_public.pem\nrepos:\n- name: poky\nurl: https://git.yoctoproject.org/git/poky.git\nbranch: hardknott\n- name: meta-intel-fpga\nurl: https://git.yoctoproject.org/git/meta-intel-fpga.git\nbranch: hardknott\nadd_layers: true\n- name: meta-intel-fpga-refdes\nurl: https://github.com/altera-opensource/meta-intel-fpga-refdes.git\n#url__: https://github.com/intel-innersource/os.linux.yocto.reference-design.meta-intel-fpga-refdes.git\n#url: git@github.com:intel-innersource/os.linux.yocto.reference-design.meta-intel-fpga-refdes.git\n#branch: rrojo/n6000\nbranch: hardknott\npatch: refdes-n6000-ppp.patch\nkeep: true\nadd_layers: true\n- name: meta-openembedded\nurl: https://github.com/openembedded/meta-openembedded.git\nbranch: hardknott\nadd_layers:\n- meta-oe\n- meta-networking\n- meta-python\n- name: fcs_prepare\nurl: https://github.com/altera-opensource/fcs_apps.git\nbranch: fcs_prepare\ningredients:\nlinux:\nname: linux-socfpga\nversion: '5.10.100'\nbranch: socfpga-5.10.100-lts\nurl: https://github.com/altera-opensource/linux-socfpga.git\nuboot:\nname: u-boot-socfpga\nversion: '2021.07'\nbranch: socfpga_v2021.07\nurl: https://github.com/altera-opensource/u-boot-socfpga.git\natf:\ndisabled: true\nversion: '2.4.1'\nbranch: socfpga_v2.4.1\nurl: https://github.com/altera-opensource/arm-trusted-firmware.git\nlocal:\nremove:\n- MACHINE\n- UBOOT_CONFIG\n- IMAGE\n- SRC_URI\nvalues:\nMACHINE: $machine\nDL_DIR: $build_dir/downloads\nDISTRO_FEATURES_append: \" systemd\"\nVIRTUAL-RUNTIME_init_manager: systemd\nIMAGE_TYPE: $image\nIMAGE_FSTYPES: \"+=cpio tar.gz\"\nPREFERRED_PROVIDER_virtual/kernel: linux-socfpga-lts\nPREFERRED_VERSION_linux-socfpga-lts: 5.10%\nUBOOT_CONFIG: agilex-n6000\nPREFERRED_PROVIDER_virtual/bootloader: u-boot-socfpga\nPREFERRED_VERSION_u-boot-socfpga: v2021.07%\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#faqs","title":"FAQs","text":"Below are the Frequently Asked Questions:
-
How will you get the software stack for HPS (FSBL, U-Boot, Kernel)? Or will there be a package available to them on Git, Intel RDC?
Answer : HPS software has been around for quite a long time. Support for the OFS and the N6000-PL FPGA SmartNIC Platform will be upstreamed and available from rocketboards.com, just like any other HPS based project.
-
What are the recommended steps for building the binaries and where will those be located?
Answer: There are many documents on building the binaries at rocketboards.com. Any reference binaries can be stored at rocketboards.com as well.
-
What are the host side commands used to put the binaries to Copy Engine and from there to HPS?
Answer: There is a single command, hps to download the single binary through the Copy Engine to the HPS.
-
What are the host side commands used to reset the HPS from Host side?
Answer: This functionality is planned to be added to the hps command.
-
What is the procedure used to debug the HPS from Host side?
Answer: Debugging the HPS from the host side is standard HPS debugging. The primary debug tool is UART on the HPS and the Arm DS debugger.
-
Do we have performance metrics about HPS, like whether any bench marking information available with any sample application is available?
Answer: Any performance metrics on the HPS would be available on rocketboards.com.
-
What is the PXE boot flow and what is required to enable the same?
Answer: On some configurations, HPS is treated as a fully-fledged SoC and can PXE boot itself. But you must add this functionality.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#further-links","title":"Further Links","text":"Description Link OFS Getting Started User Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach FPGAs https://ofs.github.io/hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/ Intel\u00ae FPGA Interface Manager Developer Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach FPGAs https://ofs.github.io/hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/ Open FPGA Stack Technical Reference Manual for Intel Agilex FPGA PCIe Attach https://ofs.github.io/hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/ Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs https://ofs.github.io/hw/N6001/dev_guides/afu_dev/ug_dev_afu_n6001/ UVM Simulation User Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach FPGAs https://ofs.github.io/hw/n6001/user_guides/ug_sim_ofs_n6001/ug_sim_ofs_n6001/ FPGA Device Feature List (DFL) Framework Overview https://github.com/OFS/linux-dfl/blob/fpga-ofs-dev/Documentation/fpga/dfl.rst#fpga-device-feature-list-dfl-framework-overview ofs-platform-afu-bbb https://github.com/OFS/ofs-platform-afu-bbb Connecting an AFU to a Platform using PIM https://github.com/OFS/ofs-platform-afu-bbb/blob/master/plat_if_develop/ofs_plat_if/docs/PIM_AFU_interface.md example AFUs https://github.com/OFS/examples-afu.git PIM Tutorial https://github.com/OFS/examples-afu/tree/main/tutorial Non-PIM AFU Development https://github.com/OFS/examples-afu/tree/main/tutorial Intel FPGA IP Subsystem for PCI Express IP User Guide https://github.com/OFS/ofs.github.io/docs/hw/common/user_guides/ug_qs_pcie_ss.pdf Memory Subsystem Intel FPGA IP User Guide https://github.com/OFS/ofs.github.io/docs/hw/common/user_guides/ug_qs_mem_ss.pdf OPAE.io https://opae.github.io/latest/docs/fpga_tools/opae.io/opae.io.html OPAE GitHub https://github.com/OFS/opae-sdk README_ofs_n6001_eval.txt https://github.com/OFS/ofs-n6001/blob/release/ofs-2023.1/eval_scripts/README_ofs_n6001_eval.txt FIM MMIO Regions https://ofs.github.io/hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#mmio_regions evaluation script https://github.com/OFS/ofs-n6001/tree/release/ofs-2023.1/eval_scripts OFS https://github.com/OFS OFS GitHub page https://ofs.github.io DFL Wiki https://github.com/OPAE/linux-dfl/wiki"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/n6001/doc_modules/Glossary/","title":"Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel Max10 or Intel Cyclone10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implemented on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Management Component Transport Protocol MCTP A standardized model for communication with management controllers. Defines the transport protocol carrying PLDM messages through the BMC. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE-SDK The OPAE-SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Level Data Model PLDM A specification for reporting telemetry data to the host, such as board temperature, voltage, and current. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/doc_modules/Notices_%26_Disclaimers/","title":"Notices & Disclaimers","text":""},{"location":"hw/n6001/doc_modules/Notices_%26_Disclaimers/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/n6001/doc_modules/n6001_wt_program_fpga_via_jtag/","title":"N6001 wt program fpga via jtag","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)] for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the n6001. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae FPGA SmartNIC N6001-PL is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the n6001, if different than your deployment machine.
-
Open the Quartus programmer GUI
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the n6001.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB014R24A2E2V device. The JTAG chain should show the divice.
-
Right click the AGFB014R24A2E2V row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGFB014R24A2E2V row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI. You can answer 'No' if a dialog pops up asking to save the 'Chain.cdf' file
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
"},{"location":"hw/n6001/doc_modules/n6001_wt_program_fpga_via_rsu/","title":"N6001 wt program fpga via rsu","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL with a BIN image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)] for instructions on setting up a deployment environment.
- This walkthrough requires a
BIN image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a BIN image. The image used for programming must be formatted with PACsign before programming. This is done automatically by the build script.
- This walkthrough requires a JTAG connection to the n6001. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae FPGA SmartNIC N6001-PL is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
Determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Use the OPAE fpgasupdate command to program a PACsign signed image to flash. The flash slot which will be programmed is determined by the PACsign header.
sudo fpgasupdate <IMAGE> <PCIe B:D.F>\n
-
Example: update User Image 1 in flash
sudo fpgasupdate ofs_top_page1_unsigned_user1.bin 98:00.0\n
-
Example: update User Image 2 in flash
sudo fpgasupdate ofs_top_page2_unsigned_user2.bin 98:00.0\n
-
Example: update Factory Image in flash
sudo fpgasupdate ofs_top_page0_unsigned_factory.bin 98:00.0\n
-
Use the OPAE rsu command to reconfigure the FPGA with the new image. You may select which image to configure from (User 1, User 2, Factory).
sudo rsu fpga --page <PAGE> <PCIe B:D.F>\n
-
Example: configure FPGA with User 1 Image
sudo rsu fpga --page user1 98:00.0\n
-
Example: configure FPGA with User 2 Image
sudo rsu fpga --page user2 98:00.0\n
-
Example: configure FPGA with Factory Image
sudo rsu fpga --page factory 98:00.0\n
"},{"location":"hw/n6001/doc_modules/n6001_wt_set_up_jtag/","title":"N6001 wt set up jtag","text":"Perform the following steps to set up a JTAG connection to the Intel\u00ae FPGA SmartNIC N6001-PL.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)] for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
- This walkthrough requires an [Intel FPGA Download Cable II].
Steps:
-
Set the board switches to dynamically select either the Agilex\u00ae 7 FPGA or MAX\u00ae 10 device on the JTAG chain.
- Set SW1.1=ON as shown in the next image. The switches are located at the back of the Intel\u00ae FPGA SmartNIC N6001-PL.
-
The Intel\u00ae FPGA SmartNIC N6001-PL has a 10 pin JTAG header on the top side of the board. Connect an Intel\u00ae FPGA Download II Cable to the JTAG header of the Intel\u00ae FPGA SmartNIC N6001-PL as shown in picture below. This picture shows the Intel\u00ae FPGA SmartNIC N6001-PL card installed in the middle bay, top slot of a SuperMicro\u00ae SYS-220HE-FTNR server where the lower slot does not have card installed allowing the Intel\u00ae Download II cables to pass through removed the slot access.
Note: If using the Intel FGPA download Cable on Linux, add the udev rule as described in [Intel FPGA Download Cable Driver for Linux].
-
Set the JTAG chain to select the ${{ env.DEVICE }} as the target by writing to the JTAG enable register in the BMC (Register 0x378). This is done via PMCI registers 0x2040C and 0x20400.
Note: The commands below are targeted to a board with PCIe B:D.F of 98:00.0. Use the correct PCIe B:D.F of your card.
sudo opae.io init -d 0000:98:00.0 $USER\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x2040c 0x100000000\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:98:00.0\n
Note: To later re-direct the JTAG back to the MAX 10 device, execute the following commands.
sudo opae.io init -d 0000:b1:00.0 $USER\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x2040c 0x000000000\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:b1:00.0\n
Optionally, rather than dynamically commanding Agilex\u00ae 7 FPGA/MAX10 selection with the PMCI register settings, you can fix the selection with the following switch settings shown in the table below:
SW1.1 SW2 JTAG Target OFF OFF Agilex\u00ae 7 FPGA OFF ON MAX\u00ae 10 FPGA ON X Agilex\u00ae 7 FPGA if BMC register 0x378=0x1 ON X MAX\u00ae 10 FPGA if BMC register 0x378=0x0 -
Use the jtagconfig tool to check that the JTAG chain contains the AGFB014R24A2E2V device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
TBD\n
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/","title":"Mnl fim ofs n6001","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#shell-technical-reference-manual-ofs-for-agilex-7-pcie-attach-fpgas","title":"Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1-overview","title":"1 Overview","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#11-about-this-document","title":"1.1 About this Document","text":"This document describes the hardware architecture for the PCIe attach reference FIMs of the Open FPGA Stack (OFS) targeting the Agilex\u00ae FPGA. After reviewing this document you should understand the features and functions of the components that comprise the FPGA Interface Manager (FIM), also known as the \"shell\". Different target boards have different default FIM configurations.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#12-glossary","title":"1.2 Glossary","text":"This table defines some of the common terms used when discussing OFS.
Table 1-1 Glossary Table
Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#13-introduction-to-open-fpga-stack","title":"1.3 Introduction to Open FPGA Stack","text":"The Open FPGA Stack (OFS) is a modular infrastructure of hardware platform components, open source upstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS provides a framework of FPGA synthesizable code, simulation environment and synthesis/simulation scripts. The key components of OFS include:
- Target development platforms such as Intel-branded Programmable Acceleration Cards (PACs), Acceleration Development Platforms (ADPs) and third-party platforms.
- Board Management Controller RTL and firmware that supports telemetry monitoring and capability for remote configuration updates.
- Source accessible, modular FPGA Interface manager (FIM) RTL with a UVM infrastructure unit tests that can be leveraged for your own custom FIM design. The FIM can be thought of as the FPGA shell that provides the I/O ring and timing closed management components for the FPGA.
- Basic building blocks for interconnect and PF/VF translation and arbitration; Platform Interface Manager (PIM) which provides Avalon\u00ae bus compliant interfaces.
- AFU examples both in the git repository and workload examples provided by 3rd party vendors.
- A OneAPI acceleration support package (oneapi-asp) that provides a bridge layer that is used by OneAPI runtime to communicate with the kernel.
- Unit level simulation test suite
- System level simulation through a unified verification methodology (UVM)
- OPAE software development kit (APIs, upstreamed Linux drivers and software tools)
- Support for other frameworks to be built on top of the OPAE such as DPDK
These components are available under the https://github.com/OFS site.
The OFS hardware repository supports hardware development and simulation. Repositories for OFS high level design support and board management controller RTL and firmware source code are also provided. These repositories can be found in the Altera Opensource Technology GitHub location.
Table 1-2 OFS Hardware Repositories
Repository Contains ofs-agx7-pcie-attach Contains FIM or shell RTL, automated compilation scripts, and unit tests and UVM framework. oneapi-asp Contains the hardware and software components you need to develop your own OneAPI board support package ofs-platform-afu-bbb Contains the files and scripts to build the platform interface manager. ofs-examples-afu Contains AFU examples you can use. ofs-bmc 1 Provides the OFS Board Management Controller RTL, firmware, scripts and collateral targeting the Intel\u00ae FPGA SmartNIC N6001-PL which can be leveraged for your own OFS design. 1 Access to BMC repositories requires entitlement access. To request access, please contact your local Altera sales representative.
Table 1-3 OFS Software Repositories
OPAE Git Repository Folder Contains opae-sdk Contains the files for building and installing OPAE SDK from source. linux-dfl Contains OFS Linux drivers that are being upstreamed to the Linux kernel. linux-dfl-backport Contains the backport version of the linux-dfl kernel driver for FPGA devices. opae-sim Contains the files for an AFU developer to build the Accelerator Functional Unit Simulation Environment (ASE) for workload development. Providing the hardware and software source code and supporting test frameworks in a GitHub repository allows you to easily customize your designs with the latest versions.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14-ofs-features","title":"1.4 OFS Features","text":"The OFS architecture within the FPGA comprises two partitions:
- FPGA Interface Manager (FIM)
- Accelerator Functional Unit (AFU)
The FIM or shell provides platform management functionality, clocks, resets and interface access to the host and peripheral features of the acceleration platform. The FIM architecture along with the supporting OPAE software supports features such as partial reconfiguration and virtualization. The FIM provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 datapath interface. The FIM resides in the static region of the FPGA.
The AFU partition is provided for custom acceleration workloads and may contain both static and partial reconfiguration regions.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#141-fpga-interface-manager-fim","title":"1.4.1 FPGA Interface Manager (FIM)","text":"The primary components of the FPGA Interface Manager, or shell, of each target board's default reference design are given in the following table:
Table: Default FIM Components
FIM Component SmartNIC N6001-PL SmartNIC N6000-PL F-Series Development Kit I-Series Development Kit PCIe Subsystem \u2714 \u2714 \u2714 \u2714 HSSI Subsystem \u2714 \u2714 \u2714 \u2714 Memory Subsystem \u2714 \u2716 [1] \u2714 \u2714 Hard Processor System \u2714 \u2714 \u2714 \u2716 [2] Reset Controller \u2714 \u2714 \u2714 \u2714 FPGA Management Engine \u2714 \u2714 \u2714 \u2714 AFU Peripheral Fabric for AFU accesses to other interface peripherals \u2714 \u2714 \u2714 \u2714 Board Peripheral Fabric for master to slave CSR accesses from Host or AFU \u2714 \u2714 \u2714 \u2714 Platform Management Controller Interface (PMCI) to the board management controller \u2714 \u2714 \u2716 [3] \u2716 [3] [1] The n6000 default shell design does not have the memory subsystem enabled. It can be enabled by following the instructions in [Section 4.7.2 of the PCIe Attach F-Series (P-Tile/E-Tile) Shell Developer Guide: Add or remove the Memory Sub-System [2] The HPS is not enabled in the FIM for the I-Series Development Kit [3] The F-Series Development Kit and I-Series Development Kit do not use the OFS BMC, and therefore do not use the PCMI.
The AFU Region provides design space for custom workloads and contains both static and partial reconfiguration regions. Partial reconfiguration allows you to update your specific logic blocks or entire workload while the rest of your static design is still in operation.
Note that the BMC RTL and firmware that works with this OFS design provided in a separate entitled repository. Please email ofs.github@intel.com if you would like to use our BMC code for your own design.
Figure: OFS FIM for Intel\u00ae FPGA SmartNIC N6001-PL Block Diagram
Figure: OFS FIM for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) Block Diagram
Figure: OFS FIM for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) Block Diagram
The table provides an overview of the OFS features targeting the Agilex\u00ae 7 FPGA. This reference FIM (shell) is a starting point for your custom FPGA design. With this initial starting point, you can add or subtract interfaces or ports to different Agilex devices.
Table 1-4 OFS FIM Features
Key Feature SmartNIC N6001-PL F-Series Development Kit I-Series Development Kit PCIe P-tile PCIe Gen4x16 F-tile PCIe Gen4x16 R-tile PCIe 1xGen5x16R-tile PCIe 2xGen5x8R-tile PCIe 1xGen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand 5 physical functions/3 virtual functions with ability to expand 5 physical functions/3 virtual functions with ability to expand Memory 5 DDR Channels:\u2022 One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each\u2022 Four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB 3 DDR Channels:\u2022 One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 2400 MHz, 1GB each\u2022 Two Fabric DDR4 banks, x64 (no ECC), 2400 MHz, 8GB 4 Fabric DDR4 channels, x64 (no ECC), 2666 MHz, 8GB Ethernet \u2022 N6001-PL: 2x4x25GbE, 2x4x10GbE, or 2x100GbE \u2022 N6000-PL: 4x100GbE 2x4x25GbE 2x4x25GbE, 2x200GbE, 2x400GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Not enabled Configuration and Board Manageability \u2022 FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration)\u2022 Platform Controller Management Interface (PMCI) Module for Board Management Controller \u2022 FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) \u2022 FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported Supported Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support \u2022 Linux DFL drivers targeting OFS FIMs\u2022 OPAE Software Development Kit\u2022 OPAE Tools \u2022 Linux DFL drivers targeting OFS FIMs\u2022 OPAE Software Development Kit\u2022 OPAE Tools \u2022 Linux DFL drivers targeting OFS FIMs\u2022 OPAE Software Development Kit\u2022 OPAE Tools Target Board \u2022 Intel\u00ae FPGA SmartNIC N6001-PL\u2022 Intel\u00ae 7 FPGA SmartNIC N6000-PL Agilex\u00ae 7 7 FPGA F-Series Development Kit (2x F-Tile) Agilex\u00ae 7 7 FPGA I-Series Development Kit (2xR-Tile, F-Tile)"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#subsystem-interfaces","title":"Subsystem Interfaces","text":"The PCIe, Memory and Ethernet interfaces in this design use a new flexible subsystem design that provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 interface. To access these FPGA IP Subsystem documents. Please go to the links below: * AXI Streaming IP for PCI Express User Guide * Memory Subsystem Intel FPGA IP User Guide * Ethernet Subsystem Intel FPGA IP User Guide (public document)
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#hard-processor-system-hps","title":"Hard Processor System (HPS)","text":"The HPS SoC contains a 64-bit quad core ARM\u00ae Cortex\u00ae-A53 MPCore with a variety of integrated modules such as on-chip RAM, Ethernet, USB, UARTs and SPI controllers and memory controllers. For more information about the Agilex HPS, please refer to the Agilex Hard Processor System Technical Reference Manual.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FIM contains only one FME, regardless of the number of host interfaces to the FIM. The FME provides management features for the platform and controls reset and loading of the AFU into the partial reconfiguration region of the FPGA.
Any feature, such as a memory interface or global error control that you want to control through FME, must expose its capability to host software drivers. New features are exposed to the FME by adding a device feature header (DFH) register at the beginning of the feature's control status register (CSR) space. The FME CSR maps to physical function 0 (PF0) Base address register 0 (BAR0) so that software can access it through a single PCIe link. For more information about DFHs, refer to the FPGA Device Feature List Framework Overview.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#streaming-datapath","title":"Streaming Datapath","text":"The FIM implements an AXI4-Stream bus protocol for data transfer in the FIM. AXI4-Stream channels send data packets to and from the host channel IP without data abstraction. Memory-mapped I/O (MMIO) CSR accesses are routed to the ST2MM module, which converts the AXI4-Stream to an AXI4 memory-mapped protocol.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#virtualization","title":"Virtualization","text":"This design supports virtualization by making use of the virtualization functionality in the PCIe Hard IP and mapping packets to the appropriate physical or virtual function through a PF/VF multiplexer.
The reference FIM example enables 5 PFs and 3 VFs by default; however, you may extend your configuration to whatever the PCIe Hard IP can support or your application requires.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#142-afu","title":"1.4.2 AFU","text":"An AFU is an acceleration workload that interfaces with the FIM. The AFU boundary in this design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required for partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components:
- ST2MM module.
- Null Host exerciser (HE_NULL) stub.
- PCIe loopback host exerciser (HE-LB).
- HSSI host exerciser (HE-HSSI).
- Memory Host Exerciser (HE-MEM).
- Traffic Generator to memory (HE-MEM-TG). Port Gasket (PRG).
- HPS Copy Engine.
- Arm\u00ae AMBA\u00ae 4 AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
For this design the PF/VF Mux provides the following mappings (found in src/afu_top/mux/top_cfg_pkg.sv):
Table 1-5 PF/VF Mapping
Module PF/VF Mapping AXI4 Stream to Memory Mapped Module (ST2MM) PF0 Memory Host Exerciser (HE_MEM) PF0-VF0 HSSI Host Exerciser (HE_HSSI) PF0-VF1 Memory Traffic Generator (HE_MEM_TG) PF0-VF2 Null Host exerciser (HE_NULL) stub PF1-VF0 PCIe Loopback (HE_LB) PF2 Null Host exerciser (HE_NULL) stub PF3 HPS Copy Engine Module PF4 The figure below highlights the AFU portion of the OFS block diagram for the SmartNIC N6001-PL as an example.
Figure: SmartNIC N6001-PL AFU Diagram
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#143-platform-interface-manager","title":"1.4.3 Platform Interface Manager","text":"The PIM provides a way to abstract the AXI4-Stream interface to the AFU by providing a library of shims that convert the host channel native packet into other protocols such as AXI4 memory-mapped, Avalon\u00ae streaming (Avalon-ST) or Avalon\u00ae memory-mapped (Avalon-MM).
The FPGA or AFU developer implements these interface abstractions in the AFU region (afu_main) of the design.
For more information, refer to Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#144-platform-feature-discovery","title":"1.4.4 Platform Feature Discovery","text":"This reference design comes with specific FPGA drivers that are upstreamed to linux-dfl. These drivers abstract the hardware and operating system specific details of the platform to the host.
The FIM implements a device feature header (DFH) at the beginning of each host-discoverable feature in a linked list format that allows an FPGA platform driver running on the host to discover FME, port, and AFU features.
You must implement a 64-bit DFH Device Feature Header register at the beginning (first 8B aligned address) of the feature CSR space for a new feature to be discovered or enumerated by a driver.
During host discovery, the driver traverses the DFH of the first feature from the first address on PF0 BAR0. Based on the information in the DFH, a driver can determine the CSR address range of the feature and other associated details. The end of the DFH contains a \"next DFH offset\" field that points the driver to the DFH of the next feature.
The software must continue traversing the linked list until it sees the EOL (End-Of-List) bit set to 1 in the \"next DFH offset\" field it inspects. A 1 indicates this is the last feature in the feature set. The figure below gives a simple illustration of the feature discovery by traversing the DFH registers. This model is similar to how PCIe enumeration occurs.
Figure 1-4 Device Feature Header Linked List Traversal
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#145-ofs-reference-design","title":"1.4.5 OFS Reference Design","text":"OFS provides FIM designs you can use as a starting point for your own custom design. These designs target a specific programmable acceleration card or development kit and exercise key FPGA device interfaces.
FIM designs are released to [https://github.com/OFS/ofs-agx7-pcie-attach] for evaluation and use. The provided reference designs can target the following boards:
- Intel\u00ae FPGA SmartNIC N6001-PL
- Intel\u00ae FPGA SmartNIC N6000-PL
- Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
- Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#146-fim-simulation","title":"1.4.6 FIM Simulation","text":"OFS provides unit tests and a UVM environment for the FIM and a framework for new feature verification. UVM provides a modular, reusable, and scalable testbench structure by providing an API framework that can be deployed across multiple projects.
The FIM testbench is UVM compliant and integrates third-party verification IPs from Synopsys that require a license.
Verification components include:
- FIM monitor to detect correct design behavior
- FIM assertions for signal level integrity testing
- Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity
- FIM coverage to collect functional data
The verification infrastructure can be found in the verification directory for evaluation and use. Please refer to the UVM Simulation User Guide: OFS for Agilex\u00ae 7 PCIe Attach for more information.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#2-ofs-high-level-architecture","title":"2 OFS High Level Architecture","text":"OFS provides distinct data paths that simplify the design and integration process for adding or removing interface modules:
- High Bandwidth data path for AFU-attached high-performance peripherals (HSSI, Memory, HPS, workload).
- Low Bandwidth data path for OFS management and slow peripheral components (JTAG, I2C, SMBus).
- AFU Peripheral Fabric (APF) to Board Peripheral Fabric (BPF) path to communicate with interface control and status registers (CSRs) and board components.
- Peer-to-peer datapath between AFU components.
- Peer-to-peer datapath between BPF components.
Depending on your design goals, you can present peripherals to software as:
- OFS managed peripherals with a device feature header that is part of a device feature list.
- Native driver managed peripherals that are exposed through an independent physical function or virtual function.
Figure 2-1 OFS Datapath Structure
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#3-pcie-subsystem","title":"3 PCIe Subsystem","text":"The FIM's PCIe Subsystem is a hierarchical design that targets the PCIe Hard IP and is configured to support Gen4/Gen5 speeds. The default FIM uses the AXI Streaming Intel FPGA IP for PCIe Express. The IP supports SR-IOV and is configured to provide 5 PFs and 3 VFs by default. Native PCIe TLP packets are sent through the PCIe using Arm\u00ae AMBA\u00ae 4 AXI4 Stream Protocol. Tag allocation and management for packets sent from the application to the host are done by the PF/VF Mux that is part of the AFU region.
Note that this default PCIe-SS does not support Arm\u00ae AMBA\u00ae 4 AXI4 Data Mover functional mode. If Data Mover mode is required, you must instead build the FIM using the Intel FPGA IP Subsystem for PCI Express by changing build settings prior to FIM compilation; refer to the Shell Developer Guides for instructions on making this change.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs
Figure 3-1 AXI Streaming Intel FPGA IP for PCIe Express Block Diagram
Some key features of the PCIe interface are:
Table: PCIe Subsystem OFS Default Configuration
Feature n6001 n6000 fseries-dk iseries-dk Mode PCIe Gen4x16 PCIe Gen4x16 PCIe Gen4x16 PCIe Gen5x16 Port Mode Native Endpoint Native Endpoint Native Endpoint Native Endpoint SR-IOV 5PFs, 3VFs 5PFs, 3VFs 5PFs, 3VFs 5PFs, 3VFs MSI-X Support Yes Yes Yes Yes Profile Basic Basic Basic Basic TLP Bypass No No No No Header Packing Scheme Simple Simple Simple HIP Native Mode Data Width 512-bit (64-byte) 512-bit (64-byte) 512-bit (64-byte) 1024-bit (128-byte) AXI-ST Clock Frequency 500 MHz 350 MHz 500 MHz 500 MHz Tags Supported 256 256 256 768 Reordering No reordering of requests, no completion reordering No reordering of requests, no completion reordering No reordering of requests, no completion reordering No reordering of requests, no completion reordering Maximum Payload Size 512 Bytes 512 Bytes 512 Bytes 512 Bytes Memory Requests Supported 1CL, 2CL, 4CL TBD TBD TBD MMIO transaction Size 4B, 8B TBD TBD TBD Control Shadow Interface Enabled Enabled Enabled Enabled Completion Timeout Interface Enabled Enabled Enabled Enabled The PCIe PF, VF and Base Address Register (BAR) configuration can be modified in the PCIe Subsystem Platform Designer GUI interface. The current implementation for the OFS FIM for Agilex FPGA is as follows:
Table 3-1 Function and BAR Table for OFS for Agilex FPGA
PF VF Feature BAR BAR Size PF0 - OFS Managed Peripherals (PCIe, Memory, Ethernet) BAR0 512 KB PF0 - AFU Peripherals BAR0 256 KB PF0 - Board Peripherals BAR0 256 KB PF0 - MSI-X BAR4 16 KB PF0 VF0 Memory Host Exerciser (HE-MEM) BAR0 256 KB PF0 VF1 HSSI Host Exerciser (HE-HSSI) BAR0 256 KB PF0 VF2 Memory Traffic Generator (HE-MEM-TG) BAR0 256 KB PF1 Null Host exerciser (HE_NULL) BAR0 4 KB PF2 PCIe Loopback (HE-LB) BAR0 256 KB PF3 Null Host exerciser (HE_NULL) BAR0 4 KB PF4 HPS Copy Engine BAR0 4 KB"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#31-pcie-subsystem-header-format","title":"3.1 PCIe Subsystem Header Format","text":"The first 32 bytes of the TLP from the PCIe subsystem denotes the PCIe header. The default PCIe-SS in the OFS FIM for Agilex FPGA only supports the Power User Mode Header. If using Data Mover Mode with the old IP, then the tuser_vendor[0] bit on the AXI4-Stream channel indicates the header format of the TLP; tuser_vendor[0] =0 indicates Power User Mode header and tuser_vendor[0] =1 indicates Data Mover Mode header.
For more detailed information about the PCIe Subsystem, see the PCIe Subsystem FPGA User Guide.
Table 3-2 PCIe Subsystem Header Format Support for OFS for Agilex FPGA
Direction Type Power User Data Mover Host to Endpoint MWr, MRd Yes No Host to Endpoint CPL/CPLd Yes Yes Host to Endpoint Msg Yes No Endpoint to Host MWr, MRd Yes Yes Endpoint to Host Intr Yes (MWr) Yes Endpoint to Host CPL/CPLd Yes Yes Endpoint to Host Msg Yes Yes"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#32-pcie-subsystem-interface-module","title":"3.2 PCIe Subsystem Interface Module","text":"The PCIe Subsystem Interface module (/ipss/pcie/rtl/pcie_ss_if.sv), provides the supporting interface between software and the PCIe subsystem.
The interface module provides the following:
- Device Feature Header Registers
- Control and Status Registers
- Indirect access to PCIe subsystem CSR registers through a CSR mailbox in the PCIe Subsystem Interface.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#33-pcie-request-cycles","title":"3.3 PCIe Request Cycles","text":"For Host read request cycles using the OFS FIM for Agilex FPGA:
- All requests in the RX direction will be MMIO.
- Requester ID from the request does get sent to the AFU. It is the AFU's responsibility to send back a completion to the host with the correct completer ID.
- Prefix is not supported.
- Memory Mapped (MM) Mode is not supported.
- Slot Number is 0.
- Base address is not sent to the AFU.
- Local Address field is not used.
For AFU/application request cycles using the OFS FIM for Agilex FPGA:
- All requests in the TX direction will be Memory Read/Write.
- The tag must be generated by the AFU/application.
- Prefix is not supported.
- MM Mode is not supported.
- Slot Number is 0 (non-0 only for switch)
- VF Active, VF number and PF number are obtained from Data Mover Header Packet.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#34-pcie-completion-cycles","title":"3.4 PCIe Completion Cycles","text":"For Host completion cycles using the OFS FIM for Agilex FPGA:
- All completions in the RX direction will be Data Completions.
- Prefix is not supported.
- MM Mode is not supported.
- Slot Number is 0.
- Data packet responses (for Memory Read requests from AFU) from the PCIe SS may come out of order when the size is >64B.
For AFU/application completion cycles using the OFS FIM for Agilex FPGA: * All requests in the TX direction will be Memory Read/Write. * Requester ID is generated within the FIM. * That tag must be generated by the AFU/application. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0. * VF Active, VF Number and PF number are obtained from the Data Mover Header Packet.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#4-platform-interface-manager","title":"4 Platform Interface Manager","text":"The FIM interfaces to an application in the AFU region through AXI4-Stream channels. This format allows the AFU to access the host channel's raw interface without any translation.
As a FIM developer, you have the option to provide the raw data format associated with the host interface channel to the workload or AFU developer or you can provide an intermediate protocol using Platform Interface Manager Components or your own custom interface.
If you expose the raw AXI4-Stream interface of the FIM, workload developers also have the option to convert to a desired protocol using the PIM resources as well.
Refer to the Workload Developer Guide for more information on using the PIM in your design.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#5-interconnect-fabric","title":"5 Interconnect Fabric","text":"There are three types of interconnect fabric in the OFS FIM design: * AXI4-Stream PF/VF mux/demux fabric * AFU Peripheral Fabric (APF) * Board Peripheral Fabric (BPF)
Figure 5-1 Interonnect Fabric Diagram
TLP packets sent from upstream PCIe Subsystem on AXI4-Stream channel are demultiplexed in the AXI4-Stream PF/VF mux/demux fabric and routed to the respective PF/VF function based on the PF/VF information in the TLP header, such as vf_active or the PF/VF number. In the opposite direction, TLP packets from downstream PF/VF function are muxed in the fabric and sent to PCIe subsystem over AXI4-Stream channel.
All host MMIO requests targeting PF0 BAR0 are routed to the ST2MM module. The ST2MM converts MMIO TLP packets into AXI-Lite memory requests and places the requests onto AFU Peripheral Fabric (APF). AFU peripherals, such as OFS managed AFU features and ST2MM, and Board Peripheral Fabric (BPF) are interconnected by APF. The BPF is the interconnect fabric one hierarchy below APF which connects all the board peripherals. Both APF and BPF allow multiple AXI4-Lite master and slave interconnect topology.
If you are modifying the APF or BPF connections, you must re-generate the fabrics. OFS provides a helper script to perform this task.
For modifying the PF/VF mux you may edit the PCIe OFS Settings (OFSS) file to implement the desired PF/VF settings.
For details on these modifications, please refer to the shell developer guide for your target board:
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#51-afu-peripheral-fabric-apf","title":"5.1 AFU Peripheral Fabric (APF)","text":"The AFU Peripheral Fabric (APF) is a 64-bit Arm AXI4-lite compliant interconnect fabric that connects AFU peripheral modules to board peripheral modules through the Board Peripheral Fabric (BPF).
The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by the APF is listed below. All components are mapped to PF0 BAR0 and implement AXI-lite slave interface. Note that none of the features in the APF mapping are designed to act as a master.
Table 5-1 APF Address Mapping
Address Size (Byte) Feature 0x00000\u20130x3FFFF 256K Board Peripherals(See BPF address mapping table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket: PR Gasket DFH (4K)\u00a0\u00a0\u00a0\u00a0Control and status (4K)\u00a0\u00a0\u00a0\u00a0Port DFH (4K)\u00a0\u00a0\u00a0\u00a0User Clock (4K)\u00a0\u00a0\u00a0\u00a0Remote STP (52K) 0x80000 \u2013 0x80FFF 4K AFU Error Reporting"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#52-board-peripheral-fabric-bpf","title":"5.2 Board Peripheral Fabric (BPF)","text":"The Board Peripheral Fabric is the 64-bit AXI4-Lite compliant interconnect fabric that connects board peripheral modules to APF. The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by BPF is listed below. All components are mapped to PF0 BAR0 and implement AXI4-lite slave interface. The Master column indicates if a component also implements AXI4-lite master interface which can send requests to the BPF.
Table 5-2 BPF Address Mapping
Address Size (Byte) Feature 0x00000 \u2013 0x0FFFF 64K FME (FME, Error, etc) 0x10000 \u2013 0x10FFF 4K PCIe Interface 0x11000 \u2013 0x11FFF 4K Reserved 0x12000 \u2013 0x12FFF 4K QSFP Controller 0 0x13000 \u2013 0x13FFF 4K QSFP Controller 1 0x14000 \u2013 0x14FFF 4K Ethernet Subsystem 0x15000 - 0x15FFF 4K External Memory Interface 0x16000 - 0x19FFF 40K Reserved 0x20000 \u2013 0x3FFFF 128K PMCI Controller"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#53-arm-amba-4-axi4-stream-pfvf-muxdemux","title":"5.3 Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux routes the PCIe TLP packets from the PCIe subsystem AXI4-Stream RX channel to downstream PF/VF based on the pf_num and vf_num information in the PCIe TLP header.
The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux arbitrates PCIe TLP packets from downstream PF/VF to the PCIe SS AXI-S TX channel. The PF/VF Mux/Demux is an M X N switch that allows any M port to target any N port, and any N port to target any M port, where M is the number of host/upstream ports, and N is the numbers functions/downstream ports.
The fpga top package file, /src/afu_top/mux/top_cfg_pkg.sv, contains the PF/VF parameters and mappings.
Figure 5-2 PF/VF Mux
The PF/VF mux integration is part of afu_top (/src/afu_top/mux/top_cfg_pkg.sv). There are two independent TX PF/VF MUX trees, labeled \"A\" and \"B\".
Both an A and a B port are passed to each AFU component with a unique PF/VF. You can design your AFU components to send all requests to the primary A port or partition requests across both A and B ports. A typical high-performance AFU sends read requests to the B port and everything else to the A port, giving the arbiter freedom to keep both the host TX and RX channels busy.
In the reference FIM provided for Agilex OFS, the A and B TX trees have been multiplexed down to a single channel for A and another for B. The A/B multiplexer merges them into a single TX stream that will be passed to the tag remapper.
The tag remapper provides unique tags as required by the PCIe specification. Tags are not provided by the PCIe Subsystem FPGA IP. When creating your own AFU you can leverage this module to generate unique tags.
Note that the primary PF/VF Mux A supports RX and TX ports. For the secondary PF/VF Mux B only TX ports are supported and the RX input to the Mux is tied off.
The default mapping is shown below:
Table 5-3 PF/VF Mapping
Module PF/VF Mapping AXI4 Stream to Memory Mapped Module (ST2MM) PF0 Memory Host Exerciser (HE_MEM) PF0-VF0 HSSI Host Exerciser (HE_HSSI) PF0-VF1 Memory Traffic Generator (HE_MEM_TG) PF0-VF2 Virtio Loopback Stub (Virtio_LB) PF1-VF0 PCIe Loopback (HE_LB) PF2 Virtio Loopback Stub (Virtio_LB) PF3 HPS Copy Engine Module PF4 For information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#54-afu-interface-handler","title":"5.4 AFU Interface Handler","text":"The AFU Interface Handler resides inline between the PCIe AXI4-Stream Adapter and the AXI4-Stream PF/VF Demux/Mux logic. Its main function is to provide: * Unique PCIe tags \u2013 Each PCIe transaction shares the 512 tags across all VFs in the AFU region * AFU error logging for all VFs in the AFU region
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#541-unified-tag-remapping","title":"5.4.1 Unified Tag Remapping","text":"When a FPGA function sends out a read cycle, it allocates a unique tag which is subsequently used to identify the read completion. The tag is considered busy; it cannot be assigned to another read cycle until read completion. While a tag may be unique within a unit, two different units could unknowingly send out two read cycles of the same tag. The PCIe subsystem requires unique tags for all read cycles irrespective of their origins. Therefore, a mechanism is needed to uniquify tag globally across different units.
OFS contains a tag remapper (/ofs-fim-common/src/common/tag_remap/tag_remap.sv) that intercepts the read cycle, finds a globally unique tag, and replaces the original tag value. It also restores the original tag value when returning completion to the read requester. tag_remap is placed between the AXI4-Stream interface of the PCIE subsystem and the PF/VF Mux/Demux.
The logic is described as follows:
- A sub-module (
/ofs-fim-common/src/common/tag_remap/ofs_fim_tag_pool.sv) maintains a pool of available tags. - TX read requests are held until a tag is available from the pool by setting tvalid=0 to the host, and tready=0 to the PF/VF Mux/Demux.
- When a TX read is dispatched, the tag is marked busy in the pool.
- The original tag is stored in tag_reg, so it can be recovered when returning a completion to the unit/function.
- Because completion to a read request can split into multiple smaller transfer sizes, responses are monitored and the final completion is detected using PCIe TLP rules.
- Tags are released in the pool only when all requested data are transferred.
- When the completion returns, the original tag is restored from
tag_reg.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#542-afu-error-handling","title":"5.4.2 AFU Error Handling","text":"In this OFS design, the AFU Interface Handler handles error logging for all VFs in the AFU. Errors handled are as follows
Table 5-4 AFU Error Descriptions
Checker Field Description AFU protocol checker (PCIe TLP) Malformed TLP AFU PCIe TLP contains unsupported format type AFU protocol checker (PCIe TLP) MaxPayloadError AFU memory write payload size exceeds max_payload_length limit AFU protocol checker (PCIe TLP) MaxReadReqSizeError AFU memory read payload size exceeds max_read_request_size limit AFU protocol checker (PCIe TLP) MaxTagError AFU memory read request tag value exceeds the maximum supported tag count AFU protocol checker (PCIe TLP) UnalignedAddrErr The address field in AFU memory write/read request TLP is not DW-aligned. AFU protocol checker (PCIe TLP) UnexpMMIOResp AFU is sending a MMIO read response with no matching MMIO read request. AFU protocol checker (PCIe TLP) MMIOTimedOut AFU is not responding to a MMIO read request within the pre-defined response timeout period. AFU protocol checker (PCIe TLP) MMIODataPayloadOverrun The number of data payload sent by AFU for a MMIO response (cplD) is more than the data length specified in the response. AFU protocol checker (PCIe TLP) MMIOInsufficientData The number of data payload sent by AFU for a memory write request is more than the data length specified in the request. AFU protocol checker (PCIe TLP) TxMWrDataPayloadOverrun The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU protocol checker (PCIe TLP) TxMWrInsufficientData The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU Protocol Checker (Arm\u00ae AMBA\u00ae 4 AXI4) TxValidViolation Three checkers are implemented in the FIM to catch errors and protocol violations."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#55-tlp-to-axi4-lite-memory-mapped-bridge-st2mm","title":"5.5 TLP to AXI4-Lite Memory Mapped Bridge (ST2MM)","text":"ST2MM implements the following key features:
- Host MMIO bridge * Maps MMIO TLP packets received from the PCIe Subsystem over streaming interface to AXI4-Lite memory-mapped request. The memory-mapped request is sent to AFU or Board peripherals over APF and BPF. * Maps AXI4-lite MM response received from AFU or Board peripherals to TLP packets and send the packets over ST streaming channel to host HIA subsystem.
- Sends MMIO response of all 0\u2019s for MMIO read to unused BAR region.
- Interrupt * Sends interrupt packets to the PCIe subsystem when interrupt requests are received from the peripherals. Interrupts can be requested by a peripheral through a memory write to interrupt CSR registers in the ST2MM.
Figure 5-3 ST2MM Module
ST2MM implements both AXI4-lite master and slave interfaces that are connected to the designated slave and master port on APF. Host memory requests are sent on the ST2MM master interface to AFP where the requests are routed to the targeted peripherals.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#6-mmio-regions","title":"6 MMIO Regions","text":"The FIM and AFU expose their functionalities to the host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). An MMIO region is an address space within a base address register (BAR) region to which features are memory mapped.
For example, when a feature is mapped to an MMIO region, the CSR registers of that feature are located within the address range of that region. There can be multiple MMIO regions within a BAR region.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#61-feature-region","title":"6.1 Feature Region","text":"A group of related CSRs can be categorized as a feature region. A feature region is contained within a single PCIe BAR and cannot span across two BAR region boundaries.
A Device Feature Header (DFH) is a block of registers that mark the start of the feature region and sub-feature region, and you must place it at the first address of the region. Each DFH starts at 4KB boundary. A DFH register contains information that OPAE software requires to enumerate the feature. It also has an offset field that points to the next DFH in a feature list. OPAE software traverses the linked list of DFHs in each BAR region to discover all the features implemented on the platform.
The EOL field in a DFH Start marks the end of a DFH list and is only set in the DFH of the last feature in the feature list. The feature type field in the DFH is used to differentiate between the different types of feature region. Basic building blocks (BBB) and private features are always a child of an AFU or FPGA Interface Unit (FIU) and must be contained within an AFU or FIU, respectively.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#611-device-feature-header-dfh-structure","title":"6.1.1 Device Feature Header (DFH) Structure","text":"All DFHs must follow a specific structure to be compatible with OPAE software. Note that only features you want to be exposed to the OPAE software must have a DFH. For the latest information on DFH structure, please refer to the FPGA DFL Framework Overview.
6.2 Control and Status Registers
All the Control and Status Registers (CSRs) in the FIM are 64-bit registers with the following MMIO write, and MMIO read support.
Table 6-5: CSR MMIO Read and Write Support
Request Memory Attribute Payload size Memory Ordering MMIO Write UC 4B or 8B Strongly ordered MMIO Read UC 4B or 8B Strongly ordered The FIM does not reorder the MMIO requests or responses. For MMIO writes, there is no reordering of requests in FIM, and uncacheable (UC) ordering rules are followed. Similarly, for MMIO reads, there is no re-ordering of requests or responses in the FIM. An AFU may opt to re-order the MMIO read responses but the FIM does not enforce read response ordering.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#621-software-access-to-registers","title":"6.2.1 Software Access to Registers","text":" - Software accesses 64-bit registers as aligned quadwords. For example, to modify a field (bit or byte) in a 64-bit register, the entire quadword is read, the appropriate field(s) are modified, and the entire quadword is written back.
- When updating registers through multiple accesses (whether in software or due to hardware disassembly), certain registers may have specific requirements on how the accesses must be ordered for proper behavior. These are documented as part of the respective register descriptions.
- For compatibility with future extensions or enhancements, software must assign the last read value to all \u201cReserved and Preserved\u201d (RsvdP) fields when written. In other words, any updates to a register must be read so that the appropriate merge between the RsvdP and updated fields occurs. Also, software must assign a value of zero for \u201cReserved and Zero\u201d (RsvdZ) fields when written.
- PCIe locked operations to FPGA hardware registers are not supported. Software must not issue locked operations to access FPGA hardware registers.
In the following two cases, the FIM terminates MMIO Read requests by sending a completion with the data (CplD) specified below: * MMIO Timeout: This occurs when the AFU does not respond within a set timeout. The timeout value is currently configured to 512 pclks (clk_2x). In this case, the FIM returns all 1s.
- Illegal MMIO Accesses: This occurs when the read is accessing undefined registers in the FIM or if an AFU access violation. An example of an access violation is when a PF attempts to access the AFU when it is set to VF mode, or when a VF attempts to access the AFU when it is set to PF mode. In this case, the FME will returns all 0s.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#622-register-attribute-definition","title":"6.2.2 Register Attribute Definition","text":"Table 6-6: OFS Register Attribute Definitions
Attribute Expansion Description RW Read/Write This bit can be read or written by software. RO Read Only The bit is set by hardware only. Software can only read this bit. Writes do not have any effect. RW1C Read/ Write 1 to Clear Software can read or clear this bit. The software must write 1 to clear this bit. Writing zero to RW1C bit has no effect. Note that a multi-bit RW1C field may exist. In this case, all bits in the field are cleared if a 1 is written to any of the bits. RW1S Read/ Write 1 to Set Software can read this bit. Writing a 1 to the bit sets it to 1. Writing a 0 has no effect. It is not possible for software to set this bit to 0. The 1 to 0 transition can only be performed by HW. RW1CS Read/Write 1 to Clear Sticky Software can read and clear this bit. Writing a 1 to a bit clears it, while writing a 0 to a bit has no effect. This bit is only reinitialized to its default value by a power-on reset. RWD Read/Write Sticky across Hard Reset The bit can be read or written by SW. This bit is sticky or unchanged by any reset type, including Hard Reset. The bit gets cleared only with power on. *S Sticky across Soft Reset The bit will be sticky or unchanged by soft reset. These bits are only re-initialized to their default value by a power-on reset. *D Sticky across Hard Reset The bit is sticky or unchanged by or unchanged by any reset type, including hard reset. The bit gets cleared only with power on. Rsvd Reserved Reserved for future definitions. Currently don\u2019t care bits. RsvdP Reserved and Protected Reserved for future RW implementations. The software must preserve the value of this bit by read modify write. RsvdZ Reserved and Zero Reserved for future RW1C implementations. The software must write zero to this bit."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#623-csr-offset-in-bars","title":"6.2.3 CSR Offset in BARs","text":"The table below captures the FIM and AFU features in the supported BAR regions. The highlighted offset indicates the first DFH in the DFH list of a BAR region where device driver starts the DFH traversal.
Table 6-7: PF0 BAR0 Features
Offset Feature CSR set 0x0_0000 FME 0x0_1000 Thermal Management 0x0_3000 Global Performance 0x0_4000 Global Error 0x1_0000 PCIe 0x1_2000 QSFP0 0x1_3000 QSFP1 0x1_4000 HSSI (Ethernet) 0x1_5000 EMIF 0x2_0000 PMCI 0x4_0000 ST2MM 0x6_0000 UART 0x7_0000 Port Gasket PR logic 0x7_1000 Port Gasket Port interface 0x7_3000 Port Gasket Remote Signal Tap 0x8_0000 AFU Interface Handler- AFU Errors Table 6-8: PF0 BAR4 Features
Offset Feature CSR set 0x0_3000 MSI-X Vector Tables"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#7-clocks","title":"7 Clocks","text":"The following table provides external clock sources which correspond to pins on the FPGA that drive blocks of internal logic or are used as a reference clock for internal PLLs. The names in the table are used in the top.sv or are the logical clock names used by their respective functional blocks.
Table 7-1: External Clock Source
Clock Frequency Description N6001 fseries-dk iseries-dk SYS_REFCLK 100 MHz Reference clock to system IOPLL (sys_pll) which provides FIM system clocks. \u2714 \u2714 \u2714 PCIE_REFCLK0 100MHz PCIe reference clock 0 \u2714 \u2714 \u2714 PCIE_REFCLK1 100MHz PCIe reference clock 1 \u2714 \u2716 \u2714 qsfp_ref_clk 156.25 MHz Ethernet Reference Clock \u2714 \u2714 \u2714 ddr4_mem[0].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2714 \u2714 ddr4_mem[1].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2714 \u2714 ddr4_mem[2].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2716 \u2714 ddr4_mem[3].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2716 \u2714 ddr4_hps.ref_clk 150 MHz Refclk for HPS EMIF \u2714 \u2714 \u2716 sdm_config_clk | FPGA_OSC_CLK1 125 MHz SDM Configuration Clock \u2714 \u2714 \u2714 hps_refclk | HPS_OSC_CLK 25 MHz Refclk for HPS \u2714 \u2714 \u2716 Table 7-2: Internal Clocks
Internal clock generated by the IOPLL as outclk outputs.
outclk# Clock Frequency Description outclk0 clk_sys 500 MHz1 System clock. Primary clock for PCIe Datapath outclk1 clk_100m 100 MHz For RemoteSTP and user clock, or any logic that requires a 100 MHz clock. outclk2 clk_sys_div2 250 MHz1 System clock div2 outclk3 clk_ptp_slv 155.56 MHz Unused outclk4 clk_50m 50 MHz Drives Virtual UART outclk5 clk_sys_div4 125 MHz1 System clock div4 outclk6 clk_350m 333.33 MHz Unused 1 The system clock frequency can be changed using OFSS files at build time.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#8-reset","title":"8 Reset","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#81-reset-signals","title":"8.1 Reset Signals","text":"The FIM system reset signals are driven according to their respective requirements derived from their use cases. The behavioral code defining the reset operation is located in the file rst_ctrl.sv. The FIM System Reset Drivers table below provides a list of the input and feedback signals that compose the various resets.
Table 8-1: FIM System Reset Drivers
Reset Description PCIE_PERST_n pin Active low PCIe reset pin from the PCIe card edge that may be set by the host machine for cold or warm resets. nCONFIG pin Active low input to the FPGA that causes the FPGA to lose its configuration data, enter a reset state, and tri-state all I/O pins. Host software must reload the FPGA FIM after nCONFIG is activated. ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. pcie_reset_status Active high reset status from PCIe hard IP. When driven high, this signal indicates that the PCIe IP core is not ready for usermode. pll_locked Active high SYS IOPLL locked signal pcie_cold_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Cold Reset sequence has been completed. pcie_warm_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Warm Reset sequence has been completed. Upon power-on, the reset module in the FIM holds the FIM in reset until all the reset conditions are de-activated:
nPERST signal is asserted.- The
INIT_DONE pin is driven high to indicate core configuration is complete. - The SYS IOPLL is locked.
- The reset status from PCIe hard IP is de-asserted indicating the IP is ready for transfer.
The reset module places the FIM back into reset if any of these conditions becomes active again. The only way to invoke a system reset to the FIM after power-up is to de-assert the nPERST pin either by performing a warm reboot or through PCIe driver intervention. There are soft reset signals set aside to allow software to reset the Port, AFU and partial reconfiguration IP.
THe following table lists the reset outputs from the rst_ctrl.sv block.
\u200b Table 8-2: FIM System Reset Outputs
Reset Description ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. rst_n_sys System general reset synchronous to clk_sys. This is a warm reset of the FPGA logic. Sticky bits in the FME and other CSR registers will not be reset by this signal. rst_n_100m System general reset synchronous to clk_100m. rst_n_50m System general reset synchronous to clk_50m. rst_n_sys_div2 System general reset synchronous to clk_sys_div2. rst_n_ptp_slv System general reset synchronous to clk_ptp_slv. pwr_good_n Hardware reset conditioned by ninit_done and the pll_locked signal. The signal is generally set when power has been cycled or a physical hardware fault has occurred, requiring a reset of the FPGA logic. This signal is synchronous to clk_sys. pcie_cold_rst_ack_n Hardware reset conditioned by the pcie_reset_status which is a summary reset driven by the PCIe Hard IP logic tile on the FPGA die. This signal is synchronous to clk_sys. pcie_warm_rst_ack_n Soft reset conditioned by nPERST when the pcie_reset_status is not asserted, meaning a warm reset is desired. Cold reset sticky bits in the PCIe subsystem will not be reset by this signal."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#9-interrupts","title":"9 Interrupts","text":"The OFS platform supports interrupts through MSI-X feature. The MSI-X Interrupt feature handles FME and AFU interrupts. FME interrupts are primarily used to notify the host of error events happened in the FIM. When any of the bits in the FME error status registers is set, an interrupt request is sent to the MSI-X module. There are FME error status registers for OFS for Agilex FPGA features. An AFU sends interrupt to the MSI-X module in the PCIE SS on the AXI interrupt request channel. The MSI-X table entries and PBA vectors are implemented in the PCIE SS.
Please refer to the AXI Streaming IP for PCI Express User Guide for more information.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#10-external-memory-interface-emif","title":"10 External Memory Interface (EMIF)","text":"The table below shows the capabilities of the memory populated on the boards for each reference design. Note that ECC is only enabled by default for HPS memory channels. You may enable ECC on channels that support it by modifying the memory subsystem. Refer to the Shell Developer Guides for instructions.
Table 10-1 Memory Subsystem Configuration for OFS Agilex PCIe Attach Target Boards
Reference Design Target Board Memory Channel FPGA I/O Bank Width ECC Width ECC Default Speed Size n6001 0 2A x32 No ECC Disabled 2400 4 GB 1 3B x32 x8 Disabled 2400 4 GB 2 2B x32 No ECC Disabled 2400 4 GB 3 3A x32 x8 Disabled 2400 4 GB HPS 3D x32 x8 Enabled 2400 1 GB fseries-dk 0 2C, 2D x64 x8 Disabled 2400 8 GB 1 2E, 2F x64 No ECC Disabled 2400 8 GB HPS 3D x32 x8 Enabled 2400 8 GB iseries-dk 0 3C,3D x64 x8 Disabled 2666 8 GB 1 3A,3B x64 x8 Disabled 2666 8 GB 2 2C,2F x64 x8 Disabled 2666 8 GB 3 2B,2E x64 x8 Disabled 2666 8 GB The diagram below shows the general connectivity of the EMIF.
Figure 10-1: EMIF Interfaces
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#101-emif-csr","title":"10.1 EMIF CSR","text":"The CSRs for the memory subsystem reside at address 0x15000 in the BPF. The CSRs are defined in ofs-fim-common/src/fpga_family/agilex/mem_ss/xls/EMIF_CSR.xls.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#11-ethernet-subsystem","title":"11 Ethernet Subsystem","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#111-ethernet-subsystem-overview","title":"11.1 Ethernet Subsystem Overview","text":"The Ethernet Subsystem (hssi_ss) provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
The table below shows the validated Ethernet configurations for each target board.
Table 11-1: Validated Ethernet Configurations for example OFS FIMs for Agilex FPGAs
Reference Design Target Board Ethernet Configuration n6001 2x4x25 GbE 2x4x10 GbE 2x100 GbE fseries-dk 2x4x25 GbE iseries-dk 2x4x25 GbE 2x200 GbE 2x400 GbE You may use OFS Settings (OFSS) files to select one of these configurations at build time. Please refer to the Shell Developer Guides for instructions.
For more information on the Ethernet Subsystem IP, please refer to the Ethernet Subsystem Intel FPGA IP User Guide.
The table below describes how the QSFP lanes are mapped to the Ethernet Subsystem ports.
Table 11-2: Transceiver Subsystem Port Mapping
Ethernet-SSPort Number Reference Design Target Board n6001 fseries-dk iseries-dk 2x4x25 GbE 2x4x10 GbE 2x100 GbE 2x4x25 GbE 2x4x25 GbE 2x200 GbE 1x400 GbE 0 QSFPA0:3 25 GbE QSFPA0:3 10 GbE QSFPA0:3 100GCAUI-4 QSFPDD0:7 25 GbE QSFPDD10:7 25 GbE - - 1 25 GbE 10 GbE - 25 GbE 25 GbE - - 2 25 GbE 10 GbE - 25 GbE 25 GbE - - 3 25 GbE 10 GbE - 25 GbE 25 GbE - - 4 QSFPB0:3 25 GbE QSFPB0:3 10 GbE QSFPB0:3 100GCAUI-4 25 GbE 25 GbE - - 5 25 GbE 10 GbE - 25 GbE 25 GbE - - 6 25 GbE 10 GbE - 25 GbE 25 GbE - - 7 25 GbE 10 GbE - 25 GbE 25 GbE - - 8 - - - - - QSFPDD10:3 200GAUI-4 QSFPDD10:7 400GAUI-8 9 - - - - - - - 10 - - - - - - - 11 - - - - - - - 12 - - - - - QSFPDD14:7 200GAUI-4 - 13 - - - - - - - 14 - - - - - - - 15 - - - - - - - 16 N/A - - - - 17 N/A - - - - 18 N/A - - - - 19 N/A - - - - Figure 11-1: Transceiver Subsystem Block Diagram
A host exerciser, named he-hssi, is provided in the pr_slot of the AFU partition. The Ethernet Subsystem interface to the AFU has an Arm\u00ae AMBA\u00ae 4 AXI4 data and sideband interface.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#112-parameterization-options","title":"11.2 Parameterization Options","text":"The Ethernet Subsystem features are highly parameterized to provide the various features/configurations required for the different FIMs. The HSSI OFSS files found in ofs-agx7-pcie-attach/tools/ofss_config/hssi can be used to change the configuration of the HSSI-SS. You may also create your own OFSS file for custom configuration. The provided OFSS files are described in the following table:
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk The following parameters are defined in ofs-fim-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv:
- MAX_NUM_ETH_CHANNELS: This indicates how many maximum ethernet channels are supported for platfrom
- NUM_QSFP_PORTS_USED: Indicates number of QSFP cages on the board that are used by the target HSSI configuration
- NUM_ETH_CHANNELS: Number of ethernet ports used by target hssi configuration. E.g. for 8x25G, 8 HSSI ports are active on HSSI IP
- NUM_QSFP_LANES: Number of lanes per QSFP cage
- NUM_LANES: Number of XCVR Lanes per Port used by the configuration. For ex. for 100GCAUI-4, 4 lanes per HSSI port are used
- ETH_PACKET_WIDTH: Datawidth of client side AXI-ST interface coming from HSSI SS IP. This is not an user configurabale parameter. User need update this value to reflect HSSI SS IP client side data width for selected configuration.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#113-ofs-ethernet-subsystem-interface-module","title":"11.3 OFS Ethernet Subsystem Interface Module","text":"A wrapper around the Transceiver Subsystem integrates the following features: * DFH registers * Control & status registers
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1131-ethernet-subsystem-control-and-status-register-csr-map","title":"11.3.1 Ethernet Subsystem Control and Status Register (CSR) Map","text":"The Ethernet Subsystem connects to the BPF which is memory mapped to PF0 BAR0. The Ethernet Subsystem CSR space in the FIM consists of two parts:
- HSSI Subsystem CSRs assigned from offset 0x000 to 0x7FF
- HSSI Wrapper CSRs are added to FIM at offset 0x800 to 0xFFF
The PCIe subsystem uses AXI Memory Mapped accesses to read and write HSSI Subsystem Control and Status Registers in the FIM. The HSSI Subsystem CSR Map structure is designed to scale according to IP capabilities.
The Ethernet Subsystem CSR Map can be found ofs-agx7-pcie-attach/ipss/hssi/HSSI_SS_CSR.xls.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#114-ethernet-subsytem-software","title":"11.4 Ethernet Subsytem Software","text":"There are two pieces of software related to running the Ethernet Subsystem: The Linux* dfl network driver and the user space OPAE Tools.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1141-ethernet-subsystem-linux-driver","title":"11.4.1 Ethernet Subsystem Linux Driver","text":"The Ethernet subystem is exposed as a feature in the PCIe PF BAR0 region. It has a Device Feature Header (DFH) specifying the interface.
The primary functionality of the driver is to interact with the ethernet MAC and PHY through an indirect register access mailbox implemented by the HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers described above. To aid in RTL bringup, the driver provides a debugfs interface directly to the indirect register access mailbox. For each HSSI interface in the system there would be a directory with the following form containing two files, regaddr and regval: /sys/kernel/debug/dfl-fme.X.Y
To read a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then read the value as string out of regval file. To write a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then write the value as a C hex string to regval file.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1142-ethernet-subsystem-opae-user-space-tool","title":"11.4.2 Ethernet Subsystem OPAE User Space Tool","text":"User space OPAE Tools that are part of OPAE SDK provide support for the Ethernet Subsystem. You can use the --help option to print more information about each of these commands:
- hssi: Provides a means of interacting with the 10G and 100G HSSI AFUs through the host exerciser.
- hssiloopback: Enables and disables Ethernet loopback.
- hssistats: Provides MAC statistics.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#12-partial-reconfiguration","title":"12 Partial Reconfiguration","text":"Partial Reconfiguration (PR) is an Altera FPGA technology that allows users to reconfigure parts of the FPGA device dynamically, while the remainder of the device continues to operate. In a non-partial reconfiguration flow, any change to the design requires full reprogramming of the entire configuration RAM (CRAM) arrays in the device. With partial reconfiguration, you can dynamically reprogram one or more CRAM frames. A partial reconfiguration design has a static region, and a PR region, which can be modified to implement new logic. The portion of the CRAM on the chip to be reconfigured is contained within a PR region.
For the PR flow, your design must be partitioned into static region and reconfigurable region. The static region is the area of your FPGA that is not reconfigured without reprogramming the entire FPGA. An area of the chip that you plan to partially reconfigure is a PR region.
The Port Gasket contains all the PR specific modules and logic, such as PR slot reset/freeze control, user clock, remote STP etc. For this reference example only one PR slot is supported.
The Figure below shows the fundamental concepts required to support PR in OFS platform. There are 4 OFS management DFH \u2013 PR, Port, User Clock and Remote STP in Port Gasket that is exposed to OFS software. These platform capabilities are generally used in conjunction to Partial Reconfiguration. The Figure below also demonstrates the concepts of adding a new interface to the PR region.
Figure 12-1 Partial Reconfiguration Gasket
The isolation logic is provided on the output signals of the PR region to ensure they don\u2019t glitch and affect the interfacing logic in the Static Region (SR). The isolation logic is controlled by the PR Freeze logic during PR operation.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#user-clock-support","title":"User Clock Support","text":"The reference platform provides two user configurable clock (uclk_usr, uclk_usr_div2) for use by the AFU. These clocks are generated by a reconfigurable IOPLL. The control and status of these clocks is expose through two pairs of command and status registers (USR_CLK_FREQ_CMD0 / USR_CLK_FREQ_STS0 & USR_CLK_FREQ_CMD1 / USR_CLK_FREQ_STS1). The first pair controls the fPLL reconfiguration functionality. The second pair controls a clock frequency measurement block.
The following are the default values of the userclk.
uclk_usr: 312.5 MHz
uclk_usr_div2: 156.2 MHz
Table 12-1 usr_clk_* Acceptable Programming Range
Fabric Frequency Range uclk_usr (H) uclk_usr_div2 (L) Details 0-400 MHz 0-800 MHz 0-400 MHz Clocks set on 2x relationship for L<400 MHz 400-800 MHz 400-800 MHz 400-800 MHz Clks are equal for L>400 MHz"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#pll-reconfiguration","title":"PLL Reconfiguration","text":"The blue bit stream hardware exposes the low level IOPLL reconfiguration interfaces directly to software control. Through the USR_CLK_FREQ_CMD0 register software can select the reference clock, assert the PLL power down pin and issue reads and writes on the IOPLL Avalon-mm reconfiguration bus. Through the USR_CLK_FREQ_STS0 register software can query the IOPLL active reference clock, locked status and readdata returned from the IOPLL AVMM interface for read requests.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#clock-frequency-counter","title":"Clock Frequency Counter","text":"The user clocks, generated by the reconfigurable IOPLL, are connected to a frequency measurement circuit. Software selects which of the two clocks to measure by setting the clock select bit in the USER_CLK_FREQ_CMD1 register. After 10ms software can read the frequency, in 10 KHz units, from the USER_CLK_FREQ_STS1 register. Reading this register before 10ms has passed will return undefined data but otherwise has no effect.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#configuring-user-clocks","title":"Configuring User Clocks","text":"To program the user clock to the desired frequency, user will set the clock-frequency-low and clock-frequency-high fields in the PR AFU GBS .json file to the desired frequency value. During PR, SW will try to provide the closest possible frequency to the value specified in the .json file.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#13-host-exercisers","title":"13 Host Exercisers","text":"The Host Exerciser (HE) modules are responsible for generating traffic with the intent of exercising the specific interface they are designed to connect to. They are intended to test the interface in simulation and hardware, enable measurement of bandwidth and other performance KPIs and, in come cases, provide an example of data movement between interfaces (PCIe to DDR for e.g.) for adoption into a customer application.
### 13.1 HE-LB and HE-MEM Host Exerciser
The Host Exerciser Loopback module is a traffic generator that can be attached both to host memory, over PCIe (HE-LB), and to local memory, such as board-attached DDR (HE-MEM). The Host Exerciser (HE) is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth as well as demonstrating data path correctness. The FIM picks up the HE-LB module behind the PIM (Platform Interface Manager). The PIM converts the AXI with TLP out of the PCIe SS to standard Arm\u00ae AMBA\u00ae 4 AXI4 (MM for completions and Lite for CSR) interfaces. The figure below shows how the HE-LB is instantiated in the FIM.
Figure 13-1 HE-LB Dataflow Diagram
The exerciser has two primary modes: loopback, in which read data is fed back as writes, and read/write mode, in which reads and writes are generated independently. Furthermore, the host_exerciser software provided with OPAE that drives the RTL has two major modes: \"lpbk\" and \"mem\". These modes only select the UUID of the HE AFU, with lpbk selecting a version configured with no local memory (56e203e9-864f-49a7-b94b-12284c31e02b) and mem seeking a version with local memory (8568ab4e-6ba5-4616-bb65-2a578330a8eb). The behavior of each engine with and without local memory is described below.
Figure 13-2 HE Engine Hierarchy
For more details of HE-LB and HE-MEM IP, please refer to ofs-fim-common/src/common/he_lb/README.md
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#132-hssi-host-exerciser-he-hssi","title":"13.2 HSSI Host Exerciser (HE-HSSI)","text":"HE-HSSI is an Ethernet AFU that handles client side ethernet traffic. The reference HE-HSSI has following features:
- HE-HSSI provides a compatible interface with the Ethernet Subsystem.
- Includes a traffic generator and checker or monitor
- Provides pause signals to the Ethernet Subsystem for XON and XOFF generation
- Generates traffic or incoming traffic that can be looped back into transmit path by enabling loopback mode, which will bypass traffic generator
- At the HE-HSSI interface boundary the Ethernet data interface is AXI4-Stream with 64-bit data at eth_clk clock
- The Traffic generator and checker modules have a 64-bit data interface at eth_clk clock.
- The traffic generator supports the following modes:
- Fixed length or Random Length
- Incremental pattern or Random pattern
- Traffic checker does a 32-bit crc check in 10/25G. In the 100G configuration, there is no data integrity check, only a packet counter. (TBD)
- The CSR of this AFU is accessible through AXI4-Stream PCIe TLP interface
- The PCIe TLP to CSR Interface Conversion module converts PCIe TLPs into simple CSR interface
- The CSR space of the traffic generator and checker modules are accessed in an indirect way using mailbox registers
- If used for more than one channel, each channel has a separate traffic generator and traffic checker with separate CSR space.
- Reads and Writes to individual traffic controller CSR spaces can be done by selecting that particular channel using channel select register.
The HE-HSSI Ethernet block diagram is below.
Figure 13-3: HE-HSSI Block Diagram Block Diagram
The diagram below shows the path followed depending on if you enable loopback mode in HE-HSSI or not. In loopback mode, traffic is looped back into the transmit path which will bypass traffic. Alternatively, the traffic generates traffic.
Figure 13-4: HE-HSSI Operation Mode Traffic Patterns
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1322-he-hssi-csr-map","title":"13.2.2 HE-HSSI CSR Map","text":"The reference HSSI AFU contains the following registers and a similar arrangement of register space can be implemented for other usecase specific HSSI AFUs. * AFU DFH Register: Device feature header for the AFU (AFU_DFH) * AFU ID Registers: 128-bit UUID for the AFU which occupies two 64-bit registers (AFU_ID_L, AFU_ID_H) * Mailbox Registers: Command and Data register for traffic controller register access. It follows the standard access method defined for OFS. Access method and implementation is same as Reconfiguration Interface defined for the HSSI FIM. (TRAFFIC_CTRL_CMD, TRAFFIC_CTRL_DATA) * Channel Select Register: Channel select register for traffic controller mailbox access. It is used in cases where more than one channel is in the AFU, else it defaults to zero, meaning channel-0 is selected. (TRAFFIC_CTRL_PORT_SEL) * Scratchpad Register: Scratchpad register for CSR access checking. (AFU_SCRATCHPAD)
The CSR excel for HE-HSSI module can be found at ofs-common/src/common/he_hssi/HE_HSSI_CSR.xls.
13.3 HE-Null Overview
This module is a simple stub that is used to replace various HE and other blocks in the FIM whenever they are bypassed using the build script command line options such as null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. To find out more about these compiler directives, refer to the Shell Developer Guides.
Table 13-1 HE-Null DFH
REGISTER NAME ADDRESS ACCESS DEFAULT DESCRIPTION DFH 0x0000 RO 0x0000_0000_1000_0000 DFH register GUID_L 0x0008 RO 0xaa31f54a3e403501 Lower 64b of GUID GUID_H 0x0010 RO 0x3e7b60a0df2d4850 Upper 64b of GUID SCRATCHPAD 0x0018 RW 0x0 Scratchpad Figure 13-5: HE-Null Block Diagram
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14-reliability-accessibility-serviceability-ras-and-error-handling-tbd","title":"14 Reliability, Accessibility, Serviceability (RAS) and Error Handling (TBD)","text":" - Downstream AFU checker: Identifies AFU violations. For example, these checker flags violations of the interface specification.
- Upstream software or PCIe link checker: Identifies invalid traffic from PCIe that violates FIM or PCIe specifications. For example, this checker flags an application sending traffic if it violates the FIM specification or creates a PCIe link issue by causing completion timeout or malformed TLP.
- FIM - Checks for bugs in the FIM fabric.
Errors reported by the checker are logged in either the FME error registers or Port error registers, or both, as shown in the table below. For more details on each of the registers, please refer to ofs-fim-common/src/common/fme/xls/n6001/FME_CSR.xls or the SystemVerilog file: ofs-common/src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#141-fme-errors","title":"14.1 FME Errors","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1411-fme_error0","title":"14.1.1 FME_ERROR0","text":"The FME_ERROR0 register flags CoreFIM FME errors in the Global Error (GLBL_ERROR) private feature. The error bits in this register are sticky bits. You can only clear these bits through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in FME_ERROR0_MASK register masks the error.
Table 14-2: FME Error Types
Error Type Description Fabric errors FIFO underflow/overflow condition in CoreFIM. These errors only occur if you have introduced bugs into the FIM or during very rare single event upset (SEU) or SEU-like events. Invalid port access A port can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the Port. If it finds a PF is trying to access a port that is mapped to a VF or vice-versa, an error will be reported. Invalid AFU access An AFU can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the AFU associated with the Port. If it finds a PF is trying to access an AFU that is mapped to a VF or vice-versa, an error is reported and a fake response is sent back to the requester to avoid a completion timeout on the host."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1412-pcie0_error","title":"14.1.2 PCIE0_ERROR","text":"The PCIe Avalon-ST to AXI4-Stream bridge monitors the PCIe link for errors and logs any such errors in the PCIE0_ERROR register (in PCIE0 feature region) and PCIE0_ERROR register in the GLBL_ERR private feature. The error bits in the PCIE0_ERROR register are sticky bits that you can only clear through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in PCIE0_ERROR0_MASK masks the error. (TBD)
If you have other external FME features, you can add similar _ERROR registers to this space. Please refer to the following spreadsheet in the release branch found at: ofs-fim-common/src/fpga_family/agilex/pcie_ss/PCIE_CSR.xls or the SystemVerilog file: ofs-fim-common/src/fpga_family/agilex/pcie_ss/pcie_csr.sv for more details on this register.
NOTE: The PCIE0_ERROR register is located in both the Global Error external feature memory space and a separate PCIe external feature memory space. OPAE software supports the PCIe external feature memory space beginning at offset 0x40000 for OFS EA and going forward. PCIe registers beginning at 0x4000 in the Global Error external feature memory space is there for backward compatibility to the Intel FPGA PAC D5005 v2.0.1 Acceleration Stack. (TBD)
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1413-fme_first_error-fme_next_error","title":"14.1.3 FME_FIRST_ERROR, FME_NEXT_ERROR","text":"The FME_FIRST_ERROR register flags which of the FME error reporting registers, such as FME_ERROR0, PCIE0_ERROR0, has reported the first error occurrence. The error fields of the first error register are then continuously logged into the FME_FIRST_ERROR register until a system reset or software clears all the errors in that first error register. Likewise, the FME_NEXT_ERROR indicates which of the FME error reporting registers (except the first error register) has reported the next occurrence of error after the first error register. The error fields of the next error register are continuously logged into the FME_NEXT_ERROR register until a system reset or software clears all the errors in the second error register.
Please refer to the file in the ofs-fim-common repository folder: src/common/fme/fme_csr.sv for individual register field descriptions or the SystemVerilog file src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1414-reliability-accessibility-serviceability-ras-error-status","title":"14.1.4 Reliability, Accessibility, Serviceability (RAS) Error Status","text":"The RAS feature in CoreFIM labels errors as non-fatal, fatal or catastrophic based on their impact to the system. * A non-fatal error usually originates from software or an AFU. With a non-fatal error, the user application may still be able to recover from the error by performing a soft reset on the AFU, fixing the user application software if necessary, and clearing the error. On the other hand, a fatal or catastrophic error is non-recoverable and requires the platform to be reset. * Non-fatal errors are logged in the RAS_NOFAT_ERR_STAT register and fatal or catastrophic errors are logged in the RAS_CATFAT_ERR_STAT register.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14141-non-fatal-errors","title":"14.1.4.1 Non-Fatal Errors","text":"The RAS_NOFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It logs the high-level status of non-fatal errors in the hardware. Unlike the error bits in the PCIE0_ERROR and FME_ERROR0 registers which are RW1C (software can write a 1 to clear the error), the error bits in this register are read-only and can only be cleared by system reset. Software has an option to mask the error using RAS_NOFAT_ERR_MASK. Please refer to the following file in the ofs-fim-common resository: src/common/fme/xls/n6000/FME_CSR.xls for individual register field descriptions or the SystemVerilog file: src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14142-catastrophic-fatal-errors","title":"14.1.4.2 Catastrophic & Fatal Errors","text":"The RAS_CATFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It captures the high-level status of errors that can only be recovered with a system reset. Therefore, the error bits in the RAS_CATFAT_ERR_STAT register are read-only and can only be cleared by system reset or masked through RAS_CATFAT_ERR_MASK. Please refer to the following file in the ofs-fim-common resository: src/common/fme/xls/n6000/FME_CSR.xls for individual register field descriptions or the SystemVerilog file: src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1415-ras-error-injection","title":"14.1.5 RAS Error Injection","text":"For software testing purposes, you can inject non-fatal, fatal and catastrophic errors into the platform through the RAS_ERROR_INJ register. These errors are reported in the RAS_CATFAT_ERR_STAT and RAS_NOFAT_ERR_STAT registers. Please refer to the following file in the ofs-fim-common resository: src/common/fme/xls/n6000/FME_CSR.xls for individual register field descriptions or the SystemVerilog file: src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1416-fme-error-interrupts","title":"14.1.6 FME Error Interrupts","text":"In an event of an FME error, the MSI-X module in the FIM generates an interrupt so the host can decide on the next course of action. The FIM does not stall upstream and downstream traffic after the FME error. However, a port error triggered by invalid request from a user AFU stalls all the traffic going from AFU to PCIe. The interrupt capability is discoverable by querying the NumbSuppInterrupt field of the PORT_CAPABILITY register in the Port private feature. The MSI-X vector number is recorded in the InterruptVectorNumber field of the GLBL_ERROR_CAPABILITY register of the Global Error external feature.
An FME error interrupt is generated in response to the FME_ERROR0, PCIE0_ERROR0, RAS_NOFAT_ERR_STAT or RAS_CATFAT_ERR_STAT registers recording a new, unmasked, error per the rules defined by CoreFIM interrupt feature.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1417-fme-error-handling","title":"14.1.7 FME Error Handling","text":"When the host receives an FME error interrupt, it must take the recommended actions described below to bring the system back to its normal operation.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14171-catastrophicfatal-error","title":"14.1.7.1 Catastrophic/Fatal Error","text":"A system reset is mandatory for any catastrophic or fatal error.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14172-non-fatal-error","title":"14.1.7.2 Non-Fatal Error","text":"When software receives a non-fatal error interrupt which does not require a system reset, it can take the following steps to clear the error after software handles the error: 1. Set the *_ERROR_MASK register to all 1\u2019s to mask all errors 2. Clear the *_FIRST_ERROR register 3. Clear the *_ERROR register 4. Set *_ERROR_MASK register to all 0\u2019s to enable all errors
- Result: The *_ERROR & *_FIRST_ERROR registers begin capturing new errors.
NOTE: A system reset can only clear RAS Error status registers.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#142-mmio-requests","title":"14.2 MMIO Requests","text":"The FIM is designed to gracefully handle MMIO request scenarios.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1421-unsupported-functions-and-bars","title":"14.2.1 Unsupported Functions and BARs","text":"The PCIe hard IP in the FIM guarantees that only TLP packets for the functions and BARs supported by the FIM (as configured in PCIe HIP IP instantiation) are sent to the FIM.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1422-mmio-request-decoding","title":"14.2.2 MMIO Request Decoding","text":"The packet router and memory decoder in the FIM ensure that only legal MMIO requests are forwarded to the targeted MMIO region. Full address and BAR decoding is done both in the packet router and the memory decoder to ensure the requests are forwarded to the designated CSR region as defined in the MMIO Regions section of this manual. Any unsolicited/illegal MMIO requests are dropped, and an error is reported back to host through the FME error register.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1423-unused-fmeport-csr-regions","title":"14.2.3 Unused FME/Port CSR Regions","text":"All the CSR slaves in the FIM which are mapped to the FME or Port CSR regions must always respond to MMIO read requests targeting its associated CSR region. A CSR slave must return all 0s for MMIO reads to its unused CSR region such as a reserved space or a region where no CSR register is implemented for the address. The FIM ensures MMIO reads to the FME or Port CSR regions that are not mapped to any CSR slave always gets a response of all 0s. The memory decoder module and fake responder module in the FIM provide this guaranteed response.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1424-unsupported-mmio-request","title":"14.2.4 Unsupported MMIO Request","text":"Any MMIO requests targeting FME or Port CSR regions with a length or address alignment that are not supported by the FIM are dropped, and an error is logged in PCIE0_ERROR register. The MMIO checker module in the FIM guarantees this response. When an unsupported MMIO read request to the FIM CSR region is detected, the FIM sends back a CPL (completion without data) with error status (UR) back to host.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1425-afu-access-violation","title":"14.2.5 AFU Access Violation","text":"AFU access violations refer to the scenarios where a PF is attempting to access the MMIO region of an AFU bound to a VF (virtualization enabled), or when a VF is trying to access the MMIO region of an AFU bound to a PF (virtualization disabled). When such a violation is detected, the FIM drops the request and logs the error in the FME_ERROR0 register. If the request is an MMIO read request, the FIM returns a fake response to the host.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1426-afu-mmio-response-timeout","title":"14.2.6 AFU MMIO Response Timeout","text":"An AFU MMIO Response timeout functions in the same manner described in the MMIO Response Timeout section.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#15-ofs-design-hierarchy","title":"15 OFS Design Hierarchy","text":"Files for design, build, and unit test simulation are found at https://github.com/OFS/ofs-agx7-pcie-attach, release branch release/ofs-2024.2.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#151-design-guidance","title":"15.1 Design Guidance","text":"The OFS FIM is designed with configurability and scalability in mind. Please refer to the shell devloper guide for your target board for detailed design guidance. You can find detailed instructions for customizing the FIM design.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122. \u00a0
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/","title":"Getting Started Guide: Open FPGA Stack for Intel\u00ae Agilex FPGAs Targeting the Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Last updated: November 19, 2024
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#10-introduction","title":"1.0 Introduction","text":"The purpose of this document is to help users get started in evaluating the 2024.2-1 version of the Open FPGA Stack (OFS) for the Intel Agilex FPGA targeting the Intel N6000/1-PL FPGA SmartNIC Platform. After reviewing the document a user shall be able to:
- Set up their server environment according to the Best Known Configuration (BKC)
- Build and install the OFS Linux Kernel drivers
- Build and install the Open Programmable Acceleration Engine Software Development Kit (OPAE SDK) software on top of the OFS Linux kernel drivers
- Load and Verify the Firmware and FIM versions loaded on their boards
- Verify the full stack functionality offered by the OFS solution
- Know where to find additional information on other OFS ingredients
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell targeting Intel N6000/1-PL FPGA SmartNIC Platforms. These platforms are Development Kits intended to be used as a starting point for evaluation and development.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#13-references-and-versions","title":"1.3 References and Versions","text":"The OFS 2024.2-1 Release targeting the Intel\u00ae FPGA SmartNIC N6000/1-PL is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions which compose this release.
The following table highlights the hardware which makes up the Best Known Configuration (BKC) for the OFS 2024.2-1 release.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-2-hardware-bkc","title":"Table 2: Hardware BKC","text":"Component Version 1 x Intel\u00ae FPGA SmartNIC N6001-PL, SKU2 1 x Supermicro Server SYS-220HE 1 x Intel FPGA Download Cable II (Only Required for manual flashing) 1 x 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing) The following table highlights the versions of the software which compose the OFS stack. The installation of the OPAE SDK on top of the Linux DFL drivers will be discussed in their relevant sections in this document.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-3-software-version-summary","title":"Table 3: Software Version Summary","text":"Component Version FPGA Platform Intel\u00ae FPGA SmartNIC N6001-PL, release notes: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 under \"Known Issues\" OPAE SDK 2.13.0-3 Backport Kernel Drivers intel-1.11.0-2 OneAPI-ASP ofs-2024.2-1 OFS FIM Source Code for N6001 ofs-2024.2-1 OFS FIM Common Resources Tag: ofs-fim-common-1.1.0-rc2 OFS Platform AFU BBB ofs-2024.2-1 Intel Quartus Prime Pro Edition Design Software* Quartus Prime Pro Version 24.1 for Linux Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platform you have. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers to be installed as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-4-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 4: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"SKU Mapping SKU Value Primary Difference fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table highlights the programmable firmware versions that are supported on the Intel N6000/1-PL FPGA SmartNIC Platform in the OFS 2024.2-1 release. Programming and verifying these components is discussed in their respective sections.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-5-intel-fpga-smartnic-n60001-pl-programmable-component-version-summary","title":"Table 5: Intel\u00ae FPGA SmartNIC N6000/1-PL Programmable Component Version Summary","text":"Component Version PR Interface ID a791757d-38a6-5159-a7fc-e1a61157a07b Bitstream ID 360571656856467345 BMC RTL 3.15.2 BMC NIOS FW 3.15.2"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#14-reference-documents","title":"1.4 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.2-1/.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#15-board-installation-and-server-setup","title":"1.5 Board Installation and Server Setup","text":"Platform installation instructions, server BIOS settings and regulatory information can be found in the Board Installation Guide: OFS for Acceleration Development Platforms.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-6-supermicro-server-bmc-bkc","title":"Table 6: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4)"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
OFS is a hardware and software infrastructure that provides an efficient approach to developing a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM), or shell of the FPGA provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The OFS architecture for Intel Agilex FPGA provides modularity, configurability and scalability. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem - a hierarchical design that targets the P-tile PCIe hard IP and is configured to support Gen4 speeds and Arm AXI4-Stream Data Mover functional mode.
- Ethernet Subsystem - provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
- Memory Subsystem - composed of 5 DDR4 channels; one HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each, and four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB
- Hard Processor System - 64-bit quad core ARM\u00ae Cortex*-A53 MPCore with integrated peripherals.
- Reset Controller
- FPGA Management Engine - Provides a way to manage the platform and enable acceleration functions on the platform.
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration. Each feature of the FME exposes itself to the kernel-level OFS drivers on the host through a Device Feature Header (DFH) register that is placed at the beginning of Control Status Register (CSR) space. Only one PCIe link can access the FME register space in a multi-host channel design architecture at a time.
Note: For more information on the FIM and its external connections, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required to support partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your entire AFU resides in a partial reconfiguration region of the FPGA.
- The AFU is part of the static region and is compiled as a flat design.
- Your AFU contains both static and PR regions.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components: ST2MM module, Virtio LB stub, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), Memory Host Exerciser (HE-MEM), Traffic Generator to memory (HE-MEM-TG), Port Gasket (PRG) and HPS Copy Engine.
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Basic HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
The AFU has the option to consume native packets from the host or interface channels or to instantiate a shim provided by the Platform Interface Manager (PIM) to translate between protocols.
Note: For more information on the Platform Interface Manager and AFU development and testing, refer to the Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#22-ofs-software-overview","title":"2.2 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack and on kernel.org.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS Backport DFL driver software provides the bottom-most API to FPGA platforms for this release. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The Backport DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
Build and installation instructions are detailed in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit the opae reference page.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access. You have two options to install OPAE as discussed below - using pre-built packages offered by Intel, or building the source code locally.
The OPAE SDK can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
You can choose to build and install the OPAE SDK from source, which is detailed in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#44-fpga-device-access-permissions","title":"4.4 FPGA Device Access Permissions","text":"Access to FPGA accelerators and devices is controlled using file access permissions on the Intel\u00ae FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
sudo chmod a+rw /dev/dfl-port.0\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#45-memlock-limit","title":"4.5 Memlock limit","text":"Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#46-opae-tools-overview","title":"4.6 OPAE Tools Overview","text":"The following section offers a brief introduction including expected output values for the utilities included with OPAE. A full explanation of each command with a description of its syntax is available in the opae-sdk GitHub repo. The following command outputs were captured using an N6001 device.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#461-board-management-with-fpgainfo","title":"4.6.1 Board Management with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files.
Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Note: Your BItstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo.
$ sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : a2b5fd0e7afca4ee6d7048f926e75ac2\nUser1 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\nUser2 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\n
$ sudo fpgainfo bmc\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\n( 1) VCCRT_GXER_0V9 Voltage : 0.91 Volts\n( 2) FPGA VCCIO_1V2 Voltage : 1.21 Volts\n( 3) Inlet 12V Aux Rail Current : 0.87 Amps\n( 4) FPGA E-Tile Temperature [Remote] : 47.00 Celsius\n( 5) AVDD_ETH_0V9_CVL Voltage : 1.48 Volts\n( 6) FPGA E-TILE Temperature #3 : 51.00 Celsius\n...\n(77) FPGA FABRIC Remote Digital Temperature#3 : 47.00 Celsius\n(78) MAX10 & Board CLK PWR 3V3 Inlet Current : 0.97 Amps\n(79) CVL Non Core Rails Inlet Current : 0.01 Amps\n(80) FPGA Core Voltage Phase 0 VR Temperature : 49.50 Celsius\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#462-sensor-monitoring-with-fpgad","title":"4.6.2 Sensor Monitoring with fpgad","text":"The fpgad is a service that can help you protect the server from crashing when the hardware reaches an upper non-recoverable or lower non-recoverable sensor threshold (also called as fatal threshold). The fpgad is capable of monitoring each of the 80 sensors reported by the Board Management Controller. This service is only available once the installation instructions in sections 3.2 Building and Installing the OFS DFL Kernel Drivers and 4.1 OPAE SDK Build Environment Setup have been completed . Note: Qualified OEM server systems should provide the required cooling for your workloads. Therefore, using fpgad may be optional.
When the opae-tools-extra-2.13.0-3.x86_64 package is installed, fpgad is placed in the OPAE binaries directory (default: /usr/bin). The configuration file fpgad.cfg is located at /etc/opae. The log file fpgad.log which monitors fpgad actions is located at /var/lib/opae/. The fpgad periodically reads the sensor values and if the values exceed the warning threshold stated in the fpgad.conf or the hardware defined warning threshold, it masks the PCIe Advanced Error Reporting (AER) registers for the Intel N6000/1-PL FPGA SmartNIC Platform to avoid system reset. Use the following command to start the fpgad service:
Use the following command to start the fpgad service:
$ sudo systemctl start fpgad\n
The configuration file only includes the threshold setting for critical sensor 12V Aux Rail Voltage (sensor 29). This sensor does not have a hardware defined warning threshold and hence fpgad relies on the configuration file. The fpgad uses information contained within this file to mask the PCIe AER register when the sensor reaches the warning threshold. You may create another entry below the 12V Aux Voltage entry for any other sensors on the board. The updated configuration file includes a new entry for (18) Board Front Side Temperature with arbitrary values:
{\n\"configurations\": {\n\"fpgad-xfpga\": {\n\"configuration\": {\n},\n\"enabled\": true,\n\"plugin\": \"libfpgad-xfpga.so\",\n\"devices\": [\n[ \"0x8086\", \"0xbcc0\" ],\n[ \"0x8086\", \"0xbcc1\" ]\n]\n},\n\"fpgad-vc\": {\n\"configuration\": {\n\"cool-down\": 30,\n\"get-aer\": [\n\"setpci -s %s ECAP_AER+0x08.L\",\n\"setpci -s %s ECAP_AER+0x14.L\"\n],\n\"disable-aer\": [\n\"setpci -s %s ECAP_AER+0x08.L=0xffffffff\",\n\"setpci -s %s ECAP_AER+0x14.L=0xffffffff\"\n],\n\"set-aer\": [\n\"setpci -s %s ECAP_AER+0x08.L=0x%08x\",\n\"setpci -s %s ECAP_AER+0x14.L=0x%08x\"\n],\n\"config-sensors-enabled\": true,\n\"sensors\": [\n{\n\"name\": \"12V AUX Voltage\",\n\"low-warn\": 11.40,\n\"low-fatal\": 10.56\n},\n{\n\u201cname\u201d: \u201c3V3 VR Temperature\u201d,\n\u201clow-warn\u201d: 50.00,\n\u201clow-fatal\u201d: 100.00\n}\n]\n},\n\"enabled\": true,\n\"plugin\": \"libfpgad-vc.so\",\n\"devices\": [\n[ \"0x8086\", \"0x0b30\" ],\n[ \"0x8086\", \"0x0b31\" ],\n[ \"0x8086\", \"0xaf00\" ],\n[ \"0x8086\", \"0xbcce\" ]\n]\n}\n},\n\"plugins\": [\n\"fpgad-xfpga\",\n\"fpgad-vc\"\n]\n}\n
You can monitor the log file to see if upper or lower warning threshold levels are hit. For example:
$ tail -f /var/lib/opae/fpgad.log | grep \u201csensor.*warning\u201d\nfpgad-vc: sensor ' Columbiaville Die Temperature ' warning\n
You must take appropriate action to recover from this warning before the sensor value reaches upper or lower fatal limits. On reaching the warning threshold limit, the daemon masks the AER registers and the log file will indicate that the sensor is tripped. Sample output: Warning message when the 'CVL Core0 Voltage VR Temperature' exceeds the upper warning threshold limit
$ tail -f /var/lib/opae/fpgad.log fpgad-vc: sensor 'CVL Core Voltage VR Temperature' warning.\nfpgad-vc: saving previous ECAP_AER+0x08 value 0x057ff030 for 0000:b0:02.0\nfpgad-vc: saving previous ECAP_AER+0x14 value 0x0000f1c1 for 0000:b0:02.0\nfpgad-vc: sensor 'CVL Core Voltage VR Temperature' still tripped.\n
If the upper or lower fatal threshold limit is reached, then a power cycle of server is required to recover the Intel N6000/1-PL SmartNIC FPGA Platform. AER is unmasked by the fpgad after the sensor values are within the normal range which is above the lower warning or below the upper warning threshold.
To stop fpgad:
$ sudo systemctl stop fpgad.service\n
To check status of fpgad:
$ sudo systemctl status fpgad.service\n
Optional: To enable fpgad to re-start on boot, execute
$ sudo systemctl enable fpgad.service\n
For a full list of systemctl commands, run the following command:
$ systemctl -h\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#463-updating-with-fpgasupdate","title":"4.6.3 Updating with fpgasupdate","text":"The fpgasupdate tool updates the Intel Max10 Board Management Controller (BMC) image and firmware (FW), root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate tool only accepts images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then you must also sign the image using the correct keys. Refer to the Security User Guide: Open FPGA Stack for information on created signed images and on programming and managing the root entry hash.
The Intel\u00ae FPGA SmartNIC N6000/1-PL ships with a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL on all cards. The platform ships with a single FIM image that can be programmed into either user1 or user2, depending in the image selected.
Use the following chart for information on the Bitstream ID and Pr Interface ID, two unique values reported by fpgainfo which can be used to identify the loaded FIM.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-7-fim-version-summary-for-ofs-20242-1-release","title":"Table 7: FIM Version Summary for OFS 2024.2-1 Release","text":"FIM Version Bitstream ID Pr Interface ID File Name Download Location ofs-2024.2-1 360571656856467345 a791757d-38a6-5159-a7fc-e1a61157a07b ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2024.2-1 Release Page ofs-n6001-0.9.0-rc2 0x50102025AD3DD11 92ec8960-2f2f-5544-9804-075d2e8a71a1 ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2.3.0 Release Page OFS-2.3.0 0x50102022267A9ED f59830f7-e716-5369-a8b0-e7ea897cbf82 ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2.3.0 Release Page OFS-2.2.0 0x501020295B081F0 8c157a52-1cf2-5d37-9514-944af0a060da ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2.2.0-beta Release Page"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-8-bmc-version-summary-for-ofs-20242-1-release","title":"Table 8: BMC Version Summary for OFS 2024.2-1 Release","text":"BMC FW and RTL Version File Name Download Location 3.15.2 AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu n/a -
Example loading a new version of the BMC RTL and FW.
$ sudo fpgasupdate AC_BMC_RSU_user_retail_3.11.0_unsigned.rsu <PCI ADDRESS>\n[2022-04-14 16:32:47.93] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:32:47.93] [INFO ] updating from file /home/user/AC_BMC_RSU_user_retail_3.11.0_unsigned.rsu with size 904064 [2022-04-14 16:32:47.94] [INFO ] waiting for idle [2022-04-14 16:32:47.94] [INFO ] preparing image file [2022-04-14 16:33:26.98] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [904064/904064 bytes][Elapsed Time: 0:00:00.00] [2022-04-14 16:33:26.98] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [Elapsed Time: 0:00:26.02] [2022-04-14 16:33:53.01] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:33:53.01] [INFO ] Secure update OK [2022-04-14 16:33:53.01] [INFO ] Total time: 0:01:05.07\nsudo rsu bmcimg\n
-
Example for loading a Static Region (SR) update image. This process will take up to 20 minutes.
$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:42:31.58] [INFO ] updating from file ofs_top_page1_pacsign_user1.bin with size 19928064 [2022-04-14 16:42:31.60] [INFO ] waiting for idle [2022-04-14 16:42:31.60] [INFO ] preparing image file [2022-04-14 16:42:38.61] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] [2022-04-14 16:42:54.63] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] [2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:49:11.03] [INFO ] Secure update OK [2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#464-signing-images-with-pacsign","title":"4.6.4 Signing Images with PACSign","text":"PACSign is an OPAE utility which allows users to insert authentication markers into bitstreams targeted for the Intel\u00ae FPGA SmartNIC N6000/1-PL. PACSign also allows users updating their Static Region (SR) to designate which partition of flash (user1, user2, factory) to overwrite given a specific FIM binary image. All binary images must be signed using PACSign before fpgasupdate can use them for an update. Assuming no Root Entry Hash (REH) has been programmed on the device, the following examples demonstrate how to prepend the required secury authentication data, and specifiy which region of flash to update. More information, including charts detailing the different certification types and their required options, are fully described in the PACsign README on GitHub.
For more information on PACSign and on general security practices surrounding the Intel N6001-PL FPGA SmartNIC device, visit the Security User Guide: Intel Open FPGA Stack.
PACSign can be run on images that have previously been signed. It will overwrite any existing authentication data.
The following example creates an unsigned SR image from an existing signed SR binary update image, targeting the user1 partition in flash.
$ PACSign SR -t UPDATE -s 0 -H openssl_manager -i ofs_top_page1_pacsign_user1.bin -o new_image.bin\nNo root key specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo CSK specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo root entry hash bitstream specified. Verification will not be done. Continue? Y = yes, N = no: y\n2021-10-18 14:42:54,490 - PACSign.log - WARNING - Bitstream is already signed - removing signature blocks\n
--->
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#465-loading-images-with-rsu","title":"4.6.5 Loading Images with rsu","text":"The rsu performs a Remote System Update operation on a N6000/1-PL device, given its PCIe address. An rsu operation sends an instruction to the device to trigger a power cycle of the card and forces reconfiguration from flash for either the BMC or FPGA image.
The Intel\u00ae FPGA SmartNIC N6000/1-PL contains two regions of flash you may store FIM images. These locations are referred to as user1 and user2. After an image has been programmed with fpgasupdate in either of these regions you may choose to perform an rsu to switch. This operation indicates to the BMC which region to configure the FPGA device from after power-on.
If the factory image has been updated, Intel strongly recommends the user to immediately RSU to the factory image to ensure the image is functional.
The user can determine which region of flash was used to configure their FPGA device using the command fpgainfo fme and referring to the row labelled Boot Page.
$ sudo fpgainfo fme | grep Boot\nBoot Page : user1\n
Swapping between user1 and user2 skips load times that are created when using fpgasupdate to flash a new FIM image.
rsu Overview
Mode 1: RSU
Perform RSU (remote system update) operation on a development platform given its PCIe address. An RSU operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either BMC, Retimer, SDM, (on devices that support these) or the FPGA.
Mode 2: Default FPGA Image
Set the default FPGA boot sequence. The --page option determines the primary FPGA boot image. The --fallback option allows a comma-separated list of values to specify fallback images.
The following example will load an image stored in user2.
$ sudo rsu fpga --page=user2 0000:b1:00.0\n2022-04-15 09:25:22,951 - [[pci_address(0000:b1:00.0), pci_id(0x8086, 0xbcce)]] performing RSU operation\n2022-04-15 09:25:22,955 - [[pci_address(0000:b0:02.0), pci_id(0x8086, 0x347a)]] removing device from PCIe bus\n2022-04-15 09:25:22,998 - waiting 10 seconds for boot\n2022-04-15 09:25:33,009 - rescanning PCIe bus: /sys/devices/pci0000:b0/pci_bus/0000:b0\n2022-04-15 09:25:34,630 - RSU operation complete\n
Note: As a result of using the rsu command, the host rescans the PCI bus and may assign a different Bus/Device/Function (B/D/F) value than the originally assigned value.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#466-verify-fme-interrupts-with-hello_events","title":"4.6.6 Verify FME Interrupts with hello_events","text":"The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors.
Sample output from sudo hello_events.
$ sudo hello_events\nWaiting for interrupts now...\ninjecting error\nFME Interrupt occurred\nSuccessfully tested Register/Unregister for FME events!\nclearing error\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#467-host-exercisor-modules","title":"4.6.7 Host Exercisor Modules","text":"The reference FIM and unchanged FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. There are three HEMs present in the Intel OFS Reference FIM - HE-LPBK, HE-MEM, and HE-HSSI. These exercisers are tied to three different VFs that must be enabled before they can be used. Execution of these exercisers requires you bind specific VF endpoint to vfio-pci. The host-side software looks for these endpoints to grab the correct FPGA resource.
Refer to the Intel Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for a full description of these modules.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-9-module-pfvf-mappings","title":"Table 9: Module PF/VF Mappings","text":"Module PF/VF ST2MM PF0 HE-MEM PF0-VF0 HE-HSSI PF0-VF1 HE-MEM_TG PF0-VF2 HE-LB Stub PF1-VF0 HE-LB PF2 VirtIO LB Stub PF3 HPS Copy Engine PF4"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#4671-he-mem-he-lb","title":"4.6.7.1 HE-MEM / HE-LB","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: - Latency (AFU to Host memory read) - MMIO latency (Write+Read) - MMIO BW (64B MMIO writes) - BW (Read/Write, Read only, Wr only)
Host Exerciser Loopback Memory (HE-MEM) AFU is used to exercise use of FPGA connected DDR, data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host.
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. When using the Intel N6001-PL FPGA SmartNIC Platform, optimal performance requires the exercisers be run at 400 MHz.
Execution of these exercisers requires you to bind specific VF endpoint to vfio-pci. The following commands will bind the correct endpoint for a device with B/D/F 0000:b1:00.0 and run through a basic loopback test.
Note: While running the opae.io init command listed below, if no output is present after completion then the command has failed. Double check that Intel VT-D and IOMMU have been enabled in the kernel as discussed in step 12 in section 3.1 Intel OFS DFL Kernel Driver Environment Setup.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci Binding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci iommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188 Assigning /dev/vfio/188 to DCPsupport Changing permissions for /dev/vfio/188 to rw-rw----\n\n$ sudo host_exerciser --clock-mhz 400 lpbk\nstarting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2895\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 9.055 GB/s\n Test lpbk(1): PASS\n
The following example will run a loopback throughput test using one cacheline per request.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\n$ sudo host_exerciser --clock-mhz 400 --mode trput --cls cl_1 lpbk\nstarting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 1663\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 15.763 GB/s\n Test lpbk(1): PASS\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#4672-traffic-generator-afu-test-application","title":"4.6.7.2 Traffic Generator AFU Test Application","text":"Beginning in OPAE version 2.0.11-1+ the TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank. The supported arguments for test configuration are:
- Number of test loops: --loops
- Number of read transfers per test loop: -r,--read
- Number of write transfers per test loop: -w,--write
- Burst size of each transfer: -b,--bls
- Address stride between each transfer: --stride
- Target memory TG: -m,--mem-channel
Below are some example commands for how to execute the test application. To run the preconfigured write/read traffic test on channel 0:
$ mem_tg tg_test\n[2024-06-26 10:27:31.317] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 0:\nTG PASS\nMem Clock Cycles: 127\nDEBUG: wcnt_ 1\nDEBUG: rcnt_ 1\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1\nDEBUG: num_ticks 127\nWrite BW: 0.151181 GB/s\nRead BW: 0.151181 GB/s\nThread on channel 0 exited with status 0\n[2024-06-26 10:27:31.318] [tg_test] [info] Test tg_test(1): PASS\n
Target channel 1 with a 1MB single-word write only test for 1000 iterations
$ mem_tg --loops 1000 -r 0 -w 2000 -m 1 tg_test\n[2024-06-26 10:28:17.861] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 4379946\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 0\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1000\nDEBUG: num_ticks 4379946\nWrite BW: 8.76723 GB/s\nRead BW: 0 GB/s\n\nThread on channel 1 exited with status 0\n
Target channel 2 with 4MB write/read test of max burst length for 10 iterations
$ mem_tg --loops 10 -r 8 -w 8 --bls 255 -m 2 tg_test\n[2024-06-26 10:29:04.653] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 2:\nTG PASS\nMem Clock Cycles: 89462\nDEBUG: wcnt_ 8\nDEBUG: rcnt_ 8\nDEBUG: bcnt_ 255\nDEBUG: loop_ 10\nDEBUG: num_ticks 89462\nWrite BW: 4.37817 GB/s\nRead BW: 4.37817 GB/s\n\nThread on channel 2 exited with status 0\n
$ sudo mem_tg --loops 1000 -r 2000 -w 2000 --stride 2 --bls 2 -m 1 tg_test\n[2024-06-26 10:29:27.509] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 17565290\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 2000\nDEBUG: bcnt_ 2\nDEBUG: loop_ 1000\nDEBUG: num_ticks 17565290\nWrite BW: 4.37226 GB/s\nRead BW: 4.37226 GB/s\n\nThread on channel 1 exited with status 0\n[2024-06-26 10:29:27.568] [tg_test] [info] Test tg_test(1): PASS\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#4673-he-hssi","title":"4.6.7.3 HE-HSSI","text":"HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic.
The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode- specific options, and displays the ethernet statistics by invoking ethtool --statistics INTERFACE.
The following example walks through the process of binding the VF corresponding with the HE-HSSI exerciser to vfio-pci, sending traffic, and verifying that traffic was received.
1. Create 3 VFs in the PR region.
$ sudo pci_device b1:00.0 vf 3
2. Verify all 3 VFs were created.
$ lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
3. Bind all of the PF/VF endpoints to the vfio-pci driver.
$ sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\n$ sudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\n
4. Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
$ sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
The following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-10-accelerator-pfvf-and-guid-mappings","title":"Table 10: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6000/1-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb 5. Check Ethernet PHY settings with fpgainfo.
$ sudo fpgainfo phy -B 0xb1 IIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\n//****** HSSI information ******//\nHSSI version : 1.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
6. Set loopback mode.
$ sudo hssiloopback --loopback enable --pcie-address 0000:b1:00.0 args Namespace(loopback='enable', pcie_address='0000:b1:00.0', port=0)\nsbdf: 0000:b1:00.0\nFPGA dev: {'segment': 0, 'bus': 177, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:b1:00.0'}\nargs.hssi_grps{0: ['dfl_dev.6', ['/sys/bus/pci/devices/0000:b1:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.6/uio/uio0']]}\nfpga uio dev:dfl_dev.6\n\n--------HSSI INFO START-------\nDFH :0x3000000010002015\nHSSI ID :0x15\nDFHv :0.5\nguidl :0x99a078ad18418b9d\nguidh :0x4118a7cbd9db4a9b\nHSSI version :1.0\nFirmware Version :1\nHSSI num ports :8\nPort0 :25GbE\nPort1 :25GbE\nPort2 :25GbE\nPort3 :25GbE\nPort4 :25GbE\nPort5 :25GbE\nPort6 :25GbE\nPort7 :25GbE\n--------HSSI INFO END-------\n\nhssi loopback enabled to port0\n
7. Send traffic through the 10G AFU.
$ sudo hssi --pci-address b1:00.6 hssi_10g --num-packets 100 10G loopback test\nport: 0\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth:\n\nNo eth interface, so not honoring --eth-loopback.\n0x40000 ETH_AFU_DFH: 0x1000010000001000\n0x40008 ETH_AFU_ID_L: 0xbb370242ac130002\n0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea\n0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000\n0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000\n0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000\n0x40048 AFU_SCRATCHPAD: 0x0000000045324511\n\n0x3c00 number_packets: 0x00000064\n0x3c01 random_length: 0x00000000\n0x3c02 random_payload: 0x00000000\n0x3c03 start: 0x00000000\n0x3c04 stop: 0x00000000\n0x3c05 source_addr0: 0x44332211\n0x3c06 source_addr1: 0x00006655\n0x3c07 dest_addr0: 0xaa998877\n0x3c08 dest_addr1: 0x0000ccbb\n0x3c09 packet_tx_count: 0x00000064\n0x3c0a rnd_seed0: 0x5eed0000\n0x3c0b rnd_seed1: 0x5eed0001\n0x3c0c rnd_seed2: 0x00025eed\n0x3c0d pkt_length: 0x00000040\n0x3cf4 tx_end_tstamp: 0x000003d2\n0x3d00 num_pkt: 0xffffffff\n0x3d01 pkt_good: 0x00000064\n0x3d02 pkt_bad: 0x00000000\n0x3d07 avst_rx_err: 0x00000000\n0x3d0b rx_sta_tstamp: 0x00000103\n0x3d0c rx_end_tstamp: 0x0000053b\n0x3e00 mac_loop: 0x00000000\n\nHSSI performance:\n Selected clock frequency : 402.832 MHz\n Latency minimum : 642.948 ns\n Latency maximum : 896.155 ns\n Achieved Tx throughput : 18.4528 GB/s\n Achieved Rx throughput : 16.7101 GB/s\n\nNo eth interface, so not showing stats.\n
The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. The hssi_loopback utility tests both external and internal loopbacks.
The hssistats tool provides the MAC statistics.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#50-upgrading-the-intel-fpga-smartnic-n60001-pl-with-20242-1-version-of-the-bmc-and-fim","title":"5.0 Upgrading the Intel\u00ae FPGA SmartNIC N6000/1-PL with 2024.2-1 Version of the BMC and FIM","text":"If your Intel\u00ae FPGA SmartNIC N6000/1-PL does not have the 2024.2-1 version of the FIM and BMC, use this section to begin your upgrade process. The upgrade process depends on both the OPAE SDK and kernel drivers, which were installed in the Software Installation Guide: PCIe Attach. Use the output of fpgainfo and compare against the table below to determine if an upgade is necessary.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-11-fim-version-summary-for-intel-ofs-20242-1-release","title":"Table 11: FIM Version Summary for Intel OFS 2024.2-1 Release","text":"FIM Version Bitstream ID Pr Interface ID File Name Download Location 1 360571656856467345 a791757d-38a6-5159-a7fc-e1a61157a07b ofs_top_page[1 / 2]_unsigned_user[1 / 2].bin ofs-2024.2-1 Release Page"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-12-bmc-version-summary-for-intel-ofs-20242-1-release","title":"Table 12: BMC Version Summary for Intel OFS 2024.2-1 Release","text":"BMC FW and RTL Version File Name Download Location 3.15.2 AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu n/a Sample output of fpgainfo with matching values:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : a2b5fd0e7afca4ee6d7048f926e75ac2\nUser1 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\nUser2 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\n
- If your output does not match the table above, download the appropriate FIM image from the Intel OFS 2024.2-1 (Intel Agilex) release page. Once downloaded transfer the file over to the server and use the fpgasupdate utility to perform an upgrade of the BMC.
$ sudo fpgasupdate AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu\n[2022-04-14 16:32:47.93] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:32:47.93] [INFO ] updating from file /home/user/AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu with size 904064 [2022-04-14 16:32:47.94] [INFO ] waiting for idle [2022-04-14 16:32:47.94] [INFO ] preparing image file [2022-04-14 16:33:26.98] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [904064/904064 bytes][Elapsed Time: 0:00:00.00] [2022-04-14 16:33:26.98] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [Elapsed Time: 0:00:26.02] [2022-04-14 16:33:53.01] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:33:53.01] [INFO ] Secure update OK [2022-04-14 16:33:53.01] [INFO ] Total time: 0:01:05.07\nsudo rsu bmcimg\n
2. Load the new FIM image. ```bash\n$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. \n[2022-04-14 16:42:31.58] [INFO ] updating from file ofs_top_page1_pacsign_user1.bin with size 19928064 \n[2022-04-14 16:42:31.60] [INFO ] waiting for idle \n[2022-04-14 16:42:31.60] [INFO ] preparing image file \n[2022-04-14 16:42:38.61] [INFO ] writing image file \n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] \n[2022-04-14 16:42:54.63] [INFO ] programming image file \n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] \n[2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete \n[2022-04-14 16:49:11.03] [INFO ] Secure update OK \n[2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\nsudo rsu fpga --page=user1 <PCI ADDRESS>\n```\n
- Verify output of fpgainfo matches the table above.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/adp_board_installation_guidelines/","title":"Board Installation Guidelines: Intel\u00ae FPGA SmartNIC N6000/1-PL, Intel\u00ae FPGA PAC D5005","text":"Last updated: November 19, 2024
"},{"location":"n6001/adp_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"n6001/adp_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup - the Intel\u00ae FPGA SmartNIC N6000/1-PL and the Intel\u00ae FPGA PAC D5005. After reading the document a user shall be able to:
- Safely install and remove an ADP
- Set up their server BIOS with the recommended settings
- Learn about thermal cooling requirements for their platform
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to all three platforms.
"},{"location":"n6001/adp_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported ADP platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"n6001/adp_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"n6001/adp_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"n6001/adp_board_installation_guidelines/#table-2-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 2: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platforms you have if you are unsure. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers from sections, whose installation is covered in the [Software Installation Guide: OFS for PCIe Attach FPGAs].
SKU Mapping SKU Value Primary Difference fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table provides a picture reference for the hardware components discussed in the rest of the document.
"},{"location":"n6001/adp_board_installation_guidelines/#table-3-hardware-bkc","title":"Table 3: Hardware BKC","text":"Component Image Intel\u00ae FPGA SmartNIC N6001-PL (SKU2) Supermicro Server SYS-220HE Intel FPGA Download Cable II (Only Required for manual flashing) 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing) In addition to the above, all OFS ADP platforms require an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector. This cable will differ between server vendors - review the pinout of the power connector on the Intel\u00ae FPGA Programmable Acceleration Card D5005 Data Sheet or Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based ADP.
"},{"location":"n6001/adp_board_installation_guidelines/#20-initial-server-setup","title":"2.0 Initial Server Setup","text":""},{"location":"n6001/adp_board_installation_guidelines/#21-server-information-for-intel-fpga-smartnic-n60001-pl","title":"2.1 Server Information for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Both the server BIOS and BMC need to match the versions listed below in Table 4: Supermicro Server BMC BKC. These updates only apply for this specific Best Known Configuration (BKC) - other server manufacturers may require different BIOS updates. Please consult your server's user guide and release notes for update information.
"},{"location":"n6001/adp_board_installation_guidelines/#table-4-supermicro-server-bmc-bkc","title":"Table 4: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4) Information about the server\u2019s currently loaded firmware can be found on the BMC web portal dashboard. Accessing this page requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d. During boot the BMC\u2019s login IP will be presented on the screen.
Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case. It is recommended the user change their BMC\u2019s default username as soon as they are able.
After logging in you should be able to review information about the BMC and BIOS by referring to the System box, visible upon initial loading of the page. Double check that the values match those in Table 4. If they do not, you may download the appropriate versions from the SuperMicro product page by selecting the BIOS option and downloading the most recent \u201cBundled Software File Name\u201d. Follow the BMC and BIOS update instructions included in the SuperMicro manuals page in the document X12/H12 BMC Manual in Appendix A.2 Updating Firmware Using BMC Web GUI.
If using a different server model, refer to that server\u2019s user guide for instructions on remote system management. Ensure that any system you end up using meets all the following requirements:
- Main Board: PCI Express 3.0 (D5005) or 4.0 (N6000/1) compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
"},{"location":"n6001/adp_board_installation_guidelines/#22-server-information-for-intel-fpga-pac-d5005","title":"2.2 Server Information for Intel\u00ae FPGA PAC D5005","text":"Refer to sections 2.1-2.3 of the Intel Acceleration Stack Quick Start Guide: Intel FPGA Programmable Acceleration Card D5005 for a complete overview of the physical installation process and ESD precautions for the D5005 platform.
Ensure that the system meets all the following requirements before proceeding to install the Intel\u00ae FPGA PAC D5005 into a server.
- Main Board: PCI Express 3.0 compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
Detailed mechanical for information can be found on the D5005 Data Sheet and in section 4.0 Mechanical Information of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837).
"},{"location":"n6001/adp_board_installation_guidelines/#30-server-settings","title":"3.0 Server Settings","text":""},{"location":"n6001/adp_board_installation_guidelines/#31-bios-settings","title":"3.1 BIOS Settings","text":"You must enable Intel VT-x/VT-d technologies for the PCIe slot housing your ADP. The following steps are known to work on a SuperMicro SYS-220HE server platform.
-
To enter the Supermicro server\u2019s BIOS setup page, reboot, and press \\<Delete> when prompted. You can browse the tabs / options with a combination of arrow keys along with \\<Escape> and \\<Enter>.
-
Navigate right to the Advanced tab, then select the following menu options: Chipset Configuration -> North Bridge -> IIO Configuration -> Intel VT for Directed I/O (VT-d).
-
If not already, enable the option Intel VT for Directed I/O (VT-d).
"},{"location":"n6001/adp_board_installation_guidelines/#31-server-fan-speed","title":"3.1 Server Fan Speed","text":"The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
- Log in to the SuperMicro server BMC. (This requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d.)
- During boot the BMC\u2019s login IP will be presented on the screen. Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case.
- On the left menu select System -> Component Info, select the Fan tab, under Advanced Settings click the circle next to Full Speed.
"},{"location":"n6001/adp_board_installation_guidelines/#32-cooling-requirements","title":"3.2 Cooling Requirements","text":"Please refer to sections 8.1 and 8.2 of the Intel FPGA Programmable Acceleration Card D5005 Data Sheet or section 6.0 of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) for guidance on cooling specifications that must be met when using these platforms. Failure to adhere to these guidelines may result in thermal runaway and/or performance degradation.
"},{"location":"n6001/adp_board_installation_guidelines/#40-board-installation-procedure","title":"4.0 Board Installation Procedure","text":""},{"location":"n6001/adp_board_installation_guidelines/#41-pcie-slot-mappings-for-intel-fpga-smartnic-n60001-pl","title":"4.1 PCIe Slot Mappings for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"The Intel N6000/1-PL FPGA SmartNIC Platforms are officially verified in the upper middle PCIe x16 slot (Slot 3). If using a different slot, refer to the information in Table 5 PCIe Slot Mapping for which port settings to change in server BIOS.
"},{"location":"n6001/adp_board_installation_guidelines/#table-5-pcie-slot-mapping","title":"Table 5: PCIe Slot Mapping","text":"CPU Number Port Number (in BIOS) PCIe Slot CPU1 Port 2 5 and 6 CPU1 Port 4 7 and 8 CPU2 Port 2 1 and 2 CPU2 Port 4 3 and 4"},{"location":"n6001/adp_board_installation_guidelines/#42-installation-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.2 Installation Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe installation of an ADP platform into a supported server. While an Intel\u00ae FPGA SmartNIC N6001-PL is shown in the images below, this procedure applies to all three platforms.
- Position the board over the selected connector on the motherboard.
- Press down gently and firmly to seat the card in the PCIe slot, and then secure the bracket to the system chassis with the retention screw.
"},{"location":"n6001/adp_board_installation_guidelines/#table-6-adp-installation-procedure","title":"Table 6: ADP Installation Procedure","text":"Callout Description 1 Retention screw 2 Press down here gently 3 Press down here gently 4 Motherboard Do not bend the card while inserting into a slot. Do not apply much pressure in regions 2 or 3 while inserting.
"},{"location":"n6001/adp_board_installation_guidelines/#43-removal-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.3 Removal Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe removal of the platforms from a supported server.
- Disconnect all power cords from the server power supply(s).
- Remove the retention bracket screw.
- Carefully lift the card out of the PCIe slot.
"},{"location":"n6001/adp_board_installation_guidelines/#table-7-adp-removal-procedure","title":"Table 7: ADP Removal Procedure","text":"Callout Description 1 Retention screw 2 Pull up here gently 3 Motherboard Do not bend the card while removing it from the slot.
"},{"location":"n6001/sw_install_pcie_attach/","title":"Software Installation Guide: Open FPGA Stack for PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"n6001/sw_install_pcie_attach/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the PCIe Attach software stack on the host. This document will not cover the process of board installation or platform bring-up. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Build and install the OPAE Software Development Kit (SDK) on the host
- Build and install the Linux DFL driver stack on the host
"},{"location":"n6001/sw_install_pcie_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating a PCIe Attach shell. The PCIe Attach shell design is supported on a number of board offerings, including the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile), Intel\u00ae FPGA SmartNIC N6000/1-PL, and Intel\u00ae FPGA PAC D5005.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"n6001/sw_install_pcie_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"n6001/sw_install_pcie_attach/#table-2-software-and-component-version-summary-for-ofs-pcie-attach","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach","text":"The OFS PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are where the source code resides for each of the components discussed in this document.
Component Version Download Link Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 link OPAE SDK 2.13.0-3 2.13.0-3 Linux DFL intel-1.11.0-2 intel-1.11.0-2"},{"location":"n6001/sw_install_pcie_attach/#table-3-release-pages-for-each-pcie-attach-platform","title":"Table 3: Release Page(s) for each PCIe Attach Platform","text":"This is a comprehensive list of the platform(s) whose software build and installation steps are covered in this document.
Platform Release Page Link Stratix\u00ae 10 FPGA https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Intel\u00ae FPGA SmartNIC N6001-PL https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1"},{"location":"n6001/sw_install_pcie_attach/#12-server-requirements","title":"1.2 Server Requirements","text":""},{"location":"n6001/sw_install_pcie_attach/#121-host-bios","title":"1.2.1 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
- Intel VT for Directed I/O (VT-d) must be enabled
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"n6001/sw_install_pcie_attach/#122-host-server-kernel-and-grub-configuration","title":"1.2.2 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
- OS: RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10
- Kernel: 4.18.0-dfl
"},{"location":"n6001/sw_install_pcie_attach/#20-ofs-software-overview","title":"2.0 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack, on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"n6001/sw_install_pcie_attach/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions, and currently only supported the PCIe Attach release. Its usage is detailed in the relevant Quick Start Demonstration Guideline for your platform and will not be covered here.
"},{"location":"n6001/sw_install_pcie_attach/#31-ofs-dfl-backport-kernel-driver-installation-environment-setup","title":"3.1 OFS DFL Backport Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL Backport GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.2.2 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 3.3 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
-
It is recommended you lock your Red Hat release version to 8.10 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
-
Install the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n\nsudo dnf install kernel-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-headers-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-devel-4.18.0-553.5.1.el8_10.x86_64\n
Now you have the choice to either follow the steps in section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source or 3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages.
"},{"location":"n6001/sw_install_pcie_attach/#32-building-and-installing-the-ofs-dfl-backport-kernel-drivers-from-source","title":"3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Select the 4.18.0-553.5.1.el8_10 kernel as your next boot target on the build machine, then reboot.
# For multiple boots (until overwritten), use the following\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\n# Or select the kernel you want during boot time\nsudo reboot\n
1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n
2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nintel-1.11.0-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n
3. Build the kernel.
```bash\ncd /home/OFS/linux-dfl-backport\nmake && make rpm\n```\n
4. Install the newly compiled RPM package and reboot.
```bash\nsudo rpm -i intel-fpga-dfl-dkms-1.11.0-2.2024.07.25.g276007e.noarch.rpm\n\nsudo reboot\n```\n
5. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/4.18.0-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\n
If an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n
6. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n
7. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n
8. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n
9. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-4.18.0-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\n
A list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"n6001/sw_install_pcie_attach/#33-installing-the-ofs-dfl-backport-kernel-drivers-from-pre-built-packages","title":"3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page under the Artifacts tab. The name will resemble kernel-*.tar.gz. For the backport repository you can also choose to download packages under the GitHub action Build dkms packages, although your version will differ from the release version of the software.
- Download and install the pre-built kernel package.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\ntar xf kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\nsudo rpm -i kernel-4.18.0_dfl_2024.06.14_276007e/intel-fpga-dfl-dkms-1.11.0-2.2024.06.14.g276007e.noarch.rpm\n\n# Set this kernel as your new default boot target\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\nsudo reboot\n
Continue from step 5 of Section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source.
"},{"location":"n6001/sw_install_pcie_attach/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 4.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"n6001/sw_install_pcie_attach/#41-opae-sdk-installation-environment-setup","title":"4.1 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"n6001/sw_install_pcie_attach/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package. -
Remove any currently installed OPAE packages.
sudo dnf remove opae*\n
-
Initialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\n
-
Verify that the correct tag/branch have been checkout out.
git describe --tags\n2.13.0-3\n
-
Set up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\n
The following packages will be built in the same directory as create:
-
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\n
-
Check that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\n
"},{"location":"n6001/sw_install_pcie_attach/#42-installing-the-opae-sdk-with-pre-built-packages","title":"4.2 Installing the OPAE SDK with Pre-Built Packages","text":"You can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.13.0-3.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.13.0-3.x86_64-*.tar.gz\n
For a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\n
"},{"location":"n6001/sw_install_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"n6001/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\n
"},{"location":"n6001/ug_dev_afu_host_software/#11-find-and-connect-to-afu","title":"1.1. Find and connect to AFU","text":"Here is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\n
In main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\n
"},{"location":"n6001/ug_dev_afu_host_software/#12-map-mmio-optional","title":"1.2. Map MMIO (optional)","text":"Mapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\n
The below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\n
"},{"location":"n6001/ug_dev_afu_host_software/#13-allocate-shared-memory-buffers","title":"1.3. Allocate Shared Memory Buffers","text":"The below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\n
In main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\n
"},{"location":"n6001/ug_dev_afu_host_software/#14-perform-acceleration","title":"1.4. Perform Acceleration","text":"The host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\n
"},{"location":"n6001/ug_dev_afu_host_software/#15-cleanup","title":"1.5. Cleanup","text":"When the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\n
"},{"location":"n6001/ug_dev_afu_host_software/#2-building-the-host-application","title":"2. Building the Host Application","text":"A Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
- Path to common_include.mk (from examples-afu)
- TEST name
- Source files: SRCS
- Path to .json file (relative to Makefile directory)
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\n
"},{"location":"n6001/ug_dev_afu_host_software/#3-running-the-host-application","title":"3. Running the Host Application","text":"To run the host application, you will need to:
- Load AFU onto the FIM
- Create VF's
- Bind VF's using the OPAE Drivers
- Run application
See the associated AFU Developer Guide for details.
"},{"location":"n6001/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
- C/C++
- Verilog/SystemVerilog
- RTL simulators such as Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"n6001/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
- Verilog
- SystemVerilog
- VHDL
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"n6001/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
- Software: OPAE API implemented in the C programming language.
- Hardware: PCIe SS TLP specification implemented in SystemVerilog.
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"n6001/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":" - The ASE provides a protocol checker to ensure protocol correctness. The ASE also provides methods to identify potential issues early, before in-system deployment.
- The ASE can help identify certain lock conditions and Configuration and Status Registers (CSR) address mapping and pointer math errors.
- The ASE tracks memory requested from the accelerator. The memory model immediately flags illegal memory transactions to locations outside of requested memory spaces. Consequently, you can fix incorrect memory accesses early, during the simulation phase.
- The ASE does not guarantee that you can synthesize an AFU. After you verify the AFU RTL functionality in the ASE, use the ASE and the Quartus\u00ae Prime Pro Edition software iteratively to generate the Accelerator Function (AF).
- The ASE does not require administrator privileges. After installing all the required tools, you can run the ASE on a plain vanilla user Linux machine.
"},{"location":"n6001/ug_dev_afu_sim_env/#23-ase-limitations","title":"2.3. ASE Limitations","text":"When using ASE in the application development cycle, consider the following limitations:
- The ASE is a transaction-level simulator. It does not model either Intel UPI- or PCIe-specific packet structures and protocol layers.
- The ASE does not simulate caching and is not a cache simulator. It cannot reliably simulate cache collisions or capacity issues.
- Although ASE models some latency parameters, it cannot model real-time system-specific latency. It is also not an accurate timing simulation of the design or latency and bandwidth of the real system. The ASE models enable you to develop functionally correct accelerators.
- The ASE does not simulate multi-AFU or multi-socket configurations.
"},{"location":"n6001/ug_dev_afu_sim_env/#24-ase-based-afu-design-workflow","title":"2.4 ASE-Based AFU Design Workflow","text":"AFU development using the ASE includes the following four stages:
-
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
-
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
-
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
-
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
- The AFU RTL code is synthesizable.
- The AFU RTL code meets timing.
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"n6001/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
- C-Compiler: gcc 8.5.0 or above
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
- CMake: version 3.15 or above
- Python: version 3.6.8 or above
- Intel Quartus Prime Pro 24.1: The ASE must find the
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.
The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
- Compilation of the SystemVerilog Direct Programming Interface (DPI) constructs
- Compilation of the standard examples that are included in the installation
- Support for SystemC
"},{"location":"n6001/ug_dev_afu_sim_env/#4-package-description","title":"4. Package Description","text":"The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\n
This directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.
"},{"location":"n6001/ug_dev_afu_sim_env/#41-ase-scripts","title":"4.1. ASE Scripts","text":"The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
"},{"location":"n6001/ug_dev_afu_sim_env/#411-simulation-tool-set-up","title":"4.1.1. Simulation Tool Set Up","text":"Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
"},{"location":"n6001/ug_dev_afu_sim_env/#412-ase-environment-check","title":"4.1.2. ASE Environment Check","text":"This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\n
"},{"location":"n6001/ug_dev_afu_sim_env/#413-afu-simulation-using-the-ase","title":"4.1.3. AFU Simulation Using the ASE","text":"Before configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
"},{"location":"n6001/ug_dev_afu_sim_env/#4131-afu_sim_setup","title":"4.1.3.1. afu_sim_setup","text":"The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\n
* The only required argument to the afu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.
afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.
generate_ase_environment.py: This script instantiates your simulated platform configuration.
"},{"location":"n6001/ug_dev_afu_sim_env/#4132-rtl_src_configpy","title":"4.1.3.2. rtl_src_config.py","text":"The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
"},{"location":"n6001/ug_dev_afu_sim_env/#4133-generate_ase_environmentpy","title":"4.1.3.3. generate_ase_environment.py","text":"The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
- Synopsys: Creates
synopsys_sim.setup and vcs_run.tcl in the configuration directory. - Siemens: Creates
vsim_run.tcl in the configuration directory.
The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"n6001/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\n
"},{"location":"n6001/ug_dev_afu_sim_env/#5-ase-usage","title":"5. ASE Usage","text":"The AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
- Server: A makefile in the
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code. - Client: The main
cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.
"},{"location":"n6001/ug_dev_afu_sim_env/#51-ase-build-instructions","title":"5.1. ASE Build Instructions","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"n6001/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"n6001/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\n
"},{"location":"n6001/ug_dev_afu_sim_env/#513-install-ase-tools","title":"5.1.3. Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\n
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"n6001/ug_dev_afu_sim_env/#514-ase-simulator-server-build-instructions","title":"5.1.4. ASE Simulator (Server) Build Instructions","text":"ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
Change directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n...
"},{"location":"n6001/ug_dev_afu_sim_env/#514-ase-runtime-instructions","title":"5.1.4. ASE Runtime Instructions","text":"The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
- Terminal 1: In the simulation directroy
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.
Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\n
You can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
- Terminal 2: First set the environment variable
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c.
# Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\n
Note: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\n
Upon completion, the simulation generates the following files:
- Waveform dump:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.
"},{"location":"n6001/ug_dev_afu_sim_env/#52-ase-makefile-targets","title":"5.2. ASE Makefile Targets","text":"COMMAND DESCRIPTION make Build the HW Model using RTL supplied make sim Run simulator - ASE can be run in one of 4 modes set in ase.cfg - A regression mode can be enabled by writing ASE_MODE = 4 in ase.cfg and supplying an ase_regress.sh script make wave Open the waveform (if created) to be run after simulation completes make clean Clean simulation files make distclean Clean ASE sub-distribution"},{"location":"n6001/ug_dev_afu_sim_env/#53-ase-makefile-variables","title":"5.3. ASE Makefile Variables","text":"Makefile switch DESCRIPTION ASE_CONFIG Directly input an ASE configuration file path (ase.cfg) ASE_SCRIPT Directly input an ASE regression file path (ase_regress.sh, for ASE_MODE=4) SIMULATOR Directly input a simulator brand (select between 'VCS' or 'QUESTA') ASE_DISABLE_CHECKER Legacy - Disable CCI-P protocol checker module (set to '1' might speed up simulation) WARNING => NO warnings on hazards, protocol checks, timeouts will be generated. This option must be ONLY used if the design is already CCI-P compliant and fast simulation of app-specific logic is needed"},{"location":"n6001/ug_dev_afu_sim_env/#54-ase-runtime-configuration-options","title":"5.4. ASE Runtime Configuration Options","text":"The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
Switch Name Default Description ASE_MODE 1 ASE mode has the following valid values: 1 : Standard Server-Client Mode2 : Simulator stops after ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"n6001/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
- ASE_INFO: Prints mandatory information messages required to specify operation.
- ASE_ERR: Prints error messages during operation.
- ASE_MSG: Prints general messages indicating check points in the ASE. Suppress these messages by setting the environment variable
ASE_LOG to 0.
Two log levels are supported in ASE, controlled by env(ASE_LOG)
- ASE_LOG=0 | ASE_LOG_SILENT : Only INFO, ERR messages are posted
- ASE_LOG!=0 | ASE_LOG_MESSAGE : All MSG, INFO, ERR messages are posted
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\n
You cannot suppress warnings and errors."},{"location":"n6001/ug_dev_afu_sim_env/#56-troubleshooting-and-error-reference","title":"5.6. Troubleshooting and Error Reference","text":"The following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If using ASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"n6001/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
- AFU Build environment
- Using the PIM to interface with an AFU
- AFU Design
- Software Development
- Packaging the AFU
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"n6001/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X A Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\n
The OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"n6001/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
- ofs_plat_if.vh: PIM's top level wrapper interface for passing all top-level interfaces into an AFU and is copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.
- PIM interfaces are defined in base_ifcs and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).
- PIM modules are defined in ifcs_classes and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support: - host_chan
- local_memory
- hssi
- Other
- PIM utilities are defined in utils and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.
"},{"location":"n6001/ug_dev_pim_based_afu/#3-using-pim-to-interface-with-an-afu","title":"3. Using PIM to interface with an AFU","text":"To interface the PIM with an AFU:
- Create top level module ofs_plat_afu.sv.
- For each Subsystem used by your AFU, create individual channel interfaces using your selected bus protocols and connect the channel PIM Shims based on selected bus protocols.
- PCIe - Host Channel
- Local Memory
- HSSI
- Tie off all unused channels/ports.
- Connect the channel interfaces to the AFU module.
"},{"location":"n6001/ug_dev_pim_based_afu/#31-top-level-module-ofs_plaf_afu","title":"3.1. Top Level Module - ofs_plaf_afu","text":"For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\n
The SystemVerilog interface ofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"n6001/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"n6001/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
- AVMM
- AVMM-RDWR
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n
The Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
- AVMM
- AXI-Lite
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n
"},{"location":"n6001/ug_dev_pim_based_afu/#322-connect-the-host-channel-to-the-pim-shim","title":"3.2.2. Connect the host channel to the PIM Shim","text":"Using the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#323-avalon-example","title":"3.2.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#33-local-memory","title":"3.3. Local Memory","text":"Local memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"n6001/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
- AVMM
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\n
"},{"location":"n6001/ug_dev_pim_based_afu/#332-connect-local-memory-to-the-pim-shim","title":"3.3.2. Connect local memory to the PIM Shim","text":"Using the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\n
"},{"location":"n6001/ug_dev_pim_based_afu/#333-avalon-example","title":"3.3.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\n
"},{"location":"n6001/ug_dev_pim_based_afu/#34-high-speed-serial-interface-hssi","title":"3.4. High Speed Serial Interface (HSSI)","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"n6001/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\n
"},{"location":"n6001/ug_dev_pim_based_afu/#35-tie-off-unused-ports","title":"3.5. Tie Off Unused ports","text":"In digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#36-afu-instantiation","title":"3.6. AFU Instantiation","text":"Instantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\n
"},{"location":"n6001/ug_dev_pim_based_afu/#4-afu","title":"4. AFU","text":"The AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"n6001/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#42-afu-interfaces","title":"4.2. AFU Interfaces","text":"The AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\n
The Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\n
"},{"location":"n6001/ug_dev_pim_based_afu/#43-csr-interface","title":"4.3. CSR Interface","text":"The MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"n6001/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"n6001/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\n
"},{"location":"n6001/ug_dev_pim_based_afu/#46-systemverilog-packages","title":"4.6. SystemVerilog Packages","text":"The AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
- Host Channel Package: ofs_plat_host_chan_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv - Local Memory Package: local_mem_cfg_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.sv
The HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
- HSSI Channel Package: ofs_fim_eth_if_pkg
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\n
"},{"location":"n6001/ug_dev_pim_based_afu/#5-host-software-development","title":"5. Host Software Development","text":"The host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"n6001/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"n6001/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\n
- power - Accelerator Function power consumption, in watts. Set to 0 for Intel ADP platforms.
- clock-frequency-high - Clock frequency for uclk_usr in MHz. (optional)
- clock-frequency-low - Clock frequency for uclk_usr_div2 in MHz. (optional)
- afu-top-interface:
- class : Set to \"ofs_plat_afu\" for PIM based AFU, \"afu_main\" for native/hybrid AFU's.
- accelerator-clusters:
- name : name of AFU
- total-contexts : Set to '1'
- accelerator-type-uuid : 128-bit Universally Unique Identifier (UUID) used to identify the AFU.
The ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\n
The Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\n
"},{"location":"n6001/ug_dev_pim_based_afu/#62-source-list-file","title":"6.2. Source List File","text":"The source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\n
"},{"location":"n6001/ug_dev_pim_based_afu/#63-directory-structure","title":"6.3. Directory Structure","text":"Below is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\n
"},{"location":"n6001/ug_dev_pim_based_afu/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
- Set up the Intel\u00ae Quartus\u2122 Prime Pro Edition Software in a host server
- Set up the Docker engine
- Build and load your Docker image to the Docker engine
- Run a Docker container with OFS preloaded
The Open FPGA Stack (OFS) Docker image has two main personas:
- Development: You can develop, simulate, and build any component of the OFS. The Docker image enables you to use your laptop or server without having drivers, FPGA Platform, or specific Linux* distribution installed in your host computer. You can follow the development flow provided to run Docker on Linux.
- Deployment: You can program, load binaries, or execute real-time testing using the OPAE and OFS. To do so, the host computer must have the specified software distribution and drivers installed.
"},{"location":"n6001/ug_docker/#12-background-information","title":"1.2 Background Information","text":"A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"n6001/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":" - What is a container?
- Docker vs. Virtual Machines
- Does the Docker container have its own Kernel?
- No, Docker image or Container uses the application layer of the host computer; this functionality is the main reason for docker having lightweight and fast applications.
- Does Docker run on Linux, macOS, and Windows?
- Intel Docker Image can use the PCIe card from the host server?
- Yes, The drivers and additional information could be shared, but this could create potential security concerns (ensure your system is secure).
- Docker security
- Docker subscription
"},{"location":"n6001/ug_docker/#20-prerequisites-and-scope","title":"2.0 Prerequisites and Scope","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"n6001/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"n6001/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"n6001/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"n6001/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\n
-
Add the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\n
-
Install the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"n6001/ug_docker/#ubuntu-2204","title":"Ubuntu 22.04","text":"The Docker installation steps for Ubuntu are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\n
-
Install packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\n
-
Add Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\n
-
The following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\n
-
Update the package manager index again:
sudo apt-get update\n
-
Install Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"n6001/ug_docker/#33-load-docker-image-installation","title":"3.3 Load Docker Image installation","text":"The Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"n6001/ug_docker/#build-the-image","title":"Build the image","text":" -
You can download the Dockefile from OFS GitHub Docker.
-
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\n
Save the file
-
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\n
Note: Never remove --no-cache this could cause issues with your environmental variables inside of the container
-
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\n
"},{"location":"n6001/ug_docker/#volumen-creation","title":"Volumen creation","text":" -
Docker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\n
For more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
-
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\n
-
Inside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\n
The docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
"},{"location":"n6001/ug_docker/#34-create-a-container","title":"3.4 Create a container","text":"Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
- Using the previous example now, you can execute the docker run command.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
-
Now the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\n
Your Container ID is bdc1289fb081.
"},{"location":"n6001/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\n
The container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
-
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the evaluation script.
-
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\n
-
The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n
Now you can control and compile the design. You can use the interactive or de-attach mode. -
If you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\n
all the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
"},{"location":"n6001/ug_docker/#40-deployment","title":"4.0 Deployment","text":"The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"n6001/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
- Follow the steps listed in sections 2.1 to 2.3
- 2.1 Quartus installation
- 2.2 Docker Engine installation
- 2.3 Load Docker Image installation
- The steps required for DFL driver installation are documented Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"n6001/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
-
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Example, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
-
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\n
-
Now, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n
\u200b Your Container ID is 25b41eb4d232.
"},{"location":"n6001/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\n
The container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
-
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the eval script.
-
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n
- The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\n
Now you can control and compile the design using the interactive or de-attach mode.
"},{"location":"n6001/ug_docker/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
- Install the necessary tools, such as virt-manager, on the host machine. This may involve downloading and installing the software from the internet.
- Enable the virtualization feature on the host machine. This may involve going into the BIOS settings and enabling hardware-assisted virtualization or using a command-line tool to enable it in the operating system.
- Use virt-manager to create a new virtual machine and configure its settings. This may involve choosing a name and operating system for the virtual machine and setting the amount of memory and storage it will use.
- Install the OPAE (Open Programmable Acceleration Engine) tool on the virtual machine. This may involve downloading and installing the OPAE software.
- Install the DFL (Data Field Level) drivers on the virtual machine. These drivers allow the virtual machine to access and use the PCIe devices on the host machine. This may involve downloading and installing the drivers from the internet.
- Once all of the steps have been completed, you should be able to use the virtual machine to access and use the PCIe devices on the host machine. You may need to configure the virtual machine's settings to enable it to use the PCIe devices, such as by assigning a specific device to the virtual machine.
"},{"location":"n6001/ug_kvm/#1-modes-of-operation","title":"1. Modes of Operation","text":"Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
-
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
-
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
"},{"location":"n6001/ug_kvm/#2-enable-virtualization","title":"2. Enable Virtualization","text":"To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\n
This command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\n
In this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"n6001/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":" - Open a terminal window and log in as a user with sudo privileges.
- Check if the virtualization kernel modules are loaded by running the following command:
lsmod | grep kvm\n
-
If the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
-
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\n
- If the kernel modules are not loaded, and you cannot load them manually, it may be because virtualization is not supported or enabled in your system's BIOS or UEFI settings. You must reboot your system and enter the BIOS or UEFI settings menu to enable virtualization. The exact steps for doing this may vary depending on your system's hardware and BIOS/UEFI version, so consult your motherboard or system documentation for specific instructions.
"},{"location":"n6001/ug_kvm/#4-install-virtual-machine-manager","title":"4. Install Virtual Machine Manager","text":"Virtual Machine Manager (also known as libvirt) can be installed by following the below steps:
- Open a terminal window and log in as a user with sudo privileges.
-
Update your system package index by running the following command:
- Redhat
sudo dnf update\n
* Ubuntu sudo apt update\n
-
Install the libvirt package and any required dependencies by running the following command:
- Redhat
sudo dnf install @virtualization\n
- Ubuntu
sudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\n
-
Start the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\n
- Optional: Install the virt-manager package, which provides a GUI application for managing virtual machines, by running the following command:
sudo dnf install virt-manager\n
- Optional: If you want to be able to run virtual machines as a non-root user, add your user to the libvirt group by running the following command, replacing \"USERNAME\" with your username:
sudo usermod -a -G libvirt USERNAME\n
- You can now launch virt-manager by running the command
virt-manager as the non-root user.
Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\n
You will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\n
- Reboot your server to apply the changes
reboot\n
After completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"n6001/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
- Start the
virt-manager GUI by running the following command:
sudo virt-manager&\n
-
Create a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
-
In the virt-manager window, click the \"New virtual machine\" button.
-
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
-
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
-
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
-
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
-
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
-
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
-
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
-
Click on the \"+\" icon (Bottom left) to create the storage pool.
-
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
-
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
-
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
-
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
-
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
"},{"location":"n6001/ug_kvm/#51-passing-devices-to-the-vm","title":"5.1 Passing Devices to the VM","text":"In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"n6001/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n
\u200b
"},{"location":"n6001/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
The following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"n6001/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Ensure you add the desired VF in your PCIE devices list.
"},{"location":"n6001/ug_kvm/#513-2-pf-mode","title":"5.1.3 2 PF Mode","text":"For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
-
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n
-
Pass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
-
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\n
"},{"location":"n6001/ug_kvm/#52-virt-manager-configuration-changes","title":"5.2 Virt-Manager Configuration Changes","text":" -
Edit the XML file for your machine and include the following
-
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\n
-
Inside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\n
-
Ensure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\n
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Save the changes \"Apply\"
-
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\n
-
Ensure your devices are enumerated properly.
-
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n
2. Deployment Mode:
B1:00.5\n
-
Under the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n
2. Deployment Mode:
177:00.0\n
-
Click on \"Begin Installation.\" and follow the wizard installation of the OS.
-
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
-
Under your virtual machine, configure your VM proxy:
- Redhat How to apply a system-wide proxy?
- Ubuntu Define proxy settings
- Configure Git to use a proxy
-
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
-
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\n
-
Use the Virtual function (Not supported at management mode)
-
Ensure the DFL kernel drivers is install in your VM system
-
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\n
-
Verify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n
-
Test the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\n
After the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
"},{"location":"n6001/ug_kvm/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"sw/build_chain/fpga_api/api_build/","title":"Building OPAE SDK Artifacts","text":""},{"location":"sw/build_chain/fpga_api/api_build/#steps","title":"Steps","text":" - Fetch the OPAE SDK source tree
- Configure the OPAE SDK CMake project
- Build OPAE SDK targets
The example below lists commands that can be used to fetch and build OPAE SDK.
# fetch the source\ngit clone https://github.com/OPAE/opae-sdk.git\ncd opae-sdk\n# configure CMake\ncmake ..\n# build\nmake\n
For a list of targets that can be built, type make help from the build directory.
CMake options that may be set during the configuration include the following:
|----------------------------|-----------------------|-------------------------------------|---------------------------------------|----------------|\n| cmake flag | Optional or Mandatory | Purpose | Valid values | Default value |\n|----------------------------|-----------------------|-------------------------------------|---------------------------------------|----------------|\n| -DCMAKE_BUILD_TYPE | Optional | Set compiler flags | Debug/Release/Coverage/RelWithDebInfo | RelWithDebInfo |\n| -DOPAE_BUILD_LEGACY | Optional | Enable/disable opae-legacy.git | ON/OFF | OFF |\n| -DOPAE_BUILD_SPHINX_DOC | Optional | Enable/disable documentation build | ON/OFF | OFF |\n| -DOPAE_BUILD_TESTS | Optional | Enable/disable building unit tests | ON/OFF | OFF |\n| -DOPAE_INSTALL_RPATH | Optional | Enable/disable rpath for install | ON/OFF | OFF |\n| -DOPAE_BUILD_LIBOPAE_CXX | Optional | Enable/disable OPAE C++ bindings | ON/OFF | ON | \n| -DOPAE_WITH_PYBIND11 | Optional | Enable/disable pybind11 binaries | ON/OFF | ON |\n| -DOPAE_BUILD_PYTHON_DIST | Optional | Enable/disable Python Distribution | ON/OFF | OFF |\n| -DOPAE_ENABLE_MOCK | Optional | Enable/disable mocks for unit tests | ON/OFF | OFF |\n| -DOPAE_BUILD_SIM | Optional | Enable/disable opae-sim.git | ON/OFF | OFF |\n
"},{"location":"sw/build_chain/fpga_driver/driver_build/","title":"Building the OPAE Intel FPGA driver (out-of-tree)","text":"The Intel FPGA driver included with OPAE SDK releases is packaged as an RPM or DEB package as well as a source tarball. Starting with OPAE SDK release of 1.4, the driver can be built from source out-of-tree but requires the following packages:
For RPM package managers (Red Hat, CentOS, Fedora, etc.) * kernel-headers * kernel-devel * gcc * make
For DEB package managers (Debian, Ubuntu, etc.) * kernel-headers-generic * gcc * make
After installation of necessary distribution packages, follow the steps in the example below to build the Intel Kernel driver. NOTE The example below references Intel FPGA Kernel driver version 2.0.2. but can be applied to later versions.
tar zxf opae-intel-fpga-driver-2.0.2-1.tar.gz\ncd opae-intel-fpga-driver-2.0.2\nmake\n
"},{"location":"sw/drv_arch/drv_arch/","title":"Open Programmable Accelerator Engine (OPAE) Linux Device Driver Architecture","text":"The OPAE FPGA Linux Device Driver provides interfaces for user-space applications to configure, enumerate, open, and access FPGA accelerators on platforms equipped with Intel FPGA solutions. The OPAE FPGA driver also enables system-level management functions such as FPGA reconfiguration and virtualization.
"},{"location":"sw/drv_arch/drv_arch/#hardware-architecture","title":"Hardware Architecture","text":"The Linux Operating System treats the FPGA hardware as a PCIe* device. A predefined data structure, Device Feature List (DFL), allows for dynamic feature discovery in an Intel FPGA solution.
The Linux Device Driver implements PCIe Single Root I/O Virtualization (SR-IOV) for the creation of Virtual Functions (VFs). The device driver can release individual accelerators for assignment to virtual machines (VMs).
"},{"location":"sw/drv_arch/drv_arch/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FPGA Management Engine provides error reporting, reconfiguration, performance reporting, and other infrastructure functions. Each FPGA has one FME which is always accessed through the Physical Function (PF). The Intel Xeon\u00ae Processor with Integrated FPGA also performs power and thermal management. These functions are not available on the Intel Programmable Acceleration Card (PAC).
User-space applications can acquire exclusive access to the FME using open(), and release it using close(). Device access may be managed by standard Linux interfaces and tools.
If an application terminates without freeing the FME or Port resources, Linux closes all file descriptors owned by the terminating process, freeing those resources.
"},{"location":"sw/drv_arch/drv_arch/#port","title":"Port","text":"A Port represents the interface between two components: * The FPGA Interface Manager (FIM) which is part of the static FPGA fabric * The Accelerator Function Unit (AFU) which is the partially reconfigurable region
The Port controls the communication from software to the AFU and makes features such as reset and debug available.
"},{"location":"sw/drv_arch/drv_arch/#accelerator-function-unit-afu","title":"Accelerator Function Unit (AFU)","text":"An AFU attaches to a Port. The AFU provides a 256 KB memory mapped I/O (MMIO) region for accelerator-specific control registers.
- Use
open() on the Port device to acquire access to an AFU associated with the Port device. - Use
close()on the Port device to release the AFU associated with the Port device. - Use
mmap() on the Port device to map accelerator MMIO regions.
"},{"location":"sw/drv_arch/drv_arch/#partial-reconfiguration-pr","title":"Partial Reconfiguration (PR)","text":"Use PR to reconfigure an AFU from a bitstream file. Successful reconfiguration has the following requirement:
- You must generate the reconfiguration AFU for the exact FIM. The AFU and FIM are compatible if their interface IDs match. You can verify this match by comparing the interface ID in the bitstream header against the interface ID that is exported by the driver in sysfs.
In all other cases PR fails and may cause system instability.
Platforms that support 512-bit Partial Reconfiguration require binutils >= version 2.25.
Close any software programs accessing the FPGA, including those running in a virtualized host before initiating PR. For virtualized environments, the recommended sequence is as follows:
- Unload the driver from the guest
- Release the VF from the guest
Releasing the VF from the guest while an application on the guest is still accessing its resources may lead to VM instabilities. We recommend closing all applications accessing the VF in the guest before releasing the VF.
- Disable SR-IOV
- Perform PR
- Enable SR-IOV
- Assign the VF to the guest
- Load the driver in the guest
"},{"location":"sw/drv_arch/drv_arch/#fpga-virtualization","title":"FPGA Virtualization","text":"To enable accelerator access from applications running on a VM, create a VF for the port using the following process:
-
Release the Port from the PF using the associated ioctl on the FME device.
-
Use the following command to enable SR-IOV and VFs. Each VF can own a single Port with an AFU. In the following command, N is the number of Port released from the PF.
echo N > $PCI_DEVICE_PATH/sriov_numvfs\n
The number, 'N', cannot be greater than the number of supported VFs. This can be read from $PCI_DEVICE_PATH/sriov_totalvfs.
-
Pass the VFs through to VMs using hypervisor interfaces.
-
Access the AFU on a VF from applications running on the VM using the same driver inside the VM.
Creating VFs is only supported for port devices. Consequently, PR and other management functions are only available through the PF.
If assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices).
"},{"location":"sw/drv_arch/drv_arch/#driver-organization","title":"Driver Organization","text":""},{"location":"sw/drv_arch/drv_arch/#pcie-module-device-driver","title":"PCIe Module Device Driver","text":"FPGA devices appear as a PCIe devices. Once enumeration detects a PCIe PF or VF, the Linux OS loads the FPGA PCIe device driver. The device driver performs the following functions:
- Walks through the Device Feature List in PCIe device base address register (BAR) memory to discover features and their sub-features and creates necessary platform devices.
- Enables SR-IOV.
- Introduces the feature device infrastructure, which abstracts operations for sub-features and provides common functions to feature device drivers.
"},{"location":"sw/drv_arch/drv_arch/#pcie-module-device-driver-functions","title":"PCIe Module Device Driver Functions","text":"The PCIe Module Device Driver performs the following functions:
- PCIe discovery, device enumeration, and feature discovery.
- Creates sysfs directories for the device, FME, and Port.
- Creates the platform driver instances, causing the Linux kernel to load their respective drivers.
"},{"location":"sw/drv_arch/drv_arch/#fme-platform-module-device-driver","title":"FME Platform Module Device Driver","text":"The FME Platform Module Device Driver loads automatically after the PCIe driver creates the FME Platform Module. It provides the following features for FPGA management:
-
Power and thermal management, error reporting, performance reporting, and other infrastructure functions. You can access these functions via sysfs interfaces the FME driver provides.
-
Partial Reconfiguration. During PR sub-feature initialization, the FME driver registers the FPGA Manager framework to support PR. When the FME receives the relevant ioctl request from user-space, it invokes the common interface function from the FPGA Manager to reconfigure the AFU using PR.
-
Port management for virtualization (releasing/assigning port device).
After a port device is released, you can use the PCIe driver SR-IOV interfaces to create/destroy VFs.
For more information, refer to \"FPGA Virtualization\".
"},{"location":"sw/drv_arch/drv_arch/#fme-platform-module-device-driver-functions","title":"FME Platform Module Device Driver Functions","text":"The FME Platform Module Device Driver performs the the following functions:
- Creates the FME character device node.
- Creates the FME sysfs files and implements the FME sysfs file accessors.
- Implements the FME private feature sub-drivers.
- FME private feature sub-drivers: * FME Header * Partial Reconfiguration * Global Error * Global Performance
"},{"location":"sw/drv_arch/drv_arch/#port-platform-module-device-driver","title":"Port Platform Module Device Driver","text":"After the PCIe Module Device Driver creates the Port Platform Module device, the FPGA Port and AFU driver are loaded. This module provides an interface for user-space applications to access the individual accelerators, including basic reset control on the Port, AFU MMIO region export, DMA buffer mapping service, and remote debug functions.
"},{"location":"sw/drv_arch/drv_arch/#port-platform-module-device-driver-functions","title":"Port Platform Module Device Driver Functions","text":"The Port Platform Module Device Driver performs the the following functions:
- Creates the Port character device node.
- Creates the Port sysfs files and implements the Port sysfs file accessors.
- Implements the following Port private feature sub-drivers. * Port Header * AFU * Port Error * Signal Tap
"},{"location":"sw/drv_arch/drv_arch/#opae-fpga-driver-interface","title":"OPAE FPGA Driver Interface","text":"The user-space interface consists of a sysfs hierarchy and ioctl requests. Most kernel attributes can be accessed/modified via sysfs nodes in this hierarchy. More complex I/O operations are controlled via ioctl requests. The OPAE API implementation, libopae-c, has been designed to use this interface to interact with the OPAE FPGA kernel drivers.
"},{"location":"sw/fpga_api/fpga_api/","title":"OPAE C API Reference","text":"The reference documentation for the OPAE C API is grouped into the following sections:
- Types
- types.h
- types_enum.h
- Enumeration API
- enum.h
- properties.h
- Access API
- access.h
- Event API
- event.h
- MMIO and Shared Memory APIs
- mmio.h
- buffer.h
- umsg.h
- Management API
- manage.h
- Metrics API
- metrics.h
- SysObject
- sysobject.h
- Utilities
- utils.h
- Samples
- hello_fpga.c
- hello_events.c
"},{"location":"sw/fpga_api/fpga_api/#types","title":"Types","text":"The OPAE C API defines a number of types; most prominent are the types fpga_token, fpga_handle, and fpga_properties. All regular types are defined in types.h, while the values of enumeration types are defined in types_enum.h.
"},{"location":"sw/fpga_api/fpga_api/#typesh","title":"types.h","text":"types.h
"},{"location":"sw/fpga_api/fpga_api/#types_enumh","title":"types_enum.h","text":"types_enum.h
"},{"location":"sw/fpga_api/fpga_api/#enumeration-api","title":"Enumeration API","text":"The OPAE enumeration API allows selective discovery of FPGA resources. When enumerating resources, a list of filter criteria can be passed to the respective function to select a subset of all resources in the system. The fpgaEnumerate() function itself then returns a list of fpga_tokens denoting resources, which can be used in subsequent API calls.
Filter criteria are specified using one or more fpga_properties object. These objects need to be created using fpgaGetProperties() (defined in ) before being passed to fpgaEnumerate(). Individual attributes of an fpga_properties object are set using specific accessors, which are also defined in ."},{"location":"sw/fpga_api/fpga_api/#enumh","title":"enum.h","text":"
enum.h
"},{"location":"sw/fpga_api/fpga_api/#propertiesh","title":"properties.h","text":"properties.h
"},{"location":"sw/fpga_api/fpga_api/#access-api","title":"Access API","text":"The access API provides functions for opening and closing FPGA resources. Opening a resource yields an fpga_handle, which denotes ownership and can be used in subsequent API calls to interact with a specific resource. Ownership can be exclusive or shared.
"},{"location":"sw/fpga_api/fpga_api/#accessh","title":"access.h","text":"access.h
"},{"location":"sw/fpga_api/fpga_api/#event-api","title":"Event API","text":"The event API provides functions and types for handling asynchronous events such as errors or accelerator interrupts.
To natively support asynchronous event, the driver for the FPGA platform needs to support events natively (in which case the OPAE C library will register the event directly with the driver). For some platforms that do not support interrupt-driven event delivery, you need to run the FPGA Daemon (fpgad) to enable asynchronous OPAE events. fpgad will act as a proxy for the application and deliver asynchronous notifications for registered events.
"},{"location":"sw/fpga_api/fpga_api/#eventh","title":"event.h","text":"event.h
"},{"location":"sw/fpga_api/fpga_api/#mmio-and-shared-memory-apis","title":"MMIO and Shared Memory APIs","text":"These APIs feature functions for mapping and accessing control registers through memory-mapped IO (mmio.h), allocating and sharing system memory buffers with an accelerator (buffer.h), and using low-latency notifications (umsg.h).
"},{"location":"sw/fpga_api/fpga_api/#mmioh","title":"mmio.h","text":"mmio.h
"},{"location":"sw/fpga_api/fpga_api/#bufferh","title":"buffer.h","text":"buffer.h
"},{"location":"sw/fpga_api/fpga_api/#umsgh","title":"umsg.h","text":"umsg.h
"},{"location":"sw/fpga_api/fpga_api/#management-api","title":"Management API","text":"The management APIs define functions for reconfiguring an FPGA (writing new partial bitstreams) as well as assigning accelerators to host interfaces.
"},{"location":"sw/fpga_api/fpga_api/#manageh","title":"manage.h","text":"manage.h
"},{"location":"sw/fpga_api/fpga_api/#metrics-api","title":"Metrics API","text":"The metrics APIs define functions for discovery/enumeration of metrics information and reading metrics values.
"},{"location":"sw/fpga_api/fpga_api/#metricsh","title":"metrics.h","text":"metrics.h
"},{"location":"sw/fpga_api/fpga_api/#sysobject","title":"SysObject","text":"The SysObject API can be used to get system objects by name. Names used with the SysObject API are driver-specific and may not be compatible across plugins and/or drivers. For example, SysObject names used with the xfpga plugin will apply to the OPAE Linux Kernel driver and refer to sysfs nodes under the sysfs tree for the resource used with the SysObject API.
"},{"location":"sw/fpga_api/fpga_api/#sysobjecth","title":"sysobject.h","text":"sysobject.h
"},{"location":"sw/fpga_api/fpga_api/#utilities","title":"Utilities","text":"Functions for mapping fpga_result values to meaningful error strings are provided by the utilities API.
"},{"location":"sw/fpga_api/fpga_api/#utilsh","title":"utils.h","text":"utils.h
"},{"location":"sw/fpga_api/fpga_api/#samples","title":"Samples","text":"Code samples demonstrate how to use OPAE C API.
"},{"location":"sw/fpga_api/fpga_api/#hello_fpgac","title":"hello_fpga.c","text":"hello_fpga.c
"},{"location":"sw/fpga_api/fpga_api/#hello_eventsc","title":"hello_events.c","text":"hello_events.c
"},{"location":"sw/fpga_api/fpga_cxx_api/","title":"OPAE C++ Core API Reference","text":"The reference documentation for the OPAE C++ Core API is grouped into the following sections:
- Overview
- Goals
- Simplicity
- Extensibility and Interoperability
- Modern C++ Coding Practices
- Error Handling
- Coding Style
- Fundamental Types
- Properties
- pvalue.h
- properties.h
- Resource Classes
- token.h
- handle.h
- shared_buffer.h
- errors.h
- events.h
- sysobject.h
- Exceptions
- except.h
- Misc
- version.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#overview","title":"Overview","text":"The OPAE C++ API enables C++ developers with the means to use FPGA resources by integrating the OPAE software stack into C++ applications.
"},{"location":"sw/fpga_api/fpga_cxx_api/#goals","title":"Goals","text":""},{"location":"sw/fpga_api/fpga_cxx_api/#simplicity","title":"Simplicity","text":"Keep the API as small and lightweight as possible. Although features such as system validation and orchestration are beyond the scope of this API, using this API for their development should be relatively easy.
"},{"location":"sw/fpga_api/fpga_cxx_api/#extensibility-and-interoperability","title":"Extensibility and Interoperability","text":"While keeping to the goal of simplicity, the OPAE C++ API is designed to allow for better reuse by either extending the API or by integrating with other languages.
"},{"location":"sw/fpga_api/fpga_cxx_api/#modern-c-coding-practices","title":"Modern C++ Coding Practices","text":"The OPAE C++ API uses the C++ 11 standard library and makes use of its features whenever practical. The OPAE C++ API is also designed to require the minimum number of third-party libraries/dependencies.
"},{"location":"sw/fpga_api/fpga_cxx_api/#error-handling","title":"Error Handling","text":"The OPAE C++ API is designed to throw exceptions when appropriate. The structure of OPAE C++ exceptions is similar to the error codes in the OPAE C API. This gives users of the API more freedom on error handling while providing better debug information in cases of failure.
"},{"location":"sw/fpga_api/fpga_cxx_api/#coding-style","title":"Coding Style","text":"For formatting of the OPAE C++ API complies with most of the recommendations of the Google C++ style. For example, the OPAE C++ API uses:
- opening braces on the same line as their scope definition
- spaces instead of tabs for indentation
- indentation of two spaces
"},{"location":"sw/fpga_api/fpga_cxx_api/#fundamental-types","title":"Fundamental Types","text":"Basic types for the OPAE C++ API are found in the opae::fpga::types namespace. They serve as an adapter layer between the OPAE C API and the OPAE C++ layer. Aside from providing a C++ binding to the C fundamental types, these types also:
- manage the lifetime and scope of the corresponding C struct.
- For example a C++ destructor will take care of calling the appropriate C function to release the data structure being wrapped.
- provide a friendly syntax for using the OPAE C type.
Most classes in this namespace have a c_type() method that returns the C data structure being wrapped, making it easy to use the OPAE C++ type with the OPAE C API. Alternatively, most classes in this namespace have implicit conversion operators that enable interoperability with the OPAE C API.
"},{"location":"sw/fpga_api/fpga_cxx_api/#properties","title":"Properties","text":"C++ class properties wraps fpga_properties and uses pvalue and guid_t to get and set properties stored in an instance of an fpga_properties. pvalue and guid_t are designed to call an accessor method in the OPAE C API to either read property values or write them. Most accessor methods in the OPAE C API share a similar signature, so pvalue generalizes them into common operations that translate into calling the corresponding C API function. guid_t follows similar patterns when reading or assigning values.
"},{"location":"sw/fpga_api/fpga_cxx_api/#pvalueh","title":"pvalue.h","text":"pvalue.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#propertiesh","title":"properties.h","text":"properties.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#resource-classes","title":"Resource Classes","text":"The token, handle, and shared_buffer classes are used to enumerate and access FPGA resources. properties are used to narrow the search space for token's. Before enumerating the accelerator resources in the system, applications can produce one or more properties objects whose values are set to the desired characteristics for the resource. For example, an application may search for an accelerator resource based on its guid.
Once one or more token's have been enumerated, the application must choose which token's to request. The token is then converted to a handle by requesting that a handle object be allocated and opened for it.
Once a handle has been successfully opened, the application can read and write the associated configuration and status space. Additionally, the application may use the handle to allocate shared_buffer's or to register event's. The shared_buffer and event objects retain a reference to their owning handle so that the handle does not lose scope before freeing the shared_buffer and event objects.
"},{"location":"sw/fpga_api/fpga_cxx_api/#tokenh","title":"token.h","text":"token.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#handleh","title":"handle.h","text":"handle.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#shared_bufferh","title":"shared_buffer.h","text":"shared_buffer.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#errorsh","title":"errors.h","text":"errors.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#eventsh","title":"events.h","text":"events.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#sysobjecth","title":"sysobject.h","text":"sysobject.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#exceptions","title":"Exceptions","text":"When the OPAE C++ API encounters an error from the OPAE C API, it captures the current source code location and the error code into an object of type except, then throws the except. Applications should implement the appropriate catch blocks required to respond to runtime exceptions.
"},{"location":"sw/fpga_api/fpga_cxx_api/#excepth","title":"except.h","text":"except.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#misc","title":"Misc","text":"The version class wraps the OPAE C version API.
"},{"location":"sw/fpga_api/fpga_cxx_api/#versionh","title":"version.h","text":"version.h
"},{"location":"sw/fpga_api/fpga_python_api/","title":"OPAE Python API Reference","text":"The reference documentation for the OPAE Python API and is grouped into the following sections:
- Module Types, Methods, and Constants
- Fundamental Types
- Properties
- Token
- Handle
- Event
- Shared Buffer
- Error
- SysObject
"},{"location":"sw/fpga_api/fpga_python_api/#module-types-methods-and-constants","title":"Module Types, Methods, and Constants","text":""},{"location":"sw/fpga_api/fpga_python_api/#fundamental-types","title":"Fundamental Types","text":""},{"location":"sw/fpga_api/fpga_python_api/#properties","title":"Properties","text":""},{"location":"sw/fpga_api/fpga_python_api/#token","title":"Token","text":""},{"location":"sw/fpga_api/fpga_python_api/#handle","title":"Handle","text":""},{"location":"sw/fpga_api/fpga_python_api/#event","title":"Event","text":""},{"location":"sw/fpga_api/fpga_python_api/#shared-buffer","title":"Shared Buffer","text":""},{"location":"sw/fpga_api/fpga_python_api/#error","title":"Error","text":""},{"location":"sw/fpga_api/fpga_python_api/#sysobject","title":"SysObject","text":""},{"location":"sw/fpga_api/plug_guide/readme/","title":"Plugin Developer's Guide","text":""},{"location":"sw/fpga_api/plug_guide/readme/#overview","title":"Overview","text":"Beginning with OPAE C library version 1.2.0, OPAE implements a plugin-centric model. This guide serves as a reference to define the makeup of an OPAE C API plugin and to describe a sequence of steps that one may follow when constructing an OPAE C API plugin.
"},{"location":"sw/fpga_api/plug_guide/readme/#plugin-required-functions","title":"Plugin Required Functions","text":"An OPAE C API plugin is a runtime-loadable shared object library, also known as a module. On Linux systems, the dl family of APIs from libdl are used to interact with shared objects. Refer to \"man dlopen\" and \"man dlsym\" for examples of using the libdl API.
An OPAE C API plugin implements one required function. This function is required to have C linkage, so that its name is not mangled.
int opae_plugin_configure(opae_api_adapter_table *table, const char *config);\n
During initialization, the OPAE plugin manager component loads each plugin, searching for its opae_plugin_configure function. If none is found, then the plugin manager rejects that plugin. When it is found, opae_plugin_configure is called passing a pointer to a freshly-created opae_api_adapter_table and a buffer consisting of configuration data for the plugin.
The job of the opae_plugin_configure function is to populate the given adapter table with each of the plugin's API entry points and to consume and comprehend the given configuration data in preparation for initialization.
"},{"location":"sw/fpga_api/plug_guide/readme/#opae-api-adapter-table","title":"OPAE API Adapter Table","text":"The adapter table is a data structure that contains function pointer entry points for each of the OPAE APIs implemented by a plugin. In this way, it adapts the plugin-specific behavior to the more general case of a flat C API. Note that OPAE applications are only required to link with opae-c. In other words, the name of the plugin library should not appear on the linker command line. In this way, plugins are truly decoupled from the OPAE C API, and they are required to adapt to the strict API specification by populating the adapter table only. No other linkage is required nor recommended.
adapter.h contains the definition of the opae_api_adapter_table. An abbreviated version is depicted below, along with supporting type opae_plugin:
typedef struct _opae_plugin {\nchar *path;\nvoid *dl_handle;\n} opae_plugin;\ntypedef struct _opae_api_adapter_table {\nstruct _opae_api_adapater_table *next;\nopae_plugin plugin;\nfpga_result (*fpgaOpen)(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result (*fpgaClose)(fpga_handle handle);\n...\nfpga_result (*fpgaEnumerate)(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n// configuration functions\nint (*initialize)(void);\nint (*finalize)(void);\n// first-level query\nbool (*supports_device)(const char *device_type);\nbool (*supports_host)(const char *hostname);\n} opae_api_adapter_table;\n
Some points worth noting are that the adapter tables are organized in memory by adding them to a linked list data structure. This is the use of the next structure member. (The list management is handled by the plugin manager.) The plugin structure member contains the handle to the shared object instance, as created by dlopen. This handle is used in the plugin's opae_plugin_configure to load plugin entry points. A plugin need only implement the portion of the OPAE C API that a target application needs. Any API entry points that are not supported should be left as NULL pointers (the default) in the adapter table. When an OPAE API that has no associated entry point in the adapter table is called, the result for objects associated with that plugin will be FPGA_NOT_SUPPORTED.
The following code illustrates a portion of the opae_plugin_configure for a theoretical OPAE C API plugin libfoo.so:
/* foo_plugin.c */\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\nadapter->fpgaOpen = dlsym(adapter->plugin.dl_handle, \"foo_fpgaOpen\");\nadapter->fpgaClose =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaClose\");\n...\nadapter->fpgaEnumerate =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaEnumerate\");\n...\nreturn 0;\n}\n
Notice that the implementations of the API entry points for plugin libfoo.so are prefixed with foo_. This is the recommended practice to avoid name collisions and to enhance the debugability of the application. Upon successful configuration, opae_plugin_configure returns 0 to indicate success. A non-zero return value indicates failure and causes the plugin manager to reject the plugin from futher consideration.
"},{"location":"sw/fpga_api/plug_guide/readme/#plugin-optional-functions","title":"Plugin Optional Functions","text":"Once the plugin manager loads and configures each plugin, it uses the adapter table to call back into the plugin so that it can be made ready for runtime. This is the job of the opae_plugin_initialize entry point, whose signature is defined as:
int opae_plugin_initialize(void);\n
The function takes no parameters, as the configuration data was already given to the plugin by opae_plugin_configure. opae_plugin_initialize returns 0 if no errors were encountered during initialization. A non-zero return code indicates that plugin initialization failed. A plugin makes its opae_plugin_initialize available to the plugin manager by populating the adapter table's initialize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_initialize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->initialize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_initialize\");\n...\nreturn 0;\n}\n
If a plugin does not implement an opae_plugin_initialize entry point, then the initialize member of the adapter table should be left uninitialized. During plugin initialization, if a plugin has no opae_plugin_initialize entry in its adapter table, the plugin initialization step will be skipped, and the plugin will be considered to have initialized successfully.
Once plugin initialization is complete for all loaded plugins, the system is considered to be running and fully functional.
During teardown, the plugin manager uses the adapter table to call into each plugin's opae_plugin_finalize entry point, whose signature is defined as:
int opae_plugin_finalize(void);\n
opae_plugin_finalize returns 0 if no errors were encountered during teardown. A non-zero return code indicates that plugin teardown failed. A plugin makes its opae_plugin_finalize available to the plugin manager by populating the adapter table's finalize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_finalize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->finalize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_finalize\");\n...\nreturn 0;\n}\n
If a plugin does not implement an opae_plugin_finalize entry point, then the finalize member of the adapter table should be left uninitialized. During plugin cleanup, if a plugin has no opae_plugin_finalize entry point in its adapter table, the plugin finalize step will be skipped, and the plugin will be considered to have finalized successfully.
In addition to initialize and finalize, an OPAE C API plugin has two further optional entry points that relate to device enumeration. During enumeration, when a plugin is being considered for a type of device, the plugin may provide input on that decision by exporting an opae_plugin_supports_device entry point in the adapter table:
bool opae_plugin_supports_device(const char *device_type);\n
opae_plugin_supports_device returns true if the given device type is supported and false if it is not. A false return value from opae_plugin_supports_device causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_device is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_device(const char *device_type)\n{\nif (/* device_type is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_device =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_device\");\n...\nreturn 0;\n}\n
.. note::\n The `opae_plugin_supports_device` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\n
Similarly to determining whether a plugin supports a type of device, a plugin may also answer questions about network host support by populating an opae_plugin_supports_host entry point in the adapter table:
bool opae_plugin_supports_host(const char *hostname);\n
opae_plugin_supports_host returns true if the given hostname is supported and false if it is not. A false return value from opae_plugin_supports_host causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_host is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_host(const char *hostname)\n{\nif (/* hostname is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_host =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_host\");\n...\nreturn 0;\n}\n
.. note::\n The `opae_plugin_supports_host` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\n
"},{"location":"sw/fpga_api/plug_guide/readme/#plugin-construction","title":"Plugin Construction","text":"The steps required to implement an OPAE C API plugin, libfoo.so, are:
- Create foo_plugin.c: implements
opae_plugin_configure, opae_plugin_initialize, opae_plugin_finalize, opae_plugin_supports_device, and opae_plugin_supports_host as described in the previous sections. - Create foo_plugin.h: implements function prototypes for each of the plugin-specific OPAE C APIs.
/* foo_plugin.h */\nfpga_result foo_fpgaOpen(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result foo_fpgaClose(fpga_handle handle);\n...\nfpga_result foo_fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n
- Create foo_types.h: implements plugin-specific types for opaque data structures.
/* foo_types.h */\nstruct _foo_token {\n...\n};\nstruct _foo_handle {\n...\n};\nstruct _foo_event_handle {\n...\n};\nstruct _foo_object {\n...\n};\n
- Create foo_enum.c: implements
foo_fpgaEnumerate, foo_fpgaCloneToken, and foo_fpgaDestroyToken. - Create foo_open.c: implements
foo_fpgaOpen. - Create foo_close.c: implements
foo_fpgaClose. - Create foo_props.c: implements
foo_fpgaGetProperties, foo_fpgaGetPropertiesFromHandle, foo_fpgaUpdateProperties - Create foo_mmio.c: implements
foo_fpgaMapMMIO, foo_fpgaUnmapMMIO foo_fpgaWriteMMIO64, foo_fpgaReadMMIO64, foo_fpgaWriteMMIO32, foo_fpgaReadMMIO32. - Create foo_buff.c: implements
foo_fpgaPrepareBuffer, foo_fpgaReleaseBuffer, foo_fpgaGetIOAddress. - Create foo_error.c: implements
foo_fpgaReadError, foo_fpgaClearError, foo_fpgaClearAllErrors, foo_fpgaGetErrorInfo. - Create foo_event.c: implements
foo_fpgaCreateEventHandle, foo_fpgaDestroyEventHandle, foo_fpgaGetOSObjectFromEventHandle, foo_fpgaRegisterEvent, foo_fpgaUnregisterEvent. - Create foo_reconf.c: implements
foo_fpgaReconfigureSlot. - Create foo_obj.c: implements
foo_fpgaTokenGetObject, foo_fpgaHandleGetObject, foo_fpgaObjectGetObject, foo_fpgaDestroyObject, foo_fpgaObjectGetSize, foo_fpgaObjectRead, foo_fpgaObjectRead64, foo_fpgaObjectWrite64. - Create foo_clk.c: implements
foo_fpgaSetUserClock, foo_fpgaGetUserClock.
"},{"location":"sw/fpga_api/prog_guide/readme/","title":"OPAE C API Programming Guide","text":""},{"location":"sw/fpga_api/prog_guide/readme/#overview","title":"Overview","text":"The OPAE C library (libopae-c) is a lightweight user-space library that provides abstractions for FPGA resources in a compute environment. The OPAE C library builds on the driver stack that supports the FPGA device, abstracting hardware- and OS-specific details. It provides access to the underlying FPGA resources as a set of features available to software programs running on the host. These features include the acceleration logic preconfigured on the FPGA and functions to manage and reconfigure the FPGA. The library enables your applications to transparently and seamlessly benefit from FPGA-based acceleration.
By providing a unified C API, the library supports different FPGA integration and deployment models, ranging from single-node systems with one or a few FPGA devices to large-scale FPGA deployments in a data center. At one end of the spectrum, the API supports a simple application using a PCIe link to reconfigure the FPGA with different accelerator functions. At the other end of the spectrum, resource management and orchestration services in a data center can use this API to discover and select FPGA resources and then allocate them for use by acceleration workloads.
"},{"location":"sw/fpga_api/prog_guide/readme/#opae-role","title":"OPAE Role","text":"The OPAE provides a common base layer for a wide range of applications without sacrificing performance or efficiency. The abstraction layer limits the details of the FPGA hardware that software applications must handle.
The OPAE provides consistent interfaces to crucial components of the platform. The OPAE does not constrain frameworks and applications by making optimizations with limited applicability. When the OPAE does provide convenience functions or optimizations, they are optional.
For example, the OPAE provides an interface to allocate physically contiguous buffers in system memory that user-space software and an accelerator can share. This interface enables the most basic feature set of allocating and sharing a large page of memory in one API call. However, it does not provide a malloc()-like interface backed by a memory pool or slab allocator. Higher layers of the software stack can make such domain-specific optimizations.
"},{"location":"sw/fpga_api/prog_guide/readme/#intel-accelerator-stack-hardware-terminology","title":"Intel Accelerator Stack Hardware Terminology","text":"The following terms define the hardware and hardware processes involved in creating an accelerator function.
- FPGA: Field Programmable Gate Array is a discrete or integrated device connecting to a host CPU via PCIe or other type of interconnects.
- Accelerator Function Unit (AFU): The AFU is the supplied implementation of an accelerator, typically in HDL. AFUs implement a function such as compression, encryption, or mathematical operations. The Quartus Prime Pro software synthesizes the RTL logic into a bitstream.
- Accelerator Function (AF): The AF is the compiled binary for an AFU. An AF is a raw binary file (.rbf) bitstream. A tool (fpgaconf) reconfigures the FPGA using an AF bitstream.
- Reconfiguration: The process of reprogramming the FPGA with a different AF.
"},{"location":"sw/fpga_api/prog_guide/readme/#opae-software-concepts-reflected-in-the-c-api","title":"OPAE Software Concepts Reflected in the C API","text":"The following OPAE data structures and functions integrate AFUs into the OPAE environment. The OPAE C API models these data structures and functions. For more information on the object models refer to the Object model section.
- Accelerator: An accelerator is an allocable accelerator function implemented in an FPGA. An accelerator tracks the ownership of an AFU (or part of it) for a process that uses it. Multiple processes can share an accelerator.
- Device: The OPAE enumerates and models two device types: the FPGA and the AFU.
- Events: Events are asynchronous notifications. The FPGA driver triggers particular events to indicate error conditions. Accelerator logic can also define its own events. User applications can choose to be notified when particular events occur and respond appropriately.
- Shared memory buffers: Software allocates shared memory buffers in user process memory on the host. Shared memory buffers facilitate data transfers between the user process and the accelerator that it owns.
"},{"location":"sw/fpga_api/prog_guide/readme/#opae-library","title":"OPAE Library","text":"Linking with this library is straightforward. Code using the OPAE library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, here is the simplest compile and link command:
gcc myprog.c -I</path/to/fpga.h> -L</path/to/libopae-c.so> -lopae-c -luuid -ljson-c -lpthread
.. note::
The OPAE library uses the third-party `libuuid` and `libjson-c` libraries that are not distributed with \nthe OPAE library. Make sure to install these libraries.\n
"},{"location":"sw/fpga_api/prog_guide/readme/#sample-code","title":"Sample Code","text":"The library source includes two code samples. Use these samples to learn how to call functions in the library. Build and run these samples to determine if your installation and environment are set up properly.
Refer to the Running the Hello FPGA Example chapter in the Intel\u00ae Acceleration Stack Quick Start Guide for for Intel Programmable Acceleration Card with Intel Arria\u00ae 10 GX FPGA for more information about using the sample code.
"},{"location":"sw/fpga_api/prog_guide/readme/#high-level-directory-structure","title":"High-Level Directory Structure","text":"Building and installing the OPAE library results in the following directory structure on the Linux OS. Windows and MacOS have similar directories and files.
Directory & Files Contents include/opae Directory containing all header files include/opae/fpga.h Top-level header for user code to include include/opae/access.h Header file for accelerator acquire/release, MMIO, memory management, event handling, and so on include/opae/bitstream.h Header file for bitstream manipulation functions include/opae/common.h Header file for error reporting functions include/opae/enum.h Header file for AFU enumeration functions include/opae/manage.h Header file for FPGA management functions include/opae/types.h Various type definitions lib Directory containing shared library files lib/libopae-c.so The shared dynamic library for linking with the user application doc Directory containing API documentation doc/html Directory for documentation of HTML format doc/latex Directory for documentation of LaTex format doc/man Directory for documentation of Unix man page format"},{"location":"sw/fpga_api/prog_guide/readme/#basic-application-flow","title":"Basic Application Flow","text":"The figure below shows the basic application flow from the viewpoint of a user-process.
"},{"location":"sw/fpga_api/prog_guide/readme/#api-components","title":"API Components","text":"The API object model abstracts the physical FPGA device and available functions. It is a generalized model and extends to describe any FPGA type.
"},{"location":"sw/fpga_api/prog_guide/readme/#object-models","title":"Object Models","text":" fpga_objtype: An enum type that represents the type of an FPGA resource, either FPGA_DEVICE or FPGA_ACCELERATOR. An FPGA_DEVICE object corresponds to a physical FPGA device. Only FPGA_DEVICE objects can invoke management functions. The FPGA_ACCELERATOR represents an instance of an AFU. fpga_token: An opaque type that represents a resource known to, but not necessarily owned by, the calling process. The calling process must own a resource before it can invoke functions of the resource.fpga_handle: An opaque type that represents a resource owned by the calling process. The API functions fpgaOpen() and fpgaClose() acquire and release ownership of a resource that an fpga_handle represents. (Refer to the Functions section for more information.)fpga_properties: An opaque type for a properties object. Your applications use these properties to query and search for appropriate resources. The FPGA Resource Properties section documents properties visible to your applications.fpga_event_handle: An opaque handle the FPGA driver uses to notify your application about an event. fpga_event_type: An enum type that represents the types of events. The following are valid values: FPGA_EVENT_INTERRUPT, FPGA_EVENT_ERROR, and FPGA_EVENT_POWER_THERMAL. (The Intel Programmable Acceleration Card (PAC) with Intel Arria 10 GX FPGA does not handle thermal and power events.)fpga_result: An enum type to represent the result of an API function. If the function returns successfully the result is FPGA_OK. Otherwise, the result is the appropriate error codes. Function fpgaErrStr() translates an error code into human-readable strings.
"},{"location":"sw/fpga_api/prog_guide/readme/#functions","title":"Functions","text":"The table below groups important API calls by their functionality. For more information about each of the functions, refer to the OPAE C API reference manual.
Functionality API Call FPGA Accelerator Description Enumeration fpgaEnumerate() Yes Yes Query FPGA resources that match certain properties Enumeration: Properties fpga[Get, Update, Clear, Clone, Destroy Properties]() Yes Yes Manage fpga_properties life cycle fpgaPropertiesGet[Prop]() Yes Yes Get the specified property Prop, from the FPGA Resource Properties table fpgaPropertiesSet[Prop]() Yes Yes Set the specified property Prop, from the FPGA Resource Properties table Access: Ownership fpga[Open, Close]() Yes Yes Acquire/release ownership Access: Reset fpgaReset() Yes Yes Reset an accelerator Access: Event handling fpga[Register, Unregister]Event() Yes Yes Register/unregister an event to be notified about fpga[Create, Destroy]EventHandle() Yes Yes Manage fpga_event_handle life cycle Access: MMIO fpgaMapMMIO(), fpgaUnMapMMIO() Yes Yes Map/unmap MMIO space fpgaGetMMIOInfo() Yes Yes Get information about the specified MMIO space fpgaReadMMIO[32, 64]() Yes Yes Read a 32-bit or 64-bit value from MMIO space fpgaWriteMMIO[32, 64]() Yes Yes Write a 32-bit or 64-bit value to MMIO space Memory management: Shared memory fpga[Prepare, Release]Buffer() Yes Yes Manage memory buffer shared between the calling process and an accelerator fpgaGetIOAddress() Yes Yes Return the device I/O address of a shared memory buffer fpgaBindSVA() Yes Yes Bind IOMMU shared virtual addressing Management: Reconfiguration fpgaReconfigureSlot() Yes No Replace an existing AFU with a new one Error report fpgaErrStr() Yes Yes Map an error code to a human readable string .. note::
The UMsg APIs are not supported for the Intel(R) PAC cards. They will be deprecated in a future release.\n
"},{"location":"sw/fpga_api/prog_guide/readme/#fpga-resource-properties","title":"FPGA Resource Properties","text":"Applications query resource properties by specifying the property name for Prop in the fpgaPropertiesGet[Prop]() and fpgaPropertiesSet[Prop]() functions. The FPGA and Accelerator columns state whether or not the Property is available for the FPGA or Accelerator objects.
Property FPGA Accelerator Description Parent No Yes fpga_token of the parent object ObjectType Yes Yes The type of the resource: either FPGA_DEVICE or FPGA_ACCELERATOR Bus Yes Yes The bus number Device Yes Yes The PCI device number Function Yes Yes The PCI function number SocketId Yes Yes The socket ID DeviceId Yes Yes The device ID NumSlots Yes No Number of AFU slots available on an FPGA_DEVICE resource BBSID Yes No The FPGA Interface Manager (FIM) ID of an FPGA_DEVICE resource BBSVersion Yes No The FIM version of an FPGA_DEVICE resource VendorId Yes No The vendor ID of an FPGA_DEVICE resource GUID Yes Yes The GUID of an FPGA_DEVICE or FPGA_ACCELERATOR resource NumMMIO No Yes The number of MMIO space of an FPGA_ACCELERATOR resource NumInterrupts No Yes The number of interrupts of an FPGA_ACCELERATOR resource AcceleratorState No Yes The state of an FPGA_ACCELERATOR resource: either FPGA_ACCELERATOR_ASSIGNED or FPGA_ACCELERATOR_UNASSIGNED"},{"location":"sw/fpga_api/prog_guide/readme/#opae-c-api-return-codes","title":"OPAE C API Return Codes","text":"The OPAE C library returns a code for every exported public API function. FPGA_OK indicates successful completion of the requested operation. Any return code other than FPGA_OK indicates an error or unexpected behavior. When using the OPAE C API, always check the API return codes.
Error Code Description FPGA_OK Operation completed successfully FPGA_INVALID_PARAM Invalid parameter supplied FPGA_BUSY Resource is busy FPGA_EXCEPTION An exception occurred FPGA_NOT_FOUND A required resource was not found FPGA_NO_MEMORY Not enough memory to complete operation FPGA_NOT_SUPPORTED Requested operation is not supported FPGA_NO_DRIVER Driver is not loaded FPGA_NO_DAEMON FPGA Daemon (fpgad) is not running FPGA_NO_ACCESS Insufficient privileges or permissions FPGA_RECONF_ERROR Error while reconfiguring FPGA"},{"location":"sw/fpga_api/prog_guide/readme/#usage-models","title":"Usage Models","text":""},{"location":"sw/fpga_api/prog_guide/readme/#query-and-search-for-a-resource","title":"Query and Search for a Resource","text":"The user-code first populates an fpga_properties object with the required properties. Then, fpgaEnumerate() searches for matching resources. fpgaEnumerate() may return more than one matching resource.
#include \"fpga/fpga.h\"\n\nfpga_guid guid;\nfpga_properties filter = NULL;\nfpga_result res;\nfpga_token tokens[MAX_NUM_TOKENS];\nuint32_t num_matches = 0;\n\n/* Start with an empty properties object */\nres = fpgaGetProperties(NULL, &filter);\n\n/* Populate the properties object with required values.\n In this case, search for accelerators that matches \n the specified GUID.\n*/\nuuid_parse(GUID, guid);\nres = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nres = fpgaPropertiesSetGuid(filter, guid);\n\n/* Query the number of matching resources */\nres = fpgaEnumerate(&filter, 1, NULL, 1, &num_matches);\n\n/* Return tokens for all matching resources */\nres = fpgaEnumerate(&filter, 1, tokens, num_matches, &num_matches);\n\n/* Destroy the properties object */\nres = fpgaDestroyProperties(&filter);\n\n/* More code */\n......\n\n/* Destroy tokens */\nfor (uint32_t i = 0; i < num_matches; ++i) {\n res = fpgaDestroyToken(tokens[i]);\n}\n
The fpgaEnumerate() function can take multiple fpga_propertiesobjects in an array. In such cases, the function performs a logical OR of the properties object and returns resources that match any of the multiple properties. The fpga_token objects that fpgaEnumerate() returns, do not signify ownership. To acquire ownership of a resource represented by a token, pass the token to fpgaOpen().
"},{"location":"sw/fpga_api/prog_guide/readme/#acquire-and-release-a-resource","title":"Acquire and Release a Resource","text":"Use fpgaOpen() and fpgaClose() to acquire and release ownership of a resource. The calling process must own the resource before it can initiate MMIO, access share memory buffers, and use functions offered by the resource.
#include \"fpga/fpga.h\"\nfpga_handle handle;\nfpga_result res;\n/* Acquire ownership of a resource that \n `fpgaEnumerate()` previously returned as a token */\nres = fpgaOpen(token, &handle);\n/* More code */\n......\n/* Release the ownership */\nres = fpgaClose(handle);\n
"},{"location":"sw/fpga_api/prog_guide/readme/#shared-memory-buffer","title":"Shared Memory Buffer","text":"This code snippet shows how to prepare a memory buffer to be shared between the calling process and an accelerator.
#include \"fpga/fpga.h\"\nfpga_handle handle;\nfpga_result res;\n/* Hint for the virtual address of the buffer */\nvolatile uint64_t *addr_hint;\n/* An ID we can use to reference the buffer later */\nuint32_t bufid;\n/* Flag to indicate whether or not the buffer is preallocated */\nint flag = 0;\n/* Allocate (if necessary), pin, and map a buffer to be accessible\n by an accelerator\n */\nres = fpgaPrepareBuffer(handle, BUF_SIZE, (void **) &addr_hint,\n&bufid, flag);\n/* The actual address mapped to the buffer */\nuint64_t iova;\n/* Get the IO virtual address for the buffer */\nres = fpgaGetIOAddress(handle, bufid, &iova);\n/* Inform the accelerator about the virtual address by writing to its mapped\n register file\n */\n......\n/* More code */\n......\n/* Release the shared buffer */\nres = fpgaReleaseBuffer(handle, bufid);\n
.. note::
The `flag` variable can take a constant `FPGA_BUF_PREALLOCATED` to\nindicate that the calling process has already allocated the address space\nthat `addr_hint` points to.\n
"},{"location":"sw/fpga_api/prog_guide/readme/#mmio","title":"MMIO","text":"This code snippet shows how to map and unmap the register file of an accelerator into the calling process's virtual memory space.
#include \"fpga/fpga.h\"\nfpga_handle handle;\nfpga_result res;\n/* Index of the MMIO space. There might be multiple spaces on an accelerator */\nuint32_t mmio_num = 0;\n/* Mapped address */\nuint64_t mmio_addr;\n/* Map MMIO */\nres = fpgaMapMMIO(handle, mmio_num, &mmio_addr);\n/* Write to a 32-bit value to the mapped register file at a certain byte\n offset.\n CSR_CTL is the offset in the mapped space to where the value will be\n written. It's defined elsewhere.\n */\nres = fpgaWriteMMIO32(handle, mmio_num, CSR_CTL, value);\n/* More code */\n......\n/* Unmap MMIO */\nres = fpgaUnmapMMIO(handle, mmio_num);\n
.. Note::
Every AFU has its own register adress space and its own protocol to control operation through \nthe registers. \n
"},{"location":"sw/fpga_api/quick_start/readme/","title":"Quick Start Guide","text":""},{"location":"sw/fpga_api/quick_start/readme/#overview","title":"Overview","text":"The OPAE C library is a lightweight user-space library that provides an abstraction for FPGA resources in a compute environment. Built on top of the OPAE Intel\u00ae FPGA driver stack that supports Intel\u00ae FPGA platforms, the library abstracts away hardware-specific and OS-specific details and exposes the underlying FPGA resources as a set of features accessible from within software programs running on the host.
These features include the acceleration logic preconfigured on the device, as well as functions to manage and reconfigure the device. Hence, the library can enable user applications to transparently and seamlessly leverage FPGA-based acceleration.
In this document, we will explore the initial steps on how to set up the required libraries and utilities to use the FPGA devices.
If you do not have access to an Intel\u00ae Xeon\u00ae processor with integrated FPGA, or a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors, you will not be able to run the examples below. However, you can still make use of the AFU simulation environment (ASE) to develop and test accelerator RTL with OPAE applications.
The source for the OPAE SDK Linux device drivers is available at the OPAE Linux DFL drivers repository.
"},{"location":"sw/fpga_api/quick_start/readme/#build-the-opae-linux-device-drivers-from-the-source","title":"Build the OPAE Linux device drivers from the source","text":"For building the OPAE kernel and kernel driver, the kernel development environment is required. So before you build the kernel, you must install the required packages. Run the following commands (we are using Fedora 32 as an example):
sudo yum install gcc gcc-c++ make kernel-headers kernel-devel elfutils-libelf-devel ncurses-devel openssl-devel bison flex\n
Download the OPAE upstream kernel tree from GitHub:
git clone https://github.com/OPAE/linux-dfl.git -b fpga-upstream-dev-5.8.0\n
Configure the kernel:
cd linux-dfl\ncp /boot/config-`uname -r` .config\ncat configs/n3000_d5005_defconfig >> .config echo 'CONFIG_LOCALVERSION=\"-dfl\"' >> .config\necho 'CONFIG_LOCALVERSION_AUTO=y' >> .config\nmake olddefconfig\n
Compile and install the new kernel:
make -j\nsudo make modules_install\nsudo make install\n
After the installation finishes, reboot your system. Log back into the system, and confirm the correct version for the kernel:
$ uname -a\nLinux localhost.localdomain 5.8.0-rc1-dfl-g73e16386cda0 #6 SMP Wed Aug 19 08:38:32 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux\n
"},{"location":"sw/fpga_api/quick_start/readme/#building-and-installing-the-opae-sdk-from-the-source","title":"Building and installing the OPAE SDK from the source","text":"Download the OPAE SDK source package:
- Go to the section corresponding to the desired release on GitHub:
- Click the
Source code (tar.gz) link under the section's Assets. -
On the command line, go through the following steps to install it:
# Unpack\ntar xfvz opae-sdk-<release>.tar.gz\n# Configure\ncd opae-sdk-<release>\nmkdir build\ncd build\n# The default installation prefix is `/usr/local`;\n# You have the option to configure for a different location\ncmake [-DCMAKE_INSTALL_PREFIX=<prefix>] ..\n# Compile\nmake\n# Install: you need system administration privileges (`sudo`)\n# if you have elected to install in the default location\n[sudo] make install\n
The remainder of this guide assumes you installed into /usr/local.
"},{"location":"sw/fpga_api/quick_start/readme/#configuring-the-fpga-loading-an-fpga-afu","title":"Configuring the FPGA (loading an FPGA AFU)","text":"The fpgaconf tool exercises the AFU reconfiguration functionality. It shows how to read a bitstream from a disk file, check its validity and compatibility, and then injects it into FPGA to be configured as a new AFU, which can then be discovered and used by user applications.
For this step, you require a valid green bitstream (GBS) file. To reconfigure the FPGA slot, you can issue the following command as system administrator:
sudo fpgaconf -b 0x5e <filename>.gbs\n
The -b option to fpgaconf indicates the target bus number of the FPGA slot to be reconfigured. Alternatively, you can also specify the target socket number of the FPGA using the -s option.
$ sudo fpgaconf --help\nUsage:\n fpgaconf [-hvn] [-b <bus>] [-d <device>] [-f <function>] [-s <socket>] <gbs>\n\n-h,--help Print this help\n-v,--verbose Increase verbosity\n -n,--dry-run Don't actually perform actions\n -b,--bus Set target bus number\n -d,--device Set target device number\n -f,--function Set target function number\n -s,--socket Set target socket number\n
The sample application on the Building a Sample Application\nsection requires loading of an AFU called \"Native Loopback\nAdapter\" (NLB) on the FPGA. Please refer to the NLB documentation\nfor the location of the NLB's green bitstream. You also can verify\nif the NLB green bitstream has already been loaded into the FPGA\nslot by typing the following command and checking the output\nmatches the following:\n\n```bash\n$ cat /sys/class/fpga_region/region0/dfl-port.0/afu_id\nd8424dc4a4a3c413f89e433683f9040b\n```\n
The fpgaconf tool is not available for the Intel PAC N3000. The NLB is\nalready included in the AFU.\n
"},{"location":"sw/fpga_api/quick_start/readme/#building-a-sample-application","title":"Building a sample application","text":"The library source includes code samples. Use these samples to learn how to call functions in the library. Build and run these samples as quick sanity checks to determine if your installation and environment are set up properly.
In this guide, we will build hello_fpga.c. This is the \"Hello World!\" example of using the library. This code searches for a predefined and known AFU called \"Native Loopback Adapter\" on the FPGA. If found, it acquires ownership and then interacts with the AFU by sending it a 2MB message and waiting for the message to be echoed back. This code exercises all major components of the API except for AFU reconfiguration: AFU search, enumeration, access, MMIO, and memory management.
You can also find the source for hello_fpga in the samples directory of the OPAE SDK repository on GitHub.
int main(int argc, char *argv[])\n{\nfpga_properties filter = NULL;\nfpga_token afu_token;\nfpga_handle afu_handle;\nfpga_guid guid;\nuint32_t num_matches;\nvolatile uint64_t *dsm_ptr = NULL;\nvolatile uint64_t *status_ptr = NULL;\nvolatile uint64_t *input_ptr = NULL;\nvolatile uint64_t *output_ptr = NULL;\nuint64_t dsm_wsid;\nuint64_t input_wsid;\nuint64_t output_wsid;\nfpga_result res = FPGA_OK;\nif (uuid_parse(NLB0_AFUID, guid) < 0) {\nfprintf(stderr, \"Error parsing guid '%s'\\n\", NLB0_AFUID);\ngoto out_exit;\n}\n/* Look for accelerator by its \"afu_id\" */\nres = fpgaGetProperties(NULL, &filter);\nON_ERR_GOTO(res, out_exit, \"creating properties object\");\nres = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nON_ERR_GOTO(res, out_destroy_prop, \"setting object type\");\nres = fpgaPropertiesSetGuid(filter, guid);\nON_ERR_GOTO(res, out_destroy_prop, \"setting GUID\");\n/* TODO: Add selection via BDF / device ID */\nres = fpgaEnumerate(&filter, 1, &afu_token, 1, &num_matches);\nON_ERR_GOTO(res, out_destroy_prop, \"enumerating accelerators\");\nif (num_matches < 1) {\nfprintf(stderr, \"accelerator not found.\\n\");\nres = fpgaDestroyProperties(&filter);\nreturn FPGA_INVALID_PARAM;\n}\n/* Open accelerator and map MMIO */\nres = fpgaOpen(afu_token, &afu_handle, 0);\nON_ERR_GOTO(res, out_destroy_tok, \"opening accelerator\");\nres = fpgaMapMMIO(afu_handle, 0, NULL);\nON_ERR_GOTO(res, out_close, \"mapping MMIO space\");\n/* Allocate buffers */\nres = fpgaPrepareBuffer(afu_handle, LPBK1_DSM_SIZE,\n(void **)&dsm_ptr, &dsm_wsid, 0);\nON_ERR_GOTO(res, out_close, \"allocating DSM buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&input_ptr, &input_wsid, 0);\nON_ERR_GOTO(res, out_free_dsm, \"allocating input buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&output_ptr, &output_wsid, 0);\nON_ERR_GOTO(res, out_free_input, \"allocating output buffer\");\nprintf(\"Running Test\\n\");\n/* Initialize buffers */\nmemset((void *)dsm_ptr, 0, LPBK1_DSM_SIZE);\nmemset((void *)input_ptr, 0xAF, LPBK1_BUFFER_SIZE);\nmemset((void *)output_ptr, 0xBE, LPBK1_BUFFER_SIZE);\ncache_line *cl_ptr = (cache_line *)input_ptr;\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE / CL(1); ++i) {\ncl_ptr[i].uint[15] = i+1; /* set the last uint in every cacheline */\n}\n/* Reset accelerator */\nres = fpgaReset(afu_handle);\nON_ERR_GOTO(res, out_free_output, \"resetting accelerator\");\n/* Program DMA addresses */\nuint64_t iova;\nres = fpgaGetIOAddress(afu_handle, dsm_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting DSM IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_AFU_DSM_BASEL, iova);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_AFU_DSM_BASEL\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 0);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 1);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaGetIOAddress(afu_handle, input_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting input IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_SRC_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_SRC_ADDR\");\nres = fpgaGetIOAddress(afu_handle, output_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting output IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_DST_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_DST_ADDR\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_NUM_LINES, LPBK1_BUFFER_SIZE / CL(1));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_NUM_LINES\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CFG, 0x42000);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nstatus_ptr = dsm_ptr + DSM_STATUS_TEST_COMPLETE/8;\n/* Start the test */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 3);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Wait for test completion */\nwhile (0 == ((*status_ptr) & 0x1)) {\nusleep(100);\n}\n/* Stop the device */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 7);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Check output buffer contents */\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE; i++) {\nif (((uint8_t*)output_ptr)[i] != ((uint8_t*)input_ptr)[i]) {\nfprintf(stderr, \"Output does NOT match input \"\n\"at offset %i!\\n\", i);\nbreak;\n}\n}\nprintf(\"Done Running Test\\n\");\n/* Release buffers */\nout_free_output:\nres = fpgaReleaseBuffer(afu_handle, output_wsid);\nON_ERR_GOTO(res, out_free_input, \"releasing output buffer\");\nout_free_input:\nres = fpgaReleaseBuffer(afu_handle, input_wsid);\nON_ERR_GOTO(res, out_free_dsm, \"releasing input buffer\");\nout_free_dsm:\nres = fpgaReleaseBuffer(afu_handle, dsm_wsid);\nON_ERR_GOTO(res, out_unmap, \"releasing DSM buffer\");\n/* Unmap MMIO space */\nout_unmap:\nres = fpgaUnmapMMIO(afu_handle, 0);\nON_ERR_GOTO(res, out_close, \"unmapping MMIO space\");\n/* Release accelerator */\nout_close:\nres = fpgaClose(afu_handle);\nON_ERR_GOTO(res, out_destroy_tok, \"closing accelerator\");\n/* Destroy token */\nout_destroy_tok:\nres = fpgaDestroyToken(&afu_token);\nON_ERR_GOTO(res, out_destroy_prop, \"destroying token\");\n/* Destroy properties object */\nout_destroy_prop:\nres = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(res, out_exit, \"destroying properties object\");\nout_exit:\nreturn res;\n}\n
Linking with the OPAE library is straightforward. Code using this library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, the minimalist compile and link line should look like:
gcc -std=c99 hello_fpga.c -I/usr/local/include -L/usr/local/lib -lopae-c -luuid -ljson-c -lpthread -o hello_fpga\n
The API uses some features from the C99 language standard. The\n`-std=c99` switch is required if the compiler does not support C99 by\ndefault.\n
Third-party library dependency: The library internally uses\n`libuuid` and `libjson-c`. But they are not distributed as part of the\nlibrary. Make sure you have these libraries properly installed.\n
The layout of AFU is different between the N3000 card and Rush Creek/Darby Creek.\nIn the N3000 card, the NLB and DMA are contained in the AFU, so we need to do\nenumeration again in AFU to discover the NLB.\nTo run the hello_fpga application on the N3000 card, it should use the `-c`\noption to support the N3000 card:\n\n```bash\n$ sudo ./hello_fpga -c\nRunning Test\nRunning on bus 0x08.\nAFU NLB0 found @ 28000\nDone Running Test\n```\n
To run the hello_fpga application; just issue:
$ sudo ./hello_fpga\nRunning Test\nDone\n
"},{"location":"sw/fpga_api/quick_start/readme/#setup-iofs-release1-bitstream-on-fpga-pcie-card","title":"Setup IOFS Release1 Bitstream on FPGA PCIe card","text":"Program IOFS Release1 bitstream on the FPGA D5005 or N6000 cards and reboot the system.
Run this command:
$ lspci | grep acc\n3b:00.0 Processing accelerators: Intel Corporation Device af00 (rev 01)\n
Number of virtual functions supported by bitstream:
$ cat /sys/bus/pci/devices/0000:3b:00.0/sriov_totalvfs output: 3\n
Enable FPGA virtual functions:
sudo sh -c \"echo 3 > /sys/bus/pci/devices/0000:3b:00.0/sriov_numvfs\"\n
List of FPGA PF and VF's:
Physical Functions (PFs):\n 3b:00.0 Processing accelerators: Intel Corporation Device af00 (rev 01)\n\nVirtual Functions (VFs).\n 3b:00.1 Processing accelerators: Intel Corporation Device af01 (rev 01)\n 3b:00.2 Processing accelerators: Intel Corporation Device af01 (rev 01)\n 3b:00.3 Processing accelerators: Intel Corporation Device af01 (rev 01)\n
Bind vfio-pcie driver to FPGA virtual functions:
sudo opaevfio -i 0000:3b:00.1 -u userid -g groupid\nsudo opaevfio -i 0000:3b:00.2 -u userid -g groupid\nsudo opaevfio -i 0000:3b:00.3 -u userid -g groupid\n
List of fpga accelerators:
$ fpgainfo port\n\n//****** PORT ******//\n Object Id : 0x600D000000000000\n PCIe s:b:d.f : 0000:3b:00.3\n Device Id : 0xAF00\n Socket Id : 0xFF\n Accelerator Id : 43425ee6-92b2-4742-b03a-bd8d4a533812\n Accelerator GUID : 43425ee6-92b2-4742-b03a-bd8d4a533812\n //****** PORT ******//\n Object Id : 0x400D000000000000\n PCIe s:b:d.f : 0000:3b:00.2\n Device Id : 0xAF00\n Socket Id : 0xFF\n Accelerator Id : 8568AB4E-6bA5-4616-BB65-2A578330A8EB\n Accelerator GUID : 8568AB4E-6bA5-4616-BB65-2A578330A8EB\n //****** PORT ******//\n Object Id : 0x200D000000000000\n PCIe s:b:d.f : 0000:3b:00.1\n Device Id : 0xAF00\n Socket Id : 0xFF\n Accelerator Id : 56e203e9-864f-49a7-b94b-12284c31e02b\n Accelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n\nFPGA VF1/3b:00.1/Host Exerciser Loopback Accelerator GUID: 56E203E9-864F-49A7-B94B-12284C31E02B\nFPGA VF2/3b:00.2/Host Exerciser Memory Accelerator GUID: 8568AB4E-6bA5-4616-BB65-2A578330A8EB\nFPGA VF3/3b:00.3/Host Exerciser HSSI Accelerator GUID: 43425ee6-92b2-4742-b03a-bd8d4a533812\n
Unbind pcie-vfio dirver to FPGA virtual functions:
sudo opaevfio -r 0000:3b:00.1\n
Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA:
$ host_exerciser lpbk\n\n[lpbk] [info] starting test run, count of 1\nInput Config:0\n Allocate SRC Buffer\n Allocate DST Buffer\n Allocate DSM Buffer\n Start Test\n Test Completed\n Host Exerciser swtest msg:0\n Host Exerciser numReads:32\n Host Exerciser numWrites:32\n Host Exerciser numPendReads:0\n Host Exerciser numPendWrites:0\n [lpbk] [info] Test lpbk(1): PASS\n
In order to successfully run hello\\_fpga, the user needs to configure\n system hugepage to reserve 2M-hugepages.\n For example, the command below reserves 20 2M-hugepages:\n\n ```bash\n echo 20 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages\n ```\n\n For x86_64 architecture CPU, user can use the following command to find out available huge page sizes:\n\n ```bash\n $ grep pse /proc/cpuinfo | uniq\n flags : ... pse ...\n ```\n\n If this command returns a non-empty string, 2MB pages are supported:\n\n ```bash\n $ grep pse /proc/cpuinfo | uniq\n flags : ... pdpe1gb ...\n ```\n\n If this commands returns a non-empty string, 1GB pages are supported.\n ````\n\n````{note}\nThe default configuration for many Linux distributions currently sets a\nrelatively low limit for pinned memory allocations per process \n(RLIMIT_MEMLOCK, often set to a default of 64kiB).\n\nTo run an OPAE application that attempts to share more memory than specified\nby this limit between software and an accelerator, you can either:\n\n* Run the application as root, or\n* Increase the limit for locked memory via `ulimit`:\n\n```bash\nulimit -l unlimited\n```\n\nSee the Installation Guide for how to permanently adjust the memlock limit.\n
"},{"location":"sw/fpga_dfl_drv/fpga_dfl_drv/","title":"Enable OPAE on FPGA PCIe drivers","text":".. toctree::\n.. highlight:: c\n.. highlight:: console\n
FPGA PCIe driver for PCIe-based Field-Programmable Gate Array (FPGA) solutions which implement the Device Feature List (DFL). This driver provides interfaces for user space applications to configure, enumerate, open and access FPGA accelerators on the FPGA DFL devices. additionally, it also enables system level management functions such as FPGA partial reconfiguration, power management, virtualization with DFL framework and DFL feature device drivers.
OPAE 1.4.0 release supports both FPGA Intel Linux driver as well as Linux FPGA DFL driver patch set2. Linux PCIe FPGA DFL driver supports Intel FPGA devices.
FPGA DFL Linux driver source code patchset2 available https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers?h=linux-5.4.y
FPGA DFL Linux driver source code patchset1 available https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/fpga?h=v4.19.14
"},{"location":"sw/fpga_tools/readme/","title":"fpga_tools","text":""},{"location":"sw/fpga_tools/readme/#fpgainfo","title":"fpgainfo","text":""},{"location":"sw/fpga_tools/readme/#name","title":"NAME","text":"fpgainfo - FPGA information tool
"},{"location":"sw/fpga_tools/readme/#synopsis","title":"SYNOPSIS","text":"fpgainfo [-h | --help] [-s | --socket-id] <command> [<args>]\n
"},{"location":"sw/fpga_tools/readme/#description","title":"DESCRIPTION","text":"fpgainfo is a tool to show FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp and is used to specify what type of information to report. Some commands may also have other arguments/options that can be used to control the behavior of that command.
"},{"location":"sw/fpga_tools/readme/#common-options","title":"COMMON OPTIONS","text":"--help, -h
Print help information and exit.\n
--socket-id, -s
Socket ID encoded in BBS. Default=0\n
"},{"location":"sw/fpga_tools/readme/#fpgainfo-commands","title":"FPGAINFO COMMANDS","text":"errors
Show/clear errors of an FPGA resource as specified by the first argument.\nError information is parsed to display in human readable form.\n
power
Show total power consumed by the FPGA hardware in watts\n
temp
Show FPGA temperature values in degrees Farenheit\n
"},{"location":"sw/fpga_tools/readme/#errors-options","title":"ERRORS OPTIONS","text":"--clear, -c
Clear errors for the given FPGA resource\n
"},{"location":"sw/fpga_tools/readme/#errors-arguments","title":"ERRORS ARGUMENTS","text":"The first argument to the errors command is used to specify what kind of resource to act on. It must be one of the following: fme,port,first_error,pcie0,pcie1,bbs,gbs,all More details on the errors reported for the resource can be found below:
"},{"location":"sw/fpga_tools/readme/#errors-resources","title":"ERRORS RESOURCES","text":"fme
Show/clear errors pertaining to the FME\n
port
Show/clear errors pertaining to the PORT\n
first_error
Show/clear first errors encountered by the FPGA\n
pcie0
Show/clear errors pertaining to the PCIE0 lane\n
pcie1
Show/clear errors pertaining to the PCIE1 lane\n
bbs
Show/clear errors pertaining to the BBS (blue bitstream)\n
gbs
Show/clear errors pertaining to the GBS (green bitstream)\n
all
Show/clear errors for all resources\n
"},{"location":"sw/fpga_tools/readme/#examples","title":"EXAMPLES","text":"This command shows the current power consumtion:
./fpgainfo power\n
This command shows the current temperature reading:
./fpgainfo temp\n
This command shows the errors for the FME resource:
./fpgainfo errors fme\n
This command clears all the errors on all resources: ./fpgainfo errors all -c\n
"},{"location":"sw/fpga_tools/readme/#fpgaconf","title":"fpgaconf","text":""},{"location":"sw/fpga_tools/readme/#name_1","title":"NAME","text":"fpgadiag - Configure a green bitstream to an FPGA
"},{"location":"sw/fpga_tools/readme/#synopsis_1","title":"SYNOPSIS","text":"fpgaconf [-hvn] [-b <bus>] [-d <device>] [-f <function>] [-s <socket>] <gbs>
"},{"location":"sw/fpga_tools/readme/#description_1","title":"DESCRIPTION","text":"fpgaconf writes accelerator configuration bitstreams (also referred to as \"green bitstreams\" to an FPGA device recognized by OPAE. In the process, it also checks the green bitstream file for compatibility with the targeted FPGA and its current infrastructure bitstream (the \"blue bistream\"). fpgaconf takes the following arguments:
-h, --help
Print usage information\n
-v, --verbose
Print more verbose messages while enumerating and configuring. Can be\ngiven more than once\n
-n, --dry-run
Perform enumeration, but skip any operations with side-effects (like the\nactual configuration of the bitstream\n
-b, --bus
PCI bus number of the FPGA to target\n
-d, --device
PCI device number of the FPGA to target\n
-f, --function
PCI function number of the FPGA to target\n
-s, --socket
Socket number of the FPGA to target\n
fpgaconf will enumerate available FPGA devices in the system and select compatible FPGAs for configuration. If there are more than one candidate FPGAs that are compatible with the given green bitstream, fpgaconf will exit and ask you to be more specific in selecting the target FPGAs (e.g. by specifying a socket number, or a PCIe bus/device/function).
"},{"location":"sw/fpga_tools/readme/#examples_1","title":"EXAMPLES","text":"fpgaconf my_green_bitstream.gbs
Program \"my_green_bitstream.gbs\" to a compatible FPGA\n
fpgaconf -v -s 0 my_green_bitstream.gbs
Program \"my_green_bitstream.gbs\" to the FPGA in socket 0, if compatible,\nwhile printing out slightly more verbose information\n
"},{"location":"sw/fpga_tools/readme/#fpgad","title":"fpgad","text":""},{"location":"sw/fpga_tools/readme/#name_2","title":"NAME","text":"fpgad - log errors and generate events
"},{"location":"sw/fpga_tools/readme/#synopsis_2","title":"SYNOPSIS","text":"fpgad --daemon [--directory=<dir>] [--logfile=<file>] [--pidfile=<file>] [--umask=<mode>] [--socket=<sock>] [--null-bitstream=<file>] fpgad [--socket=<sock>] [--null-bitstream=<file>]
"},{"location":"sw/fpga_tools/readme/#description_2","title":"DESCRIPTION","text":"Periodically monitors/reports the error status reflected in the device driver's error status sysfs files. Establishes the channel by which events are communicated to the OPAE application. Programs a NULL bitstream in response to AP6 event.
fpgad is required to be running before API calls fpgaRegisterEvent and fpgaUnregisterEvent will succeed.
Use SIGINT to stop fpgad.
-d, --daemon
When given, fpgad executes as a system demon process.\n
-D, --directory <dir>
When running in daemon mode, execute from the given directory.\nIf omitted when daemonizing, /tmp is used.\n
-l, --logfile <file>
When running in daemon mode, send output to file. When not in daemon mode, the output is sent to stdout.\nIf omitted when daemonizaing, /tmp/fpgad.log is used.\n
-p, --pidfile <file>
When running in daemon mode, write the daemon's process id to file.\nIf omitted when daemonizing, /tmp/fpgad.pid is used.\n
-m, --umask <mode>
When running in daemon mode, use the mode value as the file mode creation mask passed to umask.\nIf omitted when daemonizing, 0 is used.\n
-s, --socket <sock>
Listen for event API registration requests on sock. The default socket value used by the API is\n/tmp/fpga_event_socket.\n
-n, --null-bitstream <file>
Specify the NULL bitstream to program when an AP6 event occurs. This option may be given multiple\ntimes. The bitstream, if any, that matches the FPGA's PR interface id will be programmed when AP6\nis detected.\n
"},{"location":"sw/fpga_tools/readme/#troubleshooting","title":"TROUBLESHOOTING","text":"If any issues are encountered, try the following for additional debug information:
- Examine the log file when in daemon mode.
- Run in non-daemon mode and view stdout.
"},{"location":"sw/fpga_tools/readme/#examples_2","title":"EXAMPLES","text":"fpgad --daemon --null-bitstream=my_null_bits.gbs
"},{"location":"sw/fpga_tools/readme/#see-also","title":"SEE ALSO","text":"umask
"},{"location":"sw/fpga_tools/readme/#fpgadiag","title":"fpgadiag","text":""},{"location":"sw/fpga_tools/readme/#name_3","title":"NAME","text":"fpgadiag - FPGA diagnosis and testing tool.
"},{"location":"sw/fpga_tools/readme/#synopsis_3","title":"SYNOPSIS","text":"fpgadiag [-m | --mode=] <mode> [-t | --target=] <target> [options]\n
"},{"location":"sw/fpga_tools/readme/#description_3","title":"DESCRIPTION","text":"fpgadiag includes several tests to diagnose, test and report on the FPGA hardware.
<mode> chooses which test to run. <target> specifies on what platform to run the test. <target> can be either fpga or ase, where ase stands for \"AFU Simulation Environment\".
The tests that can be selected by <mode> include:
lpbk1
The test performs loopback test on the number of cachelines specified with \nthe `BEGIN` option. _fpgadiag_ sets up source and destination buffers in \nmain memory. The FPGA then performs a memcpy from a source buffer to the \ndestination buffer, one cacheline at a time.\n\nA cacheline is 64 bytes. When `BEGIN = END`, you perform one iteration. When \n`BEGIN = END + x`, you perform `x` iterations. The first iteration consists \nof copying `BEGIN` cachelines; the second iteration consists of copying \n`BEGIN+1` cache lines; the third iteration consists of copying `BEGIN+3` \ncache lines, etc.\n\nThe latency is shown as the number of clock ticks.\n\nWhen you specify `MULTI-CL`, you copy `MULTI-CL` cache lines at a time.\nThere is always a WrFence. `WR-FENCE` chooses what virtual channel the \nWrFence occurs on.\n\nIf you specify continuous mode with `--cont`, the program runs an iteration\nuntil the timeout specified in `TIMEOUT` completes.\n
read
This test performs only a read, not a memcpy. It is used to measure read \nbandwidth.\n
write
This test is used to measure write bandwidth.\n
trput
This test measures both read and write bandwidth by performing 50% read and \n50% write tests.\n
sw
This is a send-and-respond (ping-pong) test where one side sends data and \nwaits for answer.\n
Each test requires presence of one of these bitstreams, as documented below. Before running a test, make sure its required bitstream is properly configured on the platform.
- nlb mode 0 for the
lpbk1 test. - nlb mode 3 for the
trput, read, and write tests. - nlb mode 7 for the
sw test.
"},{"location":"sw/fpga_tools/readme/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/readme/#common-options_1","title":"Common options","text":"--help, -h
Print help information and exit.\n
--target=, -t
Values accepted for this switch are fpga or ase. Default=fpga\n
--mode=, -m
The test to run. Values accepted for this switch are `lpbk1`, `read`,\n`write`, `trput`, `sw`\n
--config=, -c
A configuration file in the JSON format that specifies options for a test.\nIf an option is specified both in the configuration file and on the command \nline, the value in the configuration file prevails\n
--socket-id=, -s
Socket ID encoded in BBS. Default=0\n
--bus-number=, -B
Bus number of the PCIe device. Default=0\n
--device=, -D
Device number of the PCIe device. Default=0\n
--function=, -F
Function number of the PCIe device. Default=0\n
--freq=, -T
Clock frequency in Hz. Default=400 MHz\n
--suppress-hdr, -S
Suppress column headers for text output. Default=off\n
--csv, -V
Comma separated value format. Default=off\n
"},{"location":"sw/fpga_tools/readme/#lpbk1-test-options","title":"lpbk1 test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=D8424DC4-A4A3-C413-F89E-433683F9040B\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -U
M can equal 1, 2, or 4. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I Default=wrline-M\n
--cache-hint=, -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. Default=auto\n
"},{"location":"sw/fpga_tools/readme/#read-test-options","title":"read test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-hint=, -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Attempt to prime the cache with hits. Default=off, Attempt to prime the \ncache with misses. Default=off\n
--cool-cpu-cache, -C
Attempt to prime the cpu cache with misses. Default=off\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. Default=auto\n
"},{"location":"sw/fpga_tools/readme/#write-test-options","title":"write test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I Default=wrline-M\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Attempt to prime the cache with hits. Default=off, Attempt to prime the \ncache with misses. Default=off\n
--cool-cpu-cache, -C
Attempt to prime the cpu cache with misses. Default=off\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1, random. Default=`WRITE-VC`\n
--alt-wr-pattern, -l
Alternate Write Pattern. Default=off\n
"},{"location":"sw/fpga_tools/readme/#trput-test-options","title":"trput test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I Default=wrline-M\n
--cache-hint=, -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. Default=`WRITE-VC`\n
"},{"location":"sw/fpga_tools/readme/#sw-test-options","title":"sw test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=7BAF4DEA-A57C-E91E-168A-455D9BDA88A3\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I. Default=wrline-M\n
--cache-hint= -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random Default=auto\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. Default=`WRITE-VC`\n
--notice=, -N
Can be poll, csr-write, umsg-data, or umsg-hint. Default=poll\n
"},{"location":"sw/fpga_tools/readme/#examples_3","title":"EXAMPLES","text":"This command starts an lpbk1 test on the FPGA on bus 0x5e. The test copies 57535, 57536, 57537, ..., up to 65535 cache lines, one line at a time. The test output is printed in the CSV format with header suppressed.
./fpgadiag --mode=lpbk1 --target=fpga -SV --bus-number=0x5e --begin=57535\n--end=65535 --cache-hint=rdline-I --cache-policy=wrpush-I --multi-cl=1\n--write-vc=vl0 --read-vc=vh1 --wrfence-vc=auto\n
This command starts a read test on the FPGA located on bus 0xbe. The test reads 2045 cache lines in the continuous mode with a 15-second timeout period. Data is accessed with a strided pattern with a 10-byte stride length.
./fpgadiag --mode=read --target=fpga -SV --bus-number=0xbe --begin=2045 --cont\n--timeout-sec=15 --cache-hint=rdline-I --multi-cl=1 -a=10 --write-vc=vh1\n--read-vc=auto --wrfence-vc=auto\n
This command starts an sw test on the FPGA located on bus 0xbe. The test notifies completion using a CSR write.
./fpgadiag --mode=sw --target=fpga -SV --bus-number=0xbe --begin=4 --end=8192\n--cache-hint=rdline-I --cache-policy=wrline-I --notice=csr-write --write-vc=vl0\n--wrfence-vc=auto --read-vc=random \n
"},{"location":"sw/fpga_tools/readme/#troubleshooting_1","title":"TROUBLESHOOTING","text":"When a test fails to run or gives errors, check the following:
- Is Intel FPGA driver properly installed? See Installation Guide for driver installation instructions.
- Are FPGA port permissions set properly? Check the permission bits of the port, for example,
/dev/intel-fpga-port-0. Users need READ and WRITE permissions to run fpgadiag tests. - Is hugepage properly configured on the system? See Installation Guide for hugepage configuration steps.
- Is the required bitstream loaded? See DESCRIPTION for information about what bitstream is required by which test.
- Are
--begin and --end values set properly? --end must be no smaller than the --begin. Also, --begin must be a multiple of the --multi-cl value. - The
--warm-fpga-cache and --cool-fpga-cache options in the read and write tests are mutually exclusive. - The timeout options are only meaningful for the continuous mode (with the
--cont option).
"},{"location":"sw/fpga_tools/readme/#mmlink","title":"mmlink","text":""},{"location":"sw/fpga_tools/readme/#name_4","title":"NAME","text":"MMLink - Debugging RTL.
"},{"location":"sw/fpga_tools/readme/#synopsis_4","title":"SYNOPSIS","text":"mmlink [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-P <TCP port>] [-I <IP Address>]
"},{"location":"sw/fpga_tools/readme/#description_4","title":"DESCRIPTION","text":"Remote signaltap is software tool used for debug RTL (AFU), effectively a signal trace capability that Quartus places into a green bitstream. Remote Signal Tap provides access the RST part of the Port MMIO space, and then runs the remote protocol on top.
"},{"location":"sw/fpga_tools/readme/#examples_4","title":"EXAMPLES","text":"./mmlink -B 0x5e -P 3333
MMLink app starts and listens for connection.
"},{"location":"sw/fpga_tools/readme/#options_1","title":"OPTIONS","text":"-B,--bus FPGA Bus number.
-D,--device FPGA Device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-P,--port TCP port number.
-I,--ip IP address of FPGA system.
"},{"location":"sw/fpga_tools/readme/#notes","title":"NOTES","text":"Driver privilege:
Change AFU driver privilege to user .
command: chmod 777 /dev/intel-fpga-port.0
set memlock:
command: ulimit -l 10000
"},{"location":"sw/fpga_tools/readme/#coreidle","title":"coreidle","text":""},{"location":"sw/fpga_tools/readme/#name_5","title":"NAME","text":"coreidle - idles cores for shared TDP sockets to run online cores at maximum capacity.
"},{"location":"sw/fpga_tools/readme/#synopsis_5","title":"SYNOPSIS","text":"coreidle [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-G <GBS path>]
"},{"location":"sw/fpga_tools/readme/#description_5","title":"DESCRIPTION","text":"This tools parses input GBS, extracts power from metadata ,calculates fpga power, number of online and idle cores. It moves threads from idle cores to online cores.
"},{"location":"sw/fpga_tools/readme/#examples_5","title":"EXAMPLES","text":"./coreidle -B 0x5e -G /home/lab/gbs/mode0.gbs
Idle cores to run online cores at maximum capacity.
"},{"location":"sw/fpga_tools/readme/#options_2","title":"OPTIONS","text":"-B,--bus FPGA Bus number.
-D,--device FPGA Device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-G,--gbs Green bitstream file path.
"},{"location":"sw/fpga_tools/readme/#fpgamux","title":"fpgamux","text":""},{"location":"sw/fpga_tools/readme/#name_6","title":"NAME","text":"fpgamux - Software MUX for running multiple AFU (accelerator functional unit) tests in one GBS (green bitsream)\n
"},{"location":"sw/fpga_tools/readme/#synopsis_6","title":"SYNOPSIS","text":"fpgamux [-h] [-S|--socket-id SOCKET_ID] [-B|--bus-number BUS] [-D|--device DEVICE] [-F|--function FUNCTION]\n [-G|--guid GUID] -m|--muxfile MUXFILE.json\n
"},{"location":"sw/fpga_tools/readme/#description_6","title":"DESCRIPTION","text":"fpgamux is a testing tool to interact with multiple AFUs that have been synthesized into one GBS along with the CCIP-MUX BBB (basic building block). The CCIP-MUX uses upper bits in the MMIO addresses to route MMIO reads/writes to the AFU running on the corresponding CCIP-MUX port. fpgamux uses a configuration file that lists the software components and configuration to use.
.. note::
Only one (the first) AFU is discoverable by the OPAE driver. Enumerating acceleration on an FPGA will find\n the accelerator associated with the first AFU only. The first software component in the configuration will\n be used to determine the GUID to use for enumeration. This can be overridden with the -G|--guid option.\n
"},{"location":"sw/fpga_tools/readme/#options_3","title":"OPTIONS","text":"-S SOCKET_ID, --socket-id SOCKET_ID\n socket id of FPGA resource\n\n-B BUS, --bus BUS\n bus id of FPGA resource\n\n-D DEVICE, --device DEVICE\n device id of FPGA resource\n\n\n-F FUNCTION, --function FUNCTION\n function id of FPGA resource\n\n-G, --guid\n specify what guid to use for the accelerator enumeration\n
"},{"location":"sw/fpga_tools/readme/#configuration","title":"CONFIGURATION","text":"fpgamux uses a configuration file (in JSON format) to determine what software components to instantiate and how to configure them for interacting with the AFUs in the GBS. This schema for this is listed below:
[\n {\n \"app\" : \"fpga_app\",\n \"name\" : \"String\",\n \"config\" : \"Object\"\n }\n]\n
"},{"location":"sw/fpga_tools/readme/#examples_6","title":"EXAMPLES","text":"An example configuration with two components is listed below:
[\n {\n \"app\" : \"nlb0\",\n \"name\" : \"nlb0\",\n \"config\" :\n {\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"cont\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n },\n {\n \"app\" : \"nlb3\",\n \"name\" : \"nlb3\",\n \"config\" :\n {\n \"mode\" : \"read\",\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"strided-access\" : 1,\n \"cont\" : false,\n \"warm-fpga-cache\" : false,\n \"cool-fpga-cache\" : false,\n \"cool-cpu-cache\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"alt-wr-pattern\" : false,\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n }\n]\n
"},{"location":"sw/fpga_tools/readme/#userclk","title":"userclk","text":""},{"location":"sw/fpga_tools/readme/#name_7","title":"NAME","text":"userclk - to set afu high and low clock frequency.
"},{"location":"sw/fpga_tools/readme/#synopsis_7","title":"SYNOPSIS","text":"userclk [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-P <Port id>] [-H <User clock high frequency>] -L <User clock low frequency>]
"},{"location":"sw/fpga_tools/readme/#description_7","title":"DESCRIPTION","text":"userclk tool used to set high and low clock frequency to acceleration function unit.
"},{"location":"sw/fpga_tools/readme/#examples_7","title":"EXAMPLES","text":"./userclk -B 0x5e -H 400 -L 200
Sets AFU frequency.
"},{"location":"sw/fpga_tools/readme/#options_4","title":"OPTIONS","text":"-B,--bus FPGA Bus number.
-D,--device FPGA Device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-P,--port Port id.
-H,--freq-high User clock high frequency.
-L,--freq-low User clock low frequency.
"},{"location":"sw/fpga_tools/coreidle/coreidle/","title":"coreidle","text":""},{"location":"sw/fpga_tools/coreidle/coreidle/#synopsis","title":"SYNOPSIS","text":"coreidle [-v] [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-G <GBS path>]
"},{"location":"sw/fpga_tools/coreidle/coreidle/#description","title":"DESCRIPTION","text":"coreidle parses the Accelerator Function Unit (AFU) metadata and extracts power information. coreidle calculates the FPGA power and calculates the number of online and idle cores. It moves threads from idle cores to online cores. coreidle is only available the Integrated FPGA Platform. You cannot run coreidle on the PCIe Accelerator Card (PAC).
"},{"location":"sw/fpga_tools/coreidle/coreidle/#examples","title":"EXAMPLES","text":"./coreidle -B 0x5e -G /home/lab/gbs/mode0.gbs
Idle cores to run online cores at maximum capacity.
"},{"location":"sw/fpga_tools/coreidle/coreidle/#options","title":"OPTIONS","text":"-v,--version Prints version information and exit.
-B,--bus FPGA bus number.
-D,--device FPGA device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-G,--gbs Green bitstream file path.
"},{"location":"sw/fpga_tools/fecmode/fecmode/","title":"fecmode (N3000 specific tool)","text":""},{"location":"sw/fpga_tools/fecmode/fecmode/#synopsis","title":"SYNOPSIS","text":"fecmode [<mode>][<args>]\n
"},{"location":"sw/fpga_tools/fecmode/fecmode/#description","title":"DESCRIPTION","text":"Fecmode changes FEC mode of external ethernet PHY, this tool only support on N3000 Card.
"},{"location":"sw/fpga_tools/fecmode/fecmode/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"--segment, -S
segment number of the PCIe device.
--bus, -B
bus number of the PCIe device.
--device, -D
device number of the PCIe device.
--function, -F function number of the PCIe device
--rsu, -r reboot card only if mode is not configured
--debug, -d output debug information
"},{"location":"sw/fpga_tools/fecmode/fecmode/#fec-mode","title":"FEC Mode","text":"no no FEC.
kr BaseR FEC (Fire-Code) correction \u2013 4 orders
rs Reed-Solomon FEC correction \u2013 7 orders
"},{"location":"sw/fpga_tools/fecmode/fecmode/#example","title":"EXAMPLE","text":"This command change FEC mode to \u201ckr\u201d:
# fecmode -B 0x25 kr\n
This command reboot card (no need to specify bus number if there is only one card):
# fecmode -r\n
This command display the current FEC mode:
# fecmode\n
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/","title":"fpgabist","text":""},{"location":"sw/fpga_tools/fpgabist/fpgabist/#synopsis","title":"SYNOPSIS","text":"fpgabist [-h] [-i device_id] [-b bus] [-d device] [-f function] [path_to_gbs1 path_to_gbs2 ...]\n
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#description","title":"DESCRIPTION","text":"The fpgabist tool performs self-diagnostic tests on supported FPGA platforms.
The tool accepts one or more Accelerator Function (AF) binaries from a predetermined set of AFs. Depending on the available binaries, the tool runs appropriate tests and reports hardware issues.
fpgabist always uses fpgainfo to report system information before running any hardware tests.
Currently, fpgabist accepts the following AFs: 1. nlb_mode_3: The native loopback (NLB) test implements a loopback from TX to RX. Use it to verify basic functionality and to measure bandwidth. 2. dma_afu: The direct memory access (DMA) AFU test transfers data from host memory to FPGA-attached local memory.
The installation includes the AF files, but you can also compile the AFs from the source.
If there are multiple PCIe\u00ae devices, use -b, -d, -f to specify the BDF for the specific PCIe\u00ae device.
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"[path_to_gbs1 path_to_gbs2 ...]
Paths to Accelerator Function (AF) files.
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"You can use the single letter or the full parameter name for the command line arguments.
-h, --help
Prints usage information
-i device_id, --device-id device_id
Device ID for Intel FPGA. Default is: 0x09c4
-B bus, --bus bus
Bus number for specific FPGA
-D device, --device device
Device number for specific FPGA
-F function, --function function
Function number for specific FPGA
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#examples","title":"EXAMPLES","text":"fpgabist <path_to_gbs_files>/dma_afu.gbs <path_to_gbs_files>/nlb_3.gbs
Runs fpgabist on any platform in the system that matches the default device ID. This command runs both the DMA and NLB_MODE_3 tests.
fpgabist -i 09c4 -b 5 <path to gbs>/dma_afu.gbs
Runs fpgabist the DMA test on the PCIe\u00ae Endpoint with device_id 09c4 on bus 5.
"},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/","title":"fpgaconf","text":""},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/#synopsis","title":"SYNOPSIS","text":"fpgaconf [-hvVn] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR] <gbs>
"},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/#description","title":"DESCRIPTION","text":"fpgaconf configures the FPGA with the accelerator function (AF). It also checks the AF for compatibility with the targeted FPGA and the FPGA Interface Manager (FIM). fpgaconf takes the following arguments:
-h, --help
Prints usage information.\n
-v, --version
Prints version information and exits.\n
-V, --verbose
Prints more verbose messages while enumerating and configuring. Can be\nrequested more than once.\n
-n, --dry-run
Performs enumeration. Skips any operations with side-effects such as the\nactual AF configuration.\n
-S, --segment
PCIe segment number of the target FPGA.\n
-B, --bus
PCIe bus number of the target FPGA.\n
-D, --device
PCIe device number of the target FPGA.\n
-F, --function
PCIe function number of the target FPGA.\n
--force
Reconfigure the AFU even if it is in use.\n
fpgaconf enumerates available FPGA devices in the system and selects compatible FPGAs for configuration. If more than one FPGA is compatible with the AF, fpgaconf exits and asks you to be more specific in selecting the target FPGAs by specifying a a PCIe BDF.
"},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/#examples","title":"EXAMPLES","text":"fpgaconf my_af.gbs
Program \"my_af.gbs\" to a compatible FPGA.\n
fpgaconf -V -B 0x3b my_af.gbs
Program \"my_af.gbs\" to the FPGA in bus 0x3b, if compatible,\nwhile printing out slightly more verbose information.\n
fpgaconf 0000:3b:00.0 my_af.gbs
Program \"my_af.gbs\" to the FPGA at address 0000:3b:00.0.\n
"},{"location":"sw/fpga_tools/fpgad/fpgad/","title":"fpgad","text":""},{"location":"sw/fpga_tools/fpgad/fpgad/#synopsis","title":"SYNOPSIS","text":"fpgad --daemon [--version] [--directory=<dir>] [--logfile=<file>] [--pidfile=<file>] [--umask=<mode>] [--socket=<sock>] [--null-bitstream=<file>] fpgad [--socket=<sock>] [--null-bitstream=<file>]
"},{"location":"sw/fpga_tools/fpgad/fpgad/#description","title":"DESCRIPTION","text":"fpgad monitors the device sensors, checking for sensor values that are out of the prescribed range.
When any of the sensors is detected to be out of bounds, fpgad will focus on keeping the server from rebooting by masking PCIE AER, and send a message to system administrator. System administrator can take further actions like stop the application and stop the FPGA, but fpgad just focus on monitor the sensors and will not take any cooling actions.
Note: fpgad must be running (as root) and actively monitoring devices when a sensor anomaly occurs in order to initiate Graceful Shutdown. If fpgad is not loaded during such a sensor anomaly, the out-of-bounds scenario will not be detected, and the resulting effect on the hardware is undefined.
"},{"location":"sw/fpga_tools/fpgad/fpgad/#arguments","title":"ARGUMENTS","text":"-v, --version
Prints version information and exits.\n
-d, --daemon
When specified, fpgad executes as a system daemon process.\n
-D, --directory <dir>
When running in daemon mode, run from the specified directory.\nIf omitted when daemonizing, `fpgad` uses /tmp.\n
-l, --logfile <file>
When running in daemon mode, send output to file. When not in daemon mode, the output goes to stdout.\nIf omitted when daemonizaing, fpgad uses /tmp/fpgad.log.\n
-p, --pidfile <file>
When running in daemon mode, write the daemon's process id to a file.\nIf omitted when daemonizing, fpgad uses /tmp/fpgad.pid.\n
-m, --umask <mode>
When running in daemon mode, use the mode value as the file mode creation mask passed to umask.\nIf omitted when daemonizing, fpgad uses 0.\n
-s, --socket <sock>
Listen for event API registration requests on the UNIX domain socket on the specified path. \nThe default=/tmp/fpga_event_socket.\n
-n, --null-bitstream <file>
Specify the NULL bitstream to program when an AP6 event occurs. This option may be specified multiple\ntimes. The AF, if any, that matches the FPGA's PR interface ID is programmed when an AP6\nevent occurs.\n
"},{"location":"sw/fpga_tools/fpgad/fpgad/#troubleshooting","title":"TROUBLESHOOTING","text":"If you encounter any issues, you can get debug information in two ways:
- By examining the log file when in daemon mode.
- By running in non-daemon mode and viewing stdout.
"},{"location":"sw/fpga_tools/fpgad/fpgad/#examples","title":"EXAMPLES","text":"fpgad --daemon --null-bitstream=my_null_bits.gbs
This command starts fpgad as a system daemon process:
sudo systemctl start fpgad\n
"},{"location":"sw/fpga_tools/fpgadiag/","title":"fpgadiag","text":""},{"location":"sw/fpga_tools/fpgadiag/#synopsis","title":"SYNOPSIS","text":"fpgadiag [-m | --mode=] <mode> [-t | --target=] <target> [options]\n
"},{"location":"sw/fpga_tools/fpgadiag/#description","title":"DESCRIPTION","text":"Includes several tests to diagnose, test, and report on the FPGA hardware.
<mode> chooses which test to run. <target> specifies the platform that runs the test. <target> can be either fpga or ase where ase. <ase> is the abbreviation for Accelerator Simulation Environment.
The <mode> selects from the following tests:
lpbk1
This test runs a loopback test on the number of cachelines specified with the BEGIN option. fpgadiag sets up source and destination buffers in main memory. The FPGA then performs a memcpy from a source buffer to the destination buffer, one cacheline at a time.
A cacheline is 64 bytes. When BEGIN = END, the test performs one iteration. When BEGIN = END + x, the test performs x iterations. The first iteration consists of copying BEGIN cachelines; the second iteration consists of copying BEGIN+1 cache lines. The third iteration consists of copying BEGIN+2 cache lines, and so on.
The latency is shown as the number of clock cycles.
When you specify MULTI-CL, you copy MULTI-CL cache lines at a time. The WR-FENCE chooses on which virtual channel the WrFence occurs.
If you specify continuous mode with --cont, the program iterates until the timeout specified in TIMEOUT completes.
read
This test performs reads. Use this test to measure read bandwidth.
write
This test performs writes. Use it to measure write bandwidth.
trput
This test measures both read and write bandwidth by performing 50% read and 50% write tests.
sw
This is a send-and-respond (ping-pong) test. One side sends data and waits for response.
Each test requires a particular AF. Before running a test, make sure the required AF is properly configured on the platform.
- The lpbk1 test requires the nlb mode 0 AF.
- The trput test requires the nlb mode 3 AF.
- The sw test requires the nlb mode 7 AF. This AF is only available for the integrated FPGA platform. You cannot run it on the PCIe accelerator card (PAC).
fpgalpbk
This enable/disable FPGA loopback.
fpgastats
This get fpga mac statistics.
mactest
This compare mac addresses that read from MAC ROM with mac addresses read from Host side.
"},{"location":"sw/fpga_tools/fpgadiag/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/fpgadiag/#common-options","title":"Common options","text":"--help, -h
Print help information and exit.\n
--target=, -t
This switch specifies fpga (hardware) or ase (simulation). The default=fpga.\n
--mode=, -m
The test to run. The valid values are `lpbk1`, `read`,\n`write`, `trput`, and `sw`.\n
--config=, -c
A configuration file in the JSON format that specifies options for a test.\nIf an option is specified both in the configuration file and on the command \nline, the value in the configuration file takes precedence.\n
--dsm-timeout-usec
Timeout in microseconds for test completion. The test fails if not completed by \nspecified timeout. The default=1000000.\n
--socket-id=, -s
Socket ID encoded in FPGA Interface Manager (FIM). The default=0.\n
--bus=, -B
Bus number of the PCIe device. The default=0.\n
--device=, -D
Device number of the PCIe device. The default=0.\n
--function=, -F
Function number of the PCIe device. The default=0.\n
--freq=, -T
Clock frequency (in Hz) used for bandwidth calculation. The default=400000000 Hz (400 MHz).\n
eval_rst .. note:: This frequency is used only when the software cannot infer the frequency from the accelerator.
--suppress-hdr, -S
Suppress column headers for text output. The default=off.\n
--csv, -V
Comma separated value format. The default=off.\n
--suppress-stats
Suppress statistics output at the end of test. The default=off.\n
"},{"location":"sw/fpga_tools/fpgadiag/#lpbk1-test-options","title":"lpbk1 test options","text":"--guid=, -g
AFU ID to enumerate. The default=D8424DC4-A4A3-C413-F89E-433683F9040B.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I The default=wrline-M.\n
--cache-hint=, -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. The default=auto.\n
"},{"location":"sw/fpga_tools/fpgadiag/#read-test-options","title":"read test options","text":"--guid=, -g
AFU ID to enumerate. The default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--strided-access=S, -a
1<= S <= 64. The default=1.\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-hint=, -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Try to prime the cache with hits. The default=off. Try to prime the \ncache with misses. The default=off.\n
--cool-cpu-cache, -C
Try to prime the cpu cache with misses. The default=off.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. The default=auto\n
"},{"location":"sw/fpga_tools/fpgadiag/#write-test-options","title":"write test options","text":"--guid=, -g
AFU ID to enumerate. The default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--strided-access=S, -a
1<= S <= 64. The default=1.\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I The default=wrline-M\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Try to prime the cache with hits. The default=off. Try to prime the \ncache with misses. The default=off.\n
--cool-cpu-cache, -C
Try to prime the cpu cache with misses. The default=off.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1, random. The default=`WRITE-VC`.\n
--alt-wr-pattern, -l
Alternate Write Pattern. The default=off.\n
"},{"location":"sw/fpga_tools/fpgadiag/#trput-test-options","title":"trput test options","text":"--guid=, -g
AFU ID to enumerate. The default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--strided-access=S, -a
1<= S <= 64. The default=1\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I The default=wrline-M.\n
--cache-hint=, -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. The default=`WRITE-VC`.\n
"},{"location":"sw/fpga_tools/fpgadiag/#sw-test-options","title":"sw test options","text":"--guid=, -g
AFU ID to enumerate. The default=7BAF4DEA-A57C-E91E-168A-455D9BDA88A3.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I. The default=wrline-M.\n
--cache-hint= -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random The default=auto.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. The default=`WRITE-VC`.\n
--notice=, -N
Can be poll or csr-write. The default=poll.\n
"},{"location":"sw/fpga_tools/fpgadiag/#enable-fpga-n3000-ethernet-group-vfio-mdev","title":"Enable FPGA N3000 Ethernet group VFIO mdev","text":"FPGA DFL driver does not support any ioctls to read/write ethernet group info and registers. Users can read/write eth group registers by enabling VFIO mdev. Unbind the dfl_eth_group driver and bind vfio-mdev-dfl driver for ethernet group dfl-device; then userspace can take full control of ethernet group feature id 10.
Ethernet group must be enabled before running fpgalpbk, mactest tools.
"},{"location":"sw/fpga_tools/fpgadiag/#steps-to-enablecreate-vfio-mdev","title":"Steps to enable/create vfio mdev","text":"unbind eth group feature id 10:\n echo dfl-fme.0.8 > /sys/bus/dfl/drivers/dfl-eth-group/unbind\n echo dfl-fme.0.7 > /sys/bus/dfl/drivers/dfl-eth-group/unbind\nbind to vfio-mdev-dfl:\n echo vfio-mdev-dfl > /sys/bus/dfl/devices/dfl-fme.0.7/driver_override\n echo vfio-mdev-dfl > /sys/bus/dfl/devices/dfl-fme.0.8/driver_override\nload vfio driver:\n modprobe vfio_pci\n modprobe vfio_iommu_type1\n modprobe vfio_mdev\n modprobe vfio_mdev_dfl\ntrigger mdev:\n echo dfl-fme.0.7 >/sys/bus/dfl/drivers_probe\n echo dfl-fme.0.8 >/sys/bus/dfl/drivers_probe\n echo 83b8f4f2-509f-382f-3c1e-e6bfe0fa1001 > /sys/bus/dfl/devices/dfl-fme.0.7/mdev_supported_types/vfio-mdev-dfl-1/create\n echo 83b8f4f2-509f-382f-3c1e-e6bfe0fa1002 > /sys/bus/dfl/devices/dfl-fme.0.8/mdev_supported_types/vfio-mdev-dfl-1/create\n\nlinux kerenl msg after enabling mdev:\n i40e 0000:b3:00.0 eth1: NIC Link is Down\n i40e 0000:b1:00.1 eth0: NIC Link is Down\n vfio-mdev-dfl dfl-fme.2.7: MDEV: Registered\n vfio-mdev-dfl dfl-fme.2.8: MDEV: Registered\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1005: Adding to iommu group 140\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1005: MDEV: group_id = 140\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1006: Adding to iommu group 141\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1006: MDEV: group_id = 141\n
"},{"location":"sw/fpga_tools/fpgadiag/#remove-vfio-mdev","title":"Remove vfio mdev","text":" echo 1 | sudo tee /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1002/remove\n echo 1 | sudo tee /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove\n\n rmmod vfio_mdev_dfl\n modprobe dfl_eth_group\n\n echo dfl-fme.0.7 >/sys/bus/dfl/drivers_probe\n echo dfl-fme.0.8 >/sys/bus/dfl/drivers_probe\n\n echo dfl-eth-group > /sys/bus/dfl/devices/dfl-fme.0.7/driver_override\n echo dfl-eth-group > /sys/bus/dfl/devices/dfl-fme.0.8/driver_override\n
"},{"location":"sw/fpga_tools/fpgadiag/#fpgalpbk-test-options","title":"fpgalpbk test options","text":"--enable
Enable fpga phy loopback.\n
--disable
Disable fpga phy loopback.\n
--direction
Can be local, remote.\n
--type
Can be serial, precdr, postcdr.\n
--side
Can be line, host.\n
--port
0 <= port <= 7, the default is all.\n
"},{"location":"sw/fpga_tools/fpgadiag/#mactest-test-options","title":"mactest test options","text":"--offset
Read mac addresses from an offset, The default=0.\n
"},{"location":"sw/fpga_tools/fpgadiag/#examples","title":"EXAMPLES","text":"This command starts a lpbk1 test for the FPGA on bus 0x5e. The test copies 57535, 57536, 57537 ... up to 65535 cache lines, one line at a time. The test prints output in the comma separated values (CSV) format with the header suppressed.
./fpgadiag --mode=lpbk1 --target=fpga -V --bus=0x5e --begin=57535\n--end=65535 --cache-hint=rdline-I --cache-policy=wrpush-I --multi-cl=1\n--write-vc=vl0 --read-vc=vh1 --wrfence-vc=auto\n
This command starts a read test on the FPGA located on bus 0xbe. The test reads 2045 cache lines in the continuous mode with a 15-second timeout period. The reads use a strided pattern with a 10-byte stride length.
./fpgadiag --mode=read --target=fpga -V --bus=0xbe --begin=2045 --cont\n--timeout-sec=15 --cache-hint=rdline-I --multi-cl=1 -a=10 \n--read-vc=auto --wrfence-vc=auto\n
This command starts a sw test on the FPGA located on bus 0xbe. The test signals completion using a CSR write.
./fpgadiag --mode=sw --target=fpga -V --bus=0xbe --begin=4 --end=8192\n--cache-hint=rdline-I --cache-policy=wrline-I --notice=csr-write --write-vc=vl0\n--wrfence-vc=auto --read-vc=random \n
This command enable a fpgalpbk on the FPGA located on bus 0xbe.
./fpgadiag -m fpgalpbk --bus 0xbe --enable --direction local --type postcdr\n--side host\n
This command show fpgastats on the FPGA located on bus 0xbe.
./fpgadiag -m fpgastats --bus 0xbe\n
"},{"location":"sw/fpga_tools/fpgadiag/#troubleshooting","title":"TROUBLESHOOTING","text":"When a test fails to run or gives errors, check the following:
- Is the Intel FPGA driver properly installed? See Installation Guide for driver installation instructions.
- Are FPGA port permissions set properly? Check the permission bits of the port, for example,
/dev/intel-fpga-port-0. You need READ and WRITE permissions to run fpgadiag tests. - Is hugepage properly configured on the system? See Installation Guide for hugepage configuration steps. In particular,
fpgadiag requires a few 1 GB pages. - Is the required AFU loaded? See DESCRIPTION for information about what AFU the test requires.
- Are
--begin and --end values set properly? --end must be larger than the --begin. Also, --begin must be a multiple of the --multi-cl value. - The
--warm-fpga-cache and --cool-fpga-cache options in the read and write tests are mutually exclusive. - The timeout options are only meaningful for the continuous mode (with the
--cont option).
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/","title":"fpgaflash","text":""},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#synopsis","title":"SYNOPSIS","text":"fpgaflash [-h] {user,factory} file [bdf]\n
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#description","title":"DESCRIPTION","text":"fpgaflash updates the static FIM image loaded from flash at power-on.
If there are multiple devices in the system, fpgaflash must specify a BDF to select the correct device. If no BDF is specified, fpgaflash prints out the BDFs of any compatible devices.
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"{user, factory}
Specifies the type of flash programming.
user
Only reprograms the user image in flash.
factory
Reprograms the entire flash. A catastrophic failure during a factory update such as a power outage requires a USB cable and quartus_pgm to recover.
file
Specifies the Raw Programming Data File (rpd) to program into flash.
bdf
Specifies the bus, device and function (BDF) of device to program such as 04:00.0 or 0000:04:00.0. This flag is optional when there is a single device in the system.
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Print usage information.
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#example","title":"EXAMPLE","text":"fpgaflash user new_image.rpd 0000:04:00.0
Programs new_image.rpd to flash of device with BDF 0000:04:00.0.
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/","title":"super-rsu","text":""},{"location":"sw/fpga_tools/fpgaflash/superrsu/#synopsis","title":"SYNOPSIS","text":"super-rsu [-h] [-n] [--verify] | [ [--log-level {trace,debug,error,warn,info,notset}]\n [--log-file <filename>] [--rsu-only] [--with-rsu] [--force-flash] ]\n rsu_config\n
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#description","title":"DESCRIPTION","text":"super-rsu is a tool that can be used for flashing image files and commanding an Intel PAC device to perform RSU (remote system update - or a board reboot). Performing an RSU on an Intel PAC device will cause it to reload any firmware or programmable logic and restart its execution, a requirement for any updated firmware or programmable logic to take effect.
At the core of super-rsu is its configuration file (referred to in this document as 'rsu_config') which is essentially a manifest file for identifying both the target device and the binary images (and their versions) to be flashed.
At a high level, the flow of super-rsu should be: 1. Read and parse rsu_config file 2. Use product identifiers (like vendor, device and any additional vendor, device pairs that may be present in the PCIe bus) to locate all compatible devices on the PCIe bus. 3. For every device found on the system, update the device using the flash images defined in the \"flash\" section in the rsu_config data (or nvmupdate section). Each item in the \"flash\" section is a \"flash spec\" that contains: * The flash type (\"user\", \"bmc_fw\", \"bmc_img\", ...) * The filename of the image to flash. super-rsu will look for this file first in the same directory of the rsu_config file, and then look in the current working directory. * The version of the image. * An optional \"force\" indicator * An optional \"requires\" indicator The \"nvmupdate\" section is used to describe an Ethernet firmware file and its version. 4. Using the data in the \"nvmupdate\" and \"flash\" sections, the update routine involves: * If an \"nvmupdate\" section is present: 1. Locate the file on the file system to use to flash the Ethernet device. 2. Call nvmupdate to get an \"inventory\" of devices matching the vendor and device id in this section. 3. Use this data to dynamically generate an nvmupdate compatible configuration file. 4. Call nvmupdate with the generated configuration file to flash the Ethernet interfaces in the Vista Creek card (if version reported by system does not match the version in this section). * For each spec in the \"flash\" section: 1. Locate the file on the file system to use to flash. 2. Compare the version listed in the \"flash spec\" to version reported by the target component. 3. Create a task to call fpgaflash if either of the following conditions is met (and the revision specified is compatible): * The \"force\" indicator is present and set to true. * The version in the spec does not match the version reported by the system OR the flash type is factory type. * For each task created from the \"flash\" section: 1. Call fpgaflash with the command line arguments that correspond to the flash type and the file name in the spec used to create the task. This opens and controls the execution of fpgaflash in another process.
NOTE: If the system reports a revision for one of the components being flashed, this revision must be in the set of revisions listed in the manifest. Example: if the system reports 'a' for bmc_img and the manifest includes 'ab', then the image will be flashed.
NOTE: Each update routine is run in a thread specific to a device located on the PCIe bus. Every task in an update routine involves opening a new process that is controlled and managed by its update routine thread. If a task includes a timeout and the timeout is reached, a termination request will be sent to its process and it will be counted as a failure. If a global timeout is reached in the main thread, a termination request will be sent to each thread performing the update. Consequently, the update routine will give the current task extra time before terminating the process. The RSU operation will only be performed if requested with either --with-rsu command line argument or with the --rsu-only command line argument. The former will perform the RSU command upon successful completion of flash operations. The latter will skip the process of version matching and flashing images and will only perform the RSU command. It is recommended that super-rsu be executed again if any flash operation is interrupted.
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"rsu config
Specifies the name of the file containing the RSU configuration (in JSON format)
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Print usage information.
--verify
Compare versions of flashable components on the system against the manifest. Return non-zero exit if compatible components are not up to date.
-n, --dry-run
Don't perform any updates, just a dry run. This will print out commands that can be executed in a Linux shell.
--log-level {trace,debug,error,warn,info,notset}
Log level to use. Default is 'info'.
`--log-file (default: /tmp/super-rsu.log)
Emit log messages (with DEBUG level) to filename NOTE: The default log file (/tmp/super-rsu.log) is set to rollover every time super-rsu is executed. This will create numbered backups before truncating the log file. The maximum number of backups is 50.
--rsu-only
Only perform the RSU command.
--with-rsu
Perform RSU after updating flash components(experimental)
--force-flash
Flash all images regardless of versions matching or not.
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#configuration","title":"CONFIGURATION","text":"The following is the JSON schema expected by super-rsu. Any deviance from this schema may result in errors executing super-rsu.
{\n\"definitions\": {},\n\"$schema\": \"http://json-schema.org/draft-07/schema#\",\n\"$id\": \"http://example.com/root.json\",\n\"type\": \"object\",\n\"title\": \"The Root Schema\",\n\"required\": [\n\"product\",\n\"vendor\",\n\"device\",\n\"flash\"\n],\n\"optional\": [\n\"nvmupdate\",\n],\n\"properties\": {\n\"product\": {\n\"$id\": \"#/properties/product\",\n\"type\": \"string\",\n\"title\": \"The Product Schema\",\n\"default\": \"\",\n\"examples\": [\n\"n3000\"\n],\n\"pattern\": \"^(.*)$\"\n},\n\"vendor\": {\n\"$id\": \"#/properties/vendor\",\n\"type\": \"string\",\n\"title\": \"The Vendor Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x8086\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"device\": {\n\"$id\": \"#/properties/device\",\n\"type\": \"string\",\n\"title\": \"The Device Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x0b30\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"nvmupdate\": {\n\"$id\": \"#/properties/nvmupdate\",\n\"type\": \"object\",\n\"title\": \"The nvmupdate Schema\",\n\"required\": [\n\"vendor\",\n\"device\",\n\"filename\",\n\"version\"\n],\n\"optional\": [\n\"interfaces\"\n],\n\"properties\": {\n\"vendor\": {\n\"$id\": \"#/properties/nvmupdate/vendor\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Vendor Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x8086\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"device\": {\n\"$id\": \"#/properties/nvmupdate/device\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Device Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x0d58\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"interfaces\": {\n\"$id\": \"#/properties/nvmupdate/interfaces\",\n\"type\": \"number\",\n\"title\": \"The nvmupdate Interfaces Schema\",\n\"default\": \"1\",\n\"examples\": [\n2, 4\n]\n},\n\"filename\": {\n\"$id\": \"#/properties/nvmupdate/filename\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Filename Schema\",\n\"default\": \"\",\n\"examples\": [\n\"PSG_XL710_6p80_XLAUI_NCSI_CFGID2p61_Dual_DID_0D58_800049C6.bin\"\n],\n\"pattern\": \"^(.*)$\"\n},\n\"version\": {\n\"$id\": \"#/properties/nvmupdate/version\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Version Schema\",\n\"default\": \"\",\n\"examples\": [\n\"800049C6\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{8})$\"\n},\n\"timeout\": {\n\"$id\": \"#/properties/nvmupdate/timeout\",\n\"type\": \"string\",\n\"title\": \"The Timeout Schema\",\n\"default\": \"\",\n\"examples\": [\n\"10m\"\n],\n\"pattern\": \"^([0-9]+(\\\\.[0-9]+)?([dhms]))+$\"\n}\n}\n},\n\"flash\": {\n\"$id\": \"#/properties/flash\",\n\"type\": \"array\",\n\"title\": \"The Flash Schema\",\n\"items\": {\n\"$id\": \"#/properties/flash/items\",\n\"type\": \"object\",\n\"title\": \"The Items Schema\",\n\"required\": [\n\"filename\",\n\"type\",\n\"version\",\n\"revision\"\n],\n\"optional\": [\n\"enabled\",\n\"force\",\n\"timeout\",\n\"requires\"\n],\n\"properties\": {\n\"enabled\": {\n\"$id\": \"#/properties/flash/items/properties/enabled\",\n\"type\": \"boolean\",\n\"title\": \"The Enabled Schema\",\n\"default\": \"true\"\n},\n\"filename\": {\n\"$id\": \"#/properties/flash/items/properties/filename\",\n\"type\": \"string\",\n\"title\": \"The Filename Schema\",\n\"default\": \"\",\n\"examples\": [\n\"vista_creek_qspi_xip_v1.0.6.ihex\"\n],\n\"pattern\": \"^(.*)$\"\n},\n\"type\": {\n\"$id\": \"#/properties/flash/items/properties/type\",\n\"type\": \"string\",\n\"title\": \"The Type Schema\",\n\"default\": \"\",\n\"examples\": [\n\"bmc_fw\"\n],\n\"enum\": [\"user\", \"bmc_fw\", \"bmc_img\", \"dtb\", \"factory_only\",\n\"phy_eeprom\"]\n},\n\"version\": {\n\"$id\": \"#/properties/flash/items/properties/version\",\n\"type\": \"string\",\n\"title\": \"The Version Schema\",\n\"default\": \"\",\n\"examples\": [\n\"1.0.6\"\n],\n\"pattern\": \"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"\n},\n\"force\": {\n\"$id\": \"#/properties/flash/items/properties/force\",\n\"type\": \"boolean\",\n\"title\": \"The Force Schema\",\n\"default\": false,\n\"examples\": [\ntrue\n]\n},\n\"revision\": {\n\"$id\": \"#/properties/flash/items/properties/revision\",\n\"type\": \"string\",\n\"title\": \"The Revision Schema\",\n\"default\": \"\",\n\"examples\": [\n\"C\"\n],\n\"pattern\": \"^([A-Za-z])$\"\n},\n\"timeout\": {\n\"$id\": \"#/properties/nvmupdate/timeout\",\n\"type\": \"string\",\n\"title\": \"The Timeout Schema\",\n\"default\": \"\",\n\"examples\": [\n\"10m\"\n],\n\"pattern\": \"^([0-9]+(\\.[0-9]+)?([dhms]))+$\"\n},\n\"requires\": {\n\"$id\": \"#/properties/flash/items/properties/requires\",\n\"type\": \"string\",\n\"title\": \"The Requires Schema\",\n\"default\": \"\",\n\"examples\": [\n\"bmc_img >= 1.0.12\"\n],\n\"pattern\": \"^(([a-z_]+) ((<>!=)?=) ([0-9a-z\\\\.]+)$\"\n}\n}\n}\n}\n}\n}\n
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/","title":"fpgainfo","text":""},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#synopsis","title":"SYNOPSIS","text":" fpgainfo [-h] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\n {errors,power,temp,fme,port,bmc,mac,phy,security}\n
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#description","title":"DESCRIPTION","text":"fpgainfo displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac,security,events. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#fpgainfo-commands","title":"FPGAINFO COMMANDS","text":"errors
Show/clear errors of an FPGA resource that the first argument specifies. fpgainfo displays information in human readable form.
Error Description Catfatal Errors Bit 8 indicates an Injected Fatal error, bit 11 indicates an Injected Catastrophic Error. Inject Errors [2:0] are mainly writeable bits. Can read back values. (FME) Next Error [59:0] 60 LSBs are taken from the given error register that was triggered second, [60:61] 0 = FME0 Error, 1 = PCIe0 Error. (FME) First Error [59:0] 60 LSBs are taken from the given error register that was triggered first, [60:61] 0 = FME0 Error, 1 = PCIe0 Error. FME Errors Error from Partial Reconfiguration Block reporting a FIFO Parity Error has occurred. Non-fatal Errors Bit 6 is used to advertise an Injected Warning Error. power
Show total the power in watts that the FPGA hardware consumes.
temp
Show FPGA temperature values in degrees Celcius.
port
Show information about the port such as the AFU ID of currently loaded AFU.
fme
Show information about the FPGA platform including the partial reconfiguration (PR) Interface ID, the OPAE version, and the FPGA Interface Manager (FIM) ID.
The User1/User2/Factory Image Info lines reflect the information of the image that is present in the Flash.
The Bitstream Id line reflects the information of the image that is programmed in the FPGA.
bmc
Show all Board Management Controller sensor values for the FPGA resource, if available.
phy
Show information about the PHY integrated in the FPGA, if available.
mac
Show information about the MAC address in ROM attached to the FPGA, if available.
security
Show information about the security keys, hashs and flash count, if available.
events
Show information about events and sensors, if available.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"--help, -h
Prints help information and exit.
--version, -v
Prints version information and exit.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#common-arguments","title":"COMMON ARGUMENTS","text":"The following arguments are common to all commands and are optional.
-S, --segment
PCIe segment number of resource.
-B, --bus
PCIe bus number of resource.
-D, --device
PCIe device number of resource.
-F, --function
PCIe function number of resource.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#errors-arguments","title":"ERRORS ARGUMENTS","text":"The first argument to the errors command specifies the resource type. It must be one of the following: fme,port,all
fme
Show/clear FME errors.
port
Show/clear PORT errors.
all
Show/clear errors for all resources.
The optional <command-args> arguments are:
--clear, -c
Clear errors for the given FPGA resource.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#phy-arguments","title":"PHY ARGUMENTS","text":"The optional <command-args> argument is:
--group, -G
Select which PHY group(s) information to show.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#events-arguments","title":"EVENTS ARGUMENTS","text":"The optional <command-args> argument is:
--list,-l
List boots (implies --all).
--boot,-b
Boot index to use, i.e: \u00a0\u00a0\u00a0\u00a00 for current boot (default). \u00a0\u00a0\u00a0\u00a01 for previous boot, etc.
--count,-c
Number of events to print.
--all,-a
Print all events.
--sensors,-s
Print sensor data too.
--bits,-i
Print bit values too.
--help,-h
Print this help.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#examples","title":"EXAMPLES","text":"This command shows the current power telemetry:
./fpgainfo power\n
This command shows the current temperature readings:
./fpgainfo temp\n
This command shows FME resource errors:
./fpgainfo errors fme\n
This command clears all errors on all resources: ./fpgainfo errors all -c\n
This command shows information of the FME on bus 0x5e ./fpgainfo fme -B 0x5e\n
This command shows information of the FPGA security on bus 0x5e ./fpgainfo security -B 0x5e\n
This command shows all events and sensors information including sensor bits: ./fpgainfo events -asi\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/","title":"fpgamux","text":""},{"location":"sw/fpga_tools/fpgamux/fpgamux/#synopsis","title":"SYNOPSIS","text":"fpgamux [-h] [-S|--socket-id SOCKET_ID] [-B|--bus-number BUS] [-D|--device DEVICE] [-F|--function FUNCTION]\n [-G|--guid GUID] -m|--muxfile <filepath.json>\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#description","title":"DESCRIPTION","text":"fpgamux tests multiple AFUs that are synthesized into a single AFU along with the CCIP-MUX basic building block (BBB). The CCIP-MUX uses the upper bits in the MMIO addresses to route MMIO reads and writes to the AFU running on the corresponding CCIP-MUX port. fpgamux uses a configuration file that lists the software components and correct configuration. fpgamux only runs on the Integrated FPGA Platform. You cannot run it on the PCIe accelerator card (PAC).
.. note::
The OPAE driver discovers only the first AFU. The first software component in the configuration \n determines the GUID to use for enumeration. Use the -G|--guid option to override the GUID\n for the first software component.\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#options","title":"OPTIONS","text":"-S SOCKET_ID, --socket-id SOCKET_ID
socket id of FPGA resource.
-B BUS, --bus BUS
bus id of FPGA resource.
-D DEVICE, --device DEVICE
The device id of FPGA resource.
-F FUNCTION, --function FUNCTION
The function id of FPGA resource.
-G, --guid
Specifies the GUID to use for the resource enumeration.
-m, --muxfile <filepath.json>
The path to the fpgamux configuration file. This file must be in JSON format following the schema described below.
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#configuration","title":"CONFIGURATION","text":"fpgamux uses a configuration file (in JSON format) to determine what software components to instantiate and how to configure them to work with the AFUs. The schema includes the following elements:
[\n {\n \"app\" : \"fpga_app\",\n \"name\" : \"String\",\n \"config\" : \"Object\"\n }\n ]\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#examples","title":"EXAMPLES","text":"The following example shows a configuration with two components:
[\n {\n \"app\" : \"nlb0\",\n \"name\" : \"nlb0\",\n \"config\" :\n {\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"cont\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n },\n {\n \"app\" : \"nlb3\",\n \"name\" : \"nlb3\",\n \"config\" :\n {\n \"mode\" : \"read\",\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"strided-access\" : 1,\n \"cont\" : false,\n \"warm-fpga-cache\" : false,\n \"cool-fpga-cache\" : false,\n \"cool-cpu-cache\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"alt-wr-pattern\" : false,\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n }\n ]\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/","title":"fpgaport","text":""},{"location":"sw/fpga_tools/fpgaport/fpgaport/#synopsis","title":"SYNOPSIS","text":"fpgaport [-h] [-N NUMVFS] [-X] [--debug] {assign,release} device [port]\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#description","title":"DESCRIPTION","text":"The fpgaport enables and disables virtualization. It assigns and releases control of the port to the virtual function (VF). By default, the driver assigns the port to the physical function (PF) in the non-virtualization use case.
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"{assign, release}
Action to perform.\n
device
The FPGA device being targeted with this action.\n
port
The number of the port.\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-N NUMVFS, --numvfs NUMVFS
Create NUMVFS virtual functions. The typical value is 1.\n
-X, --destroy-vfs
Destroy all virtual functions prior to assigning.\n
--debug
Display additional log information.\n
-h, --help
Print usage information.\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#example","title":"EXAMPLE","text":"fpgaport release /dev/dfl-fme.0 0
Release port 0 from physical function control.\n
fpgaport assign /dev/dfl-fme.0 0
Assign port 0 to physical function control.\n
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/","title":"fpgasupdate","text":""},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#synopsis","title":"SYNOPSIS","text":"fpgasupdate [--log-level=<level>] file [bdf]
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#description","title":"DESCRIPTION","text":"The fpgasupdate command implements a secure firmware update for the following programmable accelerator cards (PACs): * Intel\u00ae PAC with Intel Arria\u00ae 10 GX FPGA * Intel\u00ae FPGA PAC D5005 * Intel\u00ae PAC N3000 * Intel\u00ae FPGA SmartNIC N6001-PL with Intel® Agilex® FPGA * Intel\u00ae FPGA IPU F2000X-PL
--log-level <level>
Specifies the `log-level` which is the level of information output to your command tool.\nThe following seven levels are available: `state`, `ioctl`, `debug`, `info`, `warning`,\n`error`, `critical`. Setting `--log-level=state` provides the most verbose output.\nSetting `--log-level=ioctl` provides the second most information, and so on. The default\nlevel is `info`.\n
file
Specifies the secure update firmware file to be programmed. This file may be to program a\nstatic region (SR), programmable region (PR), root entry hash, key cancellation, or other\ndevice-specific firmware.\n
bdf
The PCIe® address of the PAC to program. `bdf` is of the form `[ssss:]bb:dd:f`,\ncorresponding to PCIe segment, bus, device, function. The segment is optional. If\nyou do not specify a segment, the segment defaults to `0000`. If the system has only\none PAC you can omit the `bdf` and let `fpgasupdate` determine the address\nautomatically.\n
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#troubleshooting","title":"TROUBLESHOOTING","text":"To gather more debug output, decrease the --log-level parameter.
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#examples","title":"EXAMPLES","text":"fpgasupdate firmware.bin fpgasupdate firmware.bin 05:00.0 fpgasupdate firmware.bin 0001:04:02.0 --log-level=ioctl
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/","title":"host_exerciser","text":""},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#synopsis","title":"SYNOPSIS","text":"Usage: host_exerciser [OPTIONS] SUBCOMMAND\nOptions:\n -h,--help Print this help message and exit\n -p,--pci-address TEXT [<domain>:]<bus>:<device>.<function>\n -l,--log-level TEXT:{trace,debug,info,warning,error,critical,off}=warning\n stdout logging level\n -s,--shared open in shared mode, default is off\n -t,--timeout UINT=60000 test timeout (msec)\n -m,--mode UINT:value in {lpbk->0,read->1,trput->3,write->2} OR {0,1,3,2}=lpbk\n host exerciser mode {lpbk,read, write, trput}\n --cls UINT:value in {cl_1->0,cl_2->1,cl_4->2,cl_8->3} OR {0,1,2,3}=cl_1\n number of CLs per request{cl_1, cl_2, cl_4, cl_8}\n --continuousmode BOOLEAN=false\n test rollover or test termination\n --atomic UINT:value in {cas_4->9,cas_8->11,fadd_4->1,fadd_8->3,off->0,swap_4->5,swap_8->7} OR {9,11,1,3,0,5,7}=off\n atomic requests (only permitted in combination with lpbk/cl_1)\n --encoding UINT:value in {default->0,dm->1,pu->2,random->3} OR {0,1,2,3}=default\n data mover or power user encoding -- random interleaves both in the same stream\n -d,--delay BOOLEAN=false Enables random delay insertion between requests\n --interleave UINT=0 Interleave requests pattern to use in throughput mode {0, 1, 2}\n indicating one of the following series of read/write requests:\n 0: rd-wr-rd-wr\n 1: rd-rd-wr-wr\n 2: rd-rd-rd-rd-wr-wr-wr-wr\n --interrupt UINT:INT in [0 - 3]\n The Interrupt Vector Number for the device\n --contmodetime UINT=1 Continuous mode time in seconds\n --testall BOOLEAN=false Run all tests\n --clock-mhz UINT=0 Clock frequency (MHz) -- when zero, read the frequency from the AFU\nSubcommands:\n lpbk run simple loopback test\n mem run simple mem test\n
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#description","title":"DESCRIPTION","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc.
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#host-exerciser-loopback-he-lbk","title":"Host Exerciser Loopback (HE-LBK)","text":"HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: 1. Latency (AFU to Host memory read) 2. MMIO latency (Write+Read) 3. MMIO BW (64B MMIO writes) 4. BW (Read/Write, Read only, Wr only)
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#host-exerciser-memory-he-mem","title":"Host Exerciser Memory (HE-MEM)","text":"HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller.
Execution of these exercisors requires the user to bind specific VF endpoint to vfio-pci Bind the correct endpoint for a device with B/D/F 0000:b1:00.0
[user@localhost]: sudo opae.io init -d 0000:b1:00.2 user:user
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#host-exerciser-sub-commands","title":"HOST EXERCISER SUB COMMANDS","text":"lpbk
run host exerciser loopback test
mem
run host exerciser memory test
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"--help, -h
Prints help information and exit.
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#common-arguments-options","title":"COMMON ARGUMENTS / OPTIONS","text":"The following arguments are common to all commands and are optional.
-p,--pci-address
PCIe domain, bus, device, function number of fpga resource.
-l,--log-level
set host exerciser tool log level, trace, debug, info, warning, error, critical, off
-s,--shared
open FPGA PCIe resource in shared mode
-t,--timeout
host exerciser tool time out, by default time out 60000
-m,--mode
host exerciser test modes are lpbk, read, write, trput
--cls
Number of cachelines per request 1, 2, 3, 4.
--continuousmode
Configures test rollover or test termination mode.
--atomic
atomic requests.
--encoding
select data mover mode or power user mode or random.
-d,--delay
Enables random delay insertion between requests.
--interleave
Enables interleave requests in throughput mode. Value:3'b000-Rd,Wr,Rd,Wr Value:3'b001-Rd,Rd,Wr,Wr Value:3'b010-Rd,Rd,Rd,Rd,Wr,Wr,Wr,Wr Value:3'b011-Not supported
--interrupt
Accelerator interrupt vector Number.
--contmodetime
Continuous mode time in seconds.
--testall
Run all host exerciser tests.
--clock-mhz
pcie clock frequency, default value 350Mhz. When specified by the user, will not check value against AFU clock before calculating performance metrics.
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#examples","title":"EXAMPLES","text":"This command exerciser Loopback afu:
host_exerciser lpbk\n
This command exerciser memory afu:
host_exerciser mem\n
This command exerciser Loopback afu on pcie 000:3b:00.0:
host_exerciser --pci-address 000:3b:00.0 lpbk\n
This command exerciser Loopback afu on pcie 000:3b:00.0 and run in write mode:
host_exerciser --pci-address 000:3b:00.0 --mode write lpbl\n
This command exerciser Loopback afu on pcie 000:3b:00.0 and run 2 cache lines per request:
host_exerciser --pci-address 000:3b:00.0 --cls cl_2 lpbk\n
This command exerciser Loopback afu on pcie 000:3b:00.0 and run continuous mode for 10 seconds:
host_exerciser --pci-address 000:3b:00.0 -cls cl_1 -m 0 --continuousmode true --contmodetime 10 lpbk\n
"},{"location":"sw/fpga_tools/hssi/hssi/","title":"hssi","text":""},{"location":"sw/fpga_tools/hssi/hssi/#synopsis","title":"SYNOPSIS","text":"hssi COMMON_OPTIONS MODE MODE_OPTIONS
"},{"location":"sw/fpga_tools/hssi/hssi/#description","title":"DESCRIPTION","text":"The hssi application provides a means of interacting with the 10G, 100G, and 200G/400F HE-HSSI AFUs. In all operating modes, the application initializes the AFU and completes the desired transfer as described by the mode- specific options.
COMMON_OPTIONS - application options common to the 10G, 100g, and 200G/400G modes.
-h, --help
Display common command-line help and exit.\n
-p, --pci-address ADDR
The PCIe address of the desired accelerator in ssss:bb:dd.f format.\n
-s, --shared on|off
Whether to open the accelerator in shared mode. The default is off.\n
-t, --timeout VALUE
The application timeout value in milliseconds. The default timeout is 60000 msec.\n
MODE - select AFU. Valid values are hssi_10g, hssi_100g, hssi_200g_400g.
MODE_OPTIONS [hssi_10g] - application options specific to the 10G AFU.
-h, --help
Display 10G AFU specific command-line help and exit.\n
--port PORT
Select the QSFP port in the range 0-7. The default is port 0.\n
--num-packets PACKETS
The number of packets to transfer. The default is 1 packet.\n
--random-length fixed|random
Specify packet length randomization. Valid values are fixed and\nrandom. The default is fixed (no randomization).\n
--random-payload incremental|random
Specify payload randomization. Valid values are incremental and\nrandom. The default is incremental.\n
--packet-length LENGTH
Specify packet length. The default is 64 bytes.\n
--src-addr ADDR
Specify the source MAC address. The default value is 11:22:33:44:55:66.\n
--dest-addr ADDR
Specify the destination MAC address. The default value is 77:88:99:aa:bb:cc.\n
--rnd-seed0 SEED0
Specify the prbs generator bits [31:0]. The default is 1592590336.\n
--rnd-seed1 SEED1
Specify the prbs generator bits [47:32]. The default is 1592590337.\n
--rnd-seed2 SEED2
Specify the prbs generator bits [91:64]. The default is 155373.\n
MODE_OPTIONS [hssi_100g] - application options specific to the 100G AFU.
--port PORT
Select the QSFP port in the range 0-7. The default is port 0.\n
--eth-loopback on|off
Whether to enable loopback on the ethernet interface. Valid values are\non and off. The default is on.\n
--num-packets PACKETS
The number of packets to transfer. The default is 1 packet.\n
--gap random|none
Inter-packet gap. Valid values are random and none. The default is none.\n
--pattern random|fixed|increment
Pattern mode. Valid values are random, fixed, or increment. The default\nis random.\n
--src-addr ADDR
Specify the source MAC address. The default value is 11:22:33:44:55:66.\n
--dest-addr ADDR
Specify the destination MAC address. The default value is 77:88:99:aa:bb:cc.\n
--start-size SIZE
Specify the packet size in bytes, or the first packet size for --pattern increment.\n
--end-size SIZE
Specify the end packet size in bytes.\n
--end-select pkt_num|gen_idle
Specify packet generation end mode.\n
MODE_OPTIONS [pkt_filt_10g] - application options specific to the Packet Filter 10G AFU.
--dfl-dev DFL_DEV
Packet Filter DFL device, eg --dfl-dev dfl_dev.0\n
MODE_OPTIONS [pkt_filt_100g] - application options specific to the Packet Filter 100G AFU.
--dfl-dev DFL_DEV
Packet Filter DFL device, eg --dfl-dev dfl_dev.1\n
MODE_OPTIONS [hssi_200g_400g] - application options specific to the 200G/400G AFU.
--num-packets PACKETS
The number of packets to transfer. Must be a multiple of 32. Default value is 32. Increasing the timeout (--timeout) may be necessary if specifying a large number of packets.\n
"},{"location":"sw/fpga_tools/hssi/hssi/#examples","title":"EXAMPLES","text":"hssi -h hssi hssi_10g -h sudo hssi --pci-address=0000:3b:00.0 hssi_10g --eth-loopback=on --num-packets=500 sudo hssi --pci-address=0000:3b:00.0 hssi_100g --pattern=increment sudo hssi --pci-address=0000:0d:00.6 hssi_200g_400g --num-packets=640000
"},{"location":"sw/fpga_tools/hssi_config/readme/","title":"hssi_config","text":""},{"location":"sw/fpga_tools/hssi_config/readme/#synopsis","title":"Synopsis","text":"hssi_config reads or writes HSSI registers on either on an Intel\u00ae FPGA using the FPGA Interface Manager (FIM) or on an HSSI retimer card attached to the board. hssi_config is only available for the Integrated FPGA Platform. You cannot run it on the PCIe accelerator card (PAC).
"},{"location":"sw/fpga_tools/hssi_config/readme/#usage","title":"Usage","text":"hssi_config [--resource|-r <sysfs resource>] [--socket-id|s 0|1] command [command options]
Where command is one of the following:
dump [outfile.csv] [--input-file inputfile.csv]\niread instance (0,1) device-addr byte-address byte-count\n iwrite instance (0,1) device-addr byte-address byte1 [byte2 [byte3...]]\nload [inputfile.csv] [--c-header]\nread lane(0-15) reg-address\n rread device(0x30, 0x32, 0x34, 0x36) channel(0-3) address\n rwrite device(0x30, 0x32, 0x34, 0x36) channel(0-3) address value\n test (rd|rw) inputfile.csv [--acktimes] [--repeat N]\nwrite lane(0-15) reg-address value\n
The first argument is the command and any additional arguments are command arguments. The following options and commands are available:
"},{"location":"sw/fpga_tools/hssi_config/readme/#options","title":"Options","text":"[--resource|-r <sysfs resource path>
The resource path in the sysfs pseudo-filesystem. Example: /sys/devices/pci0000\\:5e/0000\\:5e\\:00.0/resource0
[--socket-id 0|1]
The socket id of the target FPGA. Required on two-socket systems to differentiate between the two possible target FPGAs.
"},{"location":"sw/fpga_tools/hssi_config/readme/#commands","title":"Commands","text":"dump [outfile.csv] [--input-file inputfile.csv]
Dump registers to stdout or to a file, if provided. hssi_config has a built-in set of registers to dump. The first argument is the path to a file to write. The command dumps to stdout if you do not specify a file name. Use the --input-file option to specify a different set of registers.
load [inputfile.csv] [--c-header]
Load a set of register values from either stdin or an input file, if provided. The first argument is the path to a file containing the registers to load. Loads from stdin if omitted.
Use --c-header to generate a C header file with an array of 64-bit numbers to write to the HSSI_CTRL register. This header file can substitute for the input file. NOTE: You must perform the acknowledge routine after each write.
read lane(0-15) reg-address
Read from a single XCVR (transceiver) register. The first command argument is the XCVR lane. Use -1 to specify a read from all lanes. The second argument is the XCVR address (offset).
write lane(0-15) reg-address value
Write to a single XCVR register. The first argument is the XCVR lane. The second argument is the XCVR address(offset). The third argument is the value to write to the register.
rread device(0x30, 0x32, 0x34, 0x36) channel(0-3) address
Read from a single retimer register. The first argument is the I2C device address. The second argument is the channel. The third argument is the register address (or I2C byte address).
rwrite device(0x30, 0x32, 0x34, 0x36) channel(0-3) address value
Write to a single retimer register. The first argument is the I2C device address. The second argument is the channel. The third argument is the register address (or I2C byte address). The fourth argument is the value to write.
iread instance (0,1) device-addr byte-address byte-count
Read from a device on the I2C bus. The first argument is the I2C controller instance (0 or 1). The second argument is the device address to read from. The third argument is the byte address of the register to read from the device. The fourth argument is the number of bytes to read.
iwrite instance (0,1) device-addr byte-address byte1 [byte2 [byte3...]]
Write to a device on the I2C bus. The first argument is the I2C controller instance (0 or 1). The second argument is the device address to read from. The third argument is the byte address of the register to read from the device. All subsequent arguments are the bytes to write to the device.
test (rd|rw) inputfile.csv [--acktimes]
Perform built-in test for reading or writing XCVR registers. The first argument is the path to a file containing the registers to test.
"},{"location":"sw/fpga_tools/hssi_config/readme/#overview","title":"Overview","text":"The hssi_config utility reads or writes hssi equalization parameters stored in either the transceiver (XCVR) registers or the registers of the retimer on the I2C bus. To access registers, the hssi controller writes to the HSSI_CTRL register and reads from the HSSI_STAT register in the FPGA Management Engine (FME). These two registers implement the HSSI AUX bus mailbox protocol to access devices on other buses. Because hssi_config maps the FME MMIO space directly, the FPGA driver is not required.
"},{"location":"sw/fpga_tools/hssi_config/readme/#locating-the-fme-device","title":"Locating the FME Device","text":"The FME reads and writes the HSSI_CTRL and HSSI_STAT registers on the NIOS device. The FME maps the MMIO address space of the FME identified by its resource in the sysfs psuedo-filesystem in Linux operating systems. To identify resource paths, use the lspci utility to query for Intel devices with device id of bcc0.
Example:
lspci -d 8086:bcc0
This command should print out at least one line like the following example:
5e:00.0 Processing accelerators: Intel Corporation Device bcc0
Use the first three numbers (bus:device.function) to locate the device resource in the sysfs filesystem:
/sys/devices/pci0000\\:<bus>/0000\\:<bus>\\:<device>.<function>/resource0
For example, the example above with bus of 5e, device of 00 and function of 0 would use a resource path as follows:
/sys/devices/pci0000\\:5e/0000\\:5e\\:00.0/resource0
"},{"location":"sw/fpga_tools/hssi_config/readme/#csv-file-format","title":"CSV File Format","text":"Any CSV file parsed by hssi_config must meet have at least four columns. The following table provides the column specifications:
Column Name Description 1 Register type. Can be either FPGA_RX, FPGA_TX, RTMR_RX, RTMR_TX (or their corresponding numeric values, 1-4). 2 Lane or Channel 0-15 for XCVR lanes on FPGA, 0-3 for retimer channels. -1 to designate all lanes or channels. 3 Device address (on I2C bus) Only applies to retimer registers. 0 - 3, -1 to designate all devices. 4 Register address (or offset) Examples: 0x213, 0x100. 5 Register value to write Examples: 0x1, 1, 0. Applies only when loading or writing registers."},{"location":"sw/fpga_tools/hssi_config/readme/#examples-of-commands","title":"Examples of Commands","text":""},{"location":"sw/fpga_tools/hssi_config/readme/#dumping-registers","title":"Dumping Registers","text":"Dump default register set to stdout:
>hssi_config dump
Dump default registers to a file, data.csv:
>hssi_config dump data.csv
Dump to an output file, data.csv, Registers specified in an input file, regspec.csv:
>hssi_config dump data.csv --input-file regspec.csv
"},{"location":"sw/fpga_tools/hssi_config/readme/#reading-single-registers","title":"Reading Single Registers","text":"Read register from XCVR at 0x2e1 on lane 0:
>hssi_config read 0 0x2e1
Read register 0x109 from retimer on channel 0, device 0x30, channel 1:
>hssi_config rread 0 0x30 0x109
"},{"location":"sw/fpga_tools/hssi_config/readme/#loading-registers","title":"Loading Registers","text":"Load registers specified in an input file called data.csv:
>hssi_config load data.csv
Load registers specified from stdin:
>hssi_config load
FPGA_RX,1,-1,0x213,0\nFPGA_RX,2,-1,0x213,0\nFPGA_RX,3,-1,0x213,0\n<CTRL-D>\n
"},{"location":"sw/fpga_tools/hssi_config/readme/#writing-single-registers","title":"Writing Single Registers","text":"Write 1 to XCVR register at 0x2e1 on lane 0:
>hssi_config write 0 0x2e1 1
Read register 0x109 from retimer on channel 0, device 0x30, channel 1:
>hssi_config rread 0 0x30 0x109
"},{"location":"sw/fpga_tools/hssi_config/readme/#testing-hssi-read-and-write","title":"Testing HSSI Read and Write","text":"> test (rd|rw) register-file.csv [--acktimes]
rd|wr
Specifies either a rd or wr of transceiver registers. For writes, every register in the file is read from and written to in the following sequence:
1. Read the register, save the value 2. Write 1 to the register 3. Read the register, verify that the register value is 1 4. Write 0 to the register 5. Read the register, verify that the register value is 0 6. Write the original value to the register 7. Read the register, assert it is the original value
register=file.csv
Specifies the path to a file containing the set of registers to test.
--acktimes
Specifies the time spent in the ack routine. When measured, a summary of ack times prints to stdout. This argument is optional.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/","title":"HSSI ethernet loopback","text":""},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#synopsis","title":"SYNOPSIS","text":"hssiloopback [-h] [--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS] --loopback [{enable,disable}]\n
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#description","title":"DESCRIPTION","text":"The hssiloopback tool enables and disable ethernet loopback.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Prints usage information
--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS
The PCIe address of the desired fpga in ssss:bb:dd.f format. sbdf of device to program (e.g. 04:00.0 or 0000:04:00.0). Optional when one device in system.
--loopback [{enable,disable}]
Ethernet enable or disable loopback.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#examples","title":"EXAMPLES","text":"hssiloopback --pcie-address 0000:04:00.0 --loopback enable
Enables ethernet loopback
hssiloopback --pcie-address 0000:04:00.0 --loopback disable
Disable ethernet loopback
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/","title":"HSSI ethernet mac","text":""},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#synopsis","title":"SYNOPSIS","text":"hssimac [-h] --pcie-address PCIE_ADDRESS [--port PORT]\n
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#description","title":"DESCRIPTION","text":"The hssimac tool provides Maximum TX and RX frame size.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Prints usage information
--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS
The PCIe address of the desired fpga in ssss:bb:dd.f format. sbdf of device to program (e.g. 04:00.0 or 0000:04:00.0).
--port PORT
hssi port number.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#examples","title":"EXAMPLES","text":"hssimac --pcie-address 0000:04:00.0 --port 1
prints Maximum TX and RX frame size for port 1.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/","title":"HSSI ethernet statistics","text":""},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#synopsis","title":"SYNOPSIS","text":"hssistats [-h] [--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS]\n
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#description","title":"DESCRIPTION","text":"The hssistats tool provides the MAC statistics.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Prints usage information
--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS
The PCIe address of the desired fpga in ssss:bb:dd.f format. sbdf of device to program (e.g. 04:00.0 or 0000:04:00.0). Optional when one device in system.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#examples","title":"EXAMPLES","text":"hssistats --pcie-address 0000:04:00.0
prints the MAC statistics
"},{"location":"sw/fpga_tools/hssi_loopback/readme/","title":"hssi_loopback","text":""},{"location":"sw/fpga_tools/hssi_loopback/readme/#name","title":"NAME","text":"hssi_loopback - Software utility to run HSSI loopback tests on FPGA
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#synopsis","title":"SYNOPSIS","text":"hssi_loopback [[--bus|-b <bus number>] [--device | -d <device number>] [--function | -f <function number>]]|[--socket-id <socket-id>] [--mode|-m auto|e40|e10] [send [<source port> [<destination port>] [--packet-count|-c <count>] [--packet-delay|-d <delay>] [--packet-length|-l <length>]] |status [clear] | stop | readmacs
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#description","title":"DESCRIPTION","text":"The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. The hssi_loopback utility tests both external and internal loopbacks. hssi_loopback runs an external loopback test when the command line arguments include both source and destination ports. hssi_loopback runs an internal loopback test when command line arguments include a single port. hssi_loopback only runs on the Intel Xeon with Arria 10 FPGA. You cannot run it on the Intel PAC (programmable accelerator card).
NOTE: The following limitations apply to the current version of hssi_loopback:
- For the external loopback the two port arguments can be the same. For the e10 design, the ports should be the same.
- The
hssi_loopback test supports only the e40 and e10 E2E AFUs. The e10 E2E AFU tests HSSI with a retimer card. - The
hssi_loopback test uses the control and status registers (CSRs) defined in the AFU.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#options","title":"OPTIONS","text":"-S SOCKET_ID, --socket-id SOCKET_ID
Socket ID FPGA resource.
-B BUS, --bus BUS
Bus ID of FPGA resource.
-D DEVICE, --device DEVICE
Device ID of FPGA resource.
-F FUNCTION, --function FUNCTION
Function ID of FPGA resource.
-G, --guid
Specifies guid for the resource enumeration.
-m, --mode
One of the following: [auto, e40, e10] auto is the default and indicates that the software runs the mode based on the first accelerator functional unit it identifies.
-t, --timeout
Timeout (in seconds) before the application terminates in continuous mode. Continuous mode is the default when you do not specify the number of packets.
-y, --delay
Delay (in seconds) between printing out a simple status line. Default is 0.100 seconds (100 milliseconds).
-c, --packet-count
The number of packets to send.
-d, --packet-delay
The delay in between packets. This delay is the number of 100 MHz clock cycles, roughly 10 nanoseconds.
-s, --packet-size
The packet size to send. The minimum is 46 bytes and the maximum is 1500 bytes. The default is 46 bytes.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#commands","title":"COMMANDS","text":"send <source port> [<destination port>] [--packet-count|-c <count>] [--packet-delay|-d <delay>] [--packet-length|-l <length>]
Send packets from one port to the other. If the command line does not specify a destination port, the test runs an internal loopback. Otherwise, the test runs an external loopback from the source port to the destination port.
status [clear]
Read and interpret the status registers and print to the screen. clear clears the status registers.
stop
Issue a stop command to all Ethernet controllers in the AFU.
readmacs
Read and display the port MAC addresses. An EEPROM stores the MAC addresses.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#exit-codes","title":"EXIT CODES","text":"0 Success - Number of packets received are equal to the number of packets sent and no errors are reported.
-1 Loopback failure - Either number of packets does not match or the test detected errors.
-2 Errors parsing arguments.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#examples","title":"EXAMPLES","text":"Read the MAC addresses of the AFU loaded on bus 0x5e:
>sudo hssi_loopback readmacs -B 0x5e\n
Run an external loopback, sending 100 packets from port 0 to port 1. The AFU is on bus 0x5e:
>sudo hssi_loopback -B 0x5e send 0 1 -c 100\n
Run an internal loopback until a timeout of 5 seconds is reached. The AFU is on bus 0x5e:
>sudo hssi_loopback -B 0x5e send 0 -t 5\n
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/","title":"mem_tg","text":""},{"location":"sw/fpga_tools/mem_tg/mem_tg/#synopsis","title":"SYNOPSIS","text":"Usage: mem_tg [OPTIONS] SUBCOMMAND\nOptions:\n -h,--help Print this help message and exit\n -g,--guid TEXT=4DADEA34-2C78-48CB-A3DC-5B831F5CECBB\n GUID\n -p,--pci-address TEXT [<domain>:]<bus>:<device>.<function>\n -l,--log-level TEXT:{trace,debug,info,warning,error,critical,off}=info\n stdout logging level\n -s,--shared open in shared mode, default is off\n -t,--timeout UINT=60000 test timeout (msec)\n -m,--mem-channel UINT=0 Target memory bank for test to run on (0 indexed)\n --loops UINT=1 Number of read/write loops to be run\n -w,--writes UINT=1 Number of unique write transactions per loop\n -r,--reads UINT=1 Number of unique read transactions per loop\n -b,--bls UINT=1 Burst length of each request\n --stride UINT=1 Address stride for each sequential transaction\n --data UINT:value in {fixed->0,prbs15->2,prbs31->3,prbs7->1,rot1->3} OR {0,2,3,1,3}=fixed\n Memory traffic data pattern: fixed, prbs7, prbs15, prbs31, rot1\n -f,--mem-frequency UINT=0 Memory traffic clock frequency in MHz\nSubcommands:\n tg_test configure & run mem traffic generator test\n
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#description","title":"DESCRIPTION","text":"The memory traffic generator (TG) used to exercise and test available memory channels with a configurable traffic pattern.
Execution of this application requires the user to bind the specific VF endpoint containing the mem_tg AFU id to vfio-pci
In the TG, read responses are checked against a specified pattern. If the application is configured to perform a read only test on a region of memory that has not previously been initialized to contain that pattern it will flag a test failure.
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"--help, -h
Prints help information and exit.
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#common-arguments-options","title":"COMMON ARGUMENTS / OPTIONS","text":"The following arguments are common to all commands and are optional.
-p,--pci-address
PCIe domain, bus, device, function number of fpga resource.
-l,--log-level
set application log level, trace, debug, info, warning, error, critical, off
-s,--shared
open FPGA PCIe resource in shared mode
-t,--timeout
mem_tg application time out, by default time out 60000
-m,--mem-channel
Target memory bank for test to run on (0 indexed) default: 0
--loops
Number of read/write loops to be run default: 1
-w,--writes
Number of unique write transactions per loop. default: 1
-r,--reads
Number of unique read transactions per loop default: 1
-b,--bls
AXI4 burst length of each request. Supports 1-256 transfers beginning from 0. default: 0
--stride
Address stride for each sequential transaction (>= burst length) default: 1
--data
Memory traffic data pattern. 0 = fixed {0xFF, 0x00} 1 = prbs7 2 = prbs15 3 = prbs31 4 = rot1
default: fixed
-f, --mem-frequency
Memory traffic clock frequency in MHz default: 300 MHz
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#examples","title":"EXAMPLES","text":"This command will run a basic read/write test on the channel 0 traffic generator:
mem_tg tg_test\n
This command will run the application for an afu on pcie 000:b1:00.7:
mem_tg --pci-address 000:b1:00.7 tg_test\n
This command will test channel 2 write bandwidth:
mem_tg -loops 1000 -w 1000 -r 0 -m 2 tg_test\n
This command will perform a read bandwidth test with a burst of 16 on channel 1 and perform a data comparison with the prbs7 pattern:
mem_tg -loops 1000 -w 0 -r 1000 -b 0xF --data prbs7 -m 1 tg_test\n
This command will perform a read/write test with 1 MB strided access to channel 0 memory:
mem_tg -loops 10000 --stride 0x100000 tg_test\n
"},{"location":"sw/fpga_tools/mmlink/mmlink/","title":"mmlink","text":""},{"location":"sw/fpga_tools/mmlink/mmlink/#synopsis","title":"Synopsis","text":"mmlink [-v] [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-P <TCP port>] [-I <IP Address>]
"},{"location":"sw/fpga_tools/mmlink/mmlink/#description","title":"Description","text":"The Remote Signal Tap logic analyzer provides real-time hardware debugging for the Accelerator Function Unit (AFU). It provides a signal trace capability that the Quartus Prime software adds to the AFU. The Remote Signal Tap logic analyzer provides access to the Remote Signal Tap part of the Port MMIO space and then runs the remote protocol.
"},{"location":"sw/fpga_tools/mmlink/mmlink/#examples","title":"Examples","text":"./mmlink -B 0x5e -P 3333
MMLink app starts and listens for connection.
"},{"location":"sw/fpga_tools/mmlink/mmlink/#options","title":"Options","text":"-v,--version
Prints version information and exits.
-B,--bus
FPGA Bus number.
-D,--device
FPGA Device number.
-F,--function
FPGA function number.
-S,--socket
FPGA socket number.
-P,--port
TCP port number.
-I,--ip
IP address of FPGA system.
"},{"location":"sw/fpga_tools/mmlink/mmlink/#notes","title":"Notes","text":"Driver privilege:
Change AFU driver privilege to user:
$ chmod 777 /dev/intel-fpga-port.0\n
Change locked memory size:
edit the file /etc/security/limits.conf
$ sudo vi /etc/security/limits.conf\n\nuser hard memlock 10000\n\nuser soft memlock 10000\n
Exit terminal and log into a new terminal.
Verify that the locked memory is now set: ``` $ ulimit -l 10000
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/","title":"ofs.uio","text":""},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#synopsis","title":"SYNOPSIS","text":"ofs.uio [-h] [--pcie-address PCIE_ADDRESS] [--uio uiox] [--feature-id FEATURE_ID] [--region-index REGION_INDEX] [--mailbox-cmdcsr offset] [--bit-size {8,16,32,64}] [--peek offset] [--poke offset value] [--mailbox-read offset] [--mailbox-dump address size] [--mailbox-write address value]
ofs.uio [--uio uiox] [--peek offset] ofs.uio [--uio uiox] [--poke offset value] ofs.uio [--uio uiox] [--mailbox-read address] ofs.uio [--uio uiox] [--mailbox-write address value] ofs.uio [--uio uiox] [--mailbox-dump address size] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--peek offset] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--poke offset value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-read address] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-write address value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-dump address size]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#description","title":"DESCRIPTION","text":"ofs.uio is a tool that provides user space access to DFL UIO devices, command line options like peek, poke, mailbox-read, mailbox-write, mailbox-dump to access Configuration and Status Registers (CSRs).
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#peek","title":"Peek","text":"Peek/Read UIO CSR offset ofs.uio [--uio uio] [--peek offset] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#poke","title":"Poke","text":"Poke/Write value to UIO CSR offset ofs.uio [--uio uio] [--poke offset value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--poke offset value]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-read","title":"Mailbox Read","text":"Read CSR address using mailbox ofs.uio [--uio uio] [--mailbox-read address] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-read address]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-write","title":"Mailbox Write","text":"Write value to CSR address using mailbox ofs.uio [--uio uio] [--mailbox-write address value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-write address value]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-dump","title":"Mailbox Dump","text":"Reads/Dumps block size of CSR address using mailbox ofs.uio [--uio uio] [--mailbox-dump address size] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-dump address size]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#bit-size","title":"Bit size","text":"Read/Write bit-field 8,16,32,64 sizes ofs.uio [--uio uio] --bit-size 8 [--peek offset] ofs.uio [--uio uio] --bit-size 32 [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#pcie-address","title":"PCIe Address","text":"PCIE_ADDR PCIe address of FPGA device ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#uio-region-index","title":"UIO region index","text":"UIO region index, default region index is 0 ofs.uio [--uio uio] --region-index 0 [--peek offset] ofs.uio [--uio uio] --region-index 1 [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-command-status-csr-offset","title":"Mailbox command status csr offset","text":"Mailbox command status csr offset, default value set to dfl pcie subsystem system feature mailbox command status register offset 0x28 ofs.uio [--uio uio] --mailbox-cmdcsr 0xa8 [--mailbox-read address] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] --mailbox-cmdcsr 0xa8 [--mailbox-read address]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#examples","title":"EXAMPLES","text":"Peek/Read
ofs.uio --uio uio0 --peek 0x0\npeek(0x0): 0x3000000010002015\n\nofs.uio --uio uio6 --peek 0x0\npeek(0x0): 0x3000000100000020\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x15 --peek 0x0\npeek(0x0): 0x3000000010002015\n\nofs.uio --uio uio0 --peek 0x0 --bit-size 32\npeek(0x0): 0x10002015\n
Poke/Write
ofs.uio --uio uio6 --peek 0x8\npeek(0x8): 0x0\nofs.uio --uio uio6 --poke 0x8 0xabcdd12345\npoke(0x8):0xabcdd12345\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x15 --peek 0x0\npeek(0x8): 0x0\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x15 --poke 0x8 0x1234\npoke(0x8):0x1234\n
Mailbox Read
ofs.uio --uio uio6 --mailbox-read 0x0\nMailboxRead(0x0): 0x1000000\nofs.uio --uio uio6 --mailbox-read 0x8\nMailboxRead(0x8): 0x110c000\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-read 0x0\nMailboxRead(0x0): 0x1000000\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-read 0x8 \nMailboxRead(0x8): 0x110c000\n
Mailbox Write
ofs.uio --uio uio6 --mailbox-write 0x0 0x1234\nMailboxWrite(0x0):0x1234\nofs.uio --uio uio6 --mailbox-read 0x0\nMailboxRead(0x0):0x1234\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-write 0x0 0x1234\nMailboxWrite(0x0):0x1234\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-read 0x0 \nMailboxRead(0x0):0x1234\n
Mailbox Dump
ofs.uio --uio uio6 --mailbox-dump 0x0 0x10\nMailboxDump(0x0): 0x1000000\nMailboxDump(0x4): 0x1000000\nMailboxDump(0x8): 0x110c000\nMailboxDump(0xc): 0x110c000\nMailboxDump(0x10): 0x0\nMailboxDump(0x14): 0x0\nMailboxDump(0x18): 0x0\nMailboxDump(0x1c): 0x0\nMailboxDump(0x20): 0x0\nMailboxDump(0x24): 0x0\nMailboxDump(0x28): 0x0\nMailboxDump(0x2c): 0x0\nMailboxDump(0x30): 0x0\nMailboxDump(0x34): 0x0\nMailboxDump(0x38): 0x0\nMailboxDump(0x3c): 0x0\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-dump 0x0 0x10\nMailboxDump(0x0): 0x1000000\nMailboxDump(0x4): 0x1000000\nMailboxDump(0x8): 0x110c000\nMailboxDump(0xc): 0x110c000\nMailboxDump(0x10): 0x0\nMailboxDump(0x14): 0x0\nMailboxDump(0x18): 0x0\nMailboxDump(0x1c): 0x0\nMailboxDump(0x20): 0x0\nMailboxDump(0x24): 0x0\nMailboxDump(0x28): 0x0\nMailboxDump(0x2c): 0x0\nMailboxDump(0x30): 0x0\nMailboxDump(0x34): 0x0\nMailboxDump(0x38): 0x0\nMailboxDump(0x3c): 0x0\n
"},{"location":"sw/fpga_tools/opae-mem/opae-mem/","title":"opae-mem","text":""},{"location":"sw/fpga_tools/opae-mem/opae-mem/#synopsis","title":"SYNOPSIS","text":"opae-mem ls [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-t,--token-type {device,accel}] [-i,--interface {dfl,vfio,dfl_ase,vfio_ase,uio}] opae-mem peek [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-c,--count COUNT] [-o,--output OUTPUT] [-F,--format {simple,hex,json}] offset opae-mem poke [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-j,--json JSON_FILE] [offset] [value] opae-mem mb-read [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-b,--mailbox-base BASE] [-c,--count COUNT] [-t,--timeout TIMEOUT] [-s,--sleep SLEEP] [-o,--output OUTPUT] [-F,--format {simple,hex,json}] address opae-mem mb-write [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-b,--mailbox-base BASE] [-t,--timeout TIMEOUT] [-s,--sleep SLEEP] [-j,--json JSON_FILE] [address] [value] opae-mem lock [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] opae-mem unlock [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID]
"},{"location":"sw/fpga_tools/opae-mem/opae-mem/#description","title":"DESCRIPTION","text":"opae-mem is a utility that provides access to FPGA accelerator MMIO. It is written on top of the OPAE Python bindings; so it works with any FPGA accelerator, whether the accelerator is connected via DFL, VFIO, UIO, or ASE.
Accelerators are enumerated using the opae-mem ls command:
$ sudo chmod 666 /dev/uio* $ opae-mem ls [0000:d8:00.0] (8086:bcce 8086:0000) UIO 0x14 00000000-0000-0000-0000-000000000000 UIO 0x20 00000000-0000-0001-0000-000000000000 [0000:3b:00.0] (8086:bcce 8086:0000) UIO 0x15 00042415-0004-2415-0000-001100010000 UIO 0x20 00000000-0000-0001-0000-000000000000 UIO 0x14 00000000-0000-0000-0000-000000000000
This output shows two FPGA cards with a total of five FPGA MMIO regions.
The first card at address 0000:d8:00.0 has two MMIO regions accessible via UIO:
Feature ID 0x14: s10 IOPLL Feature ID 0x20: PCIe Subsystem
The second card at address 0000:3b:00.0 has three MMIO regions accessible via UIO:
Feature ID 0x15: HSSI Subsystem Feature ID 0x20: PCIe Subsystem Feature ID 0x14: s10 IOPLL
The peek command provides a way to view a range of MMIO addresses:
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 0x0 00000000: 3000000010000020 0000000000000000 | ......0........| 00000010: 0000000000000001 0000000000000000 |................|
Here, --count 4 causes the command to display four 64-bit qwords, from addresses 0x0 through 0x18.
The default format is hex display, which is modeled after the output of hexdump -C. The format can be controlled using the -F option. Valid values for -F are simple, hex, and json.
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 -F simple 0x0 00000000: 3000000010000020 00000008: 0000000000000000 00000010: 0000000000000001 00000018: 0000000000000000
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 -F json 0x0 { \"0x00000000\": \"0x3000000010000020\", \"0x00000008\": \"0x0000000000000000\", \"0x00000010\": \"0x0000000000000001\", \"0x00000018\": \"0x0000000000000000\" }
The output of opae-mem peek can be sent to a file using the -o option. This is useful for capture/playback. Playback is available with the opae-mem poke command.
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 -F json -o file.json 0x0 $ cat file.json { \"0x00000000\": \"0x3000000010000020\", \"0x00000008\": \"0x0000000000000000\", \"0x00000010\": \"0x0000000000000001\", \"0x00000018\": \"0x0000000000000000\" }
The poke command provides a way to write MMIO addresses:
$ opae-mem poke -d 0000:3b:00.0 -f 0x20 0x0 0xc0cac01a
In the above poke command, 0x0 is the MMIO offset to write and 0xc0cac01a is the value.
The output of the opae-mem peek command with -F json can be played back as a series of writes to an MMIO region using the opae-mem poke --json option:
$ opae-mem poke -d 0000:3b:00.0 -f 0x20 --json file.json
The opae-mem mb-read command is used to issue read requests to an FPGA mailbox interface:
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 0x0 mb-read [0000:3b:00.0] 0x20 This command needs -b mailbox_base. For feature_id 0x20, try -b 0x0028
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 0x0 00000000: 01000000 00000001 01104000 00000000 |.........@......|
Each mailbox address represents a 32-bit data value.
Like peek, the default display format for mb-read is hex. To change the display format, use the -F option, which accepts simple, hex, or json.
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 -F simple 0x0 00000000: 01000000 00000004: 00000001 00000008: 01104000 0000000c: 00000000
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 -F json 0x0 { \"0x00000028\": { \"0x00000000\": \"0x01000000\", \"0x00000004\": \"0x00000001\", \"0x00000008\": \"0x01104000\", \"0x0000000c\": \"0x00000000\" } }
The mb-write command provides a way to issue write commands to an FPGA mailbox interface.
$ opae-mem mb-write -d 0000:3b:00.0 -f 0x20 -b 0x28 0x0 0xc0cac01a
In the above command, 0x0 is the mailbox address and 0xc0cac01a is the 32-bit data value to be written.
The output of the opae-mem mb-read command with -F json can be played back as a series of writes to a mailbox interface using the opae-mem mb-write --json option:
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 -F json -o mb.json 0x0 $ opae-mem mb-write -d 0000:3b:00.0 -f 0x20 -b 0x28 --json mb.json
Each of the above commands has explicitly specified the -d PCIE_ADDR and -f FEATURE_ID parameters, making for some long command lines. To shorten the length, opae-mem can be \"locked\" to a (device, feature_id) pair:
$ opae-mem lock -d 0000:3b:00.0 -f 0x20 lock [0000:3b:00.0] 0x20 OK
Once \"locked\" to a device, issuing the command again displays the lock status:
$ opae-mem lock [locked 0000:3b:00.0 0x20] lock currently held by 0000:3b:00.0 0x20.
From now until the time the session is unlocked, opae-mem commands may omit the explicit -d and -f parameters:
$ opae-mem peek 0x0 00000000: 3000000010000020 | ......0 |
\"Locking\" is simply a convenient way to shorten the opae-mem command line. Each of the other commands operates in the same way, as if -d and -f were specified explicitly.
Note: a \"lock\" can be overridden by specifying -d and/or -f:
$ opae-mem -V peek -d 0000:d8:00.0 -f 0x14 -c 4 0x0 [locked 0000:3b:00.0 0x20 [override 0000:d8:00.0 0x14]] peek [0000:d8:00.0] 0x14 offset=0x0 region=0 format=hex 00000000: 3000000010000014 0000000000000000 |.......0........| 00000010: 0000000000000000 1000000000000000 |................|
The preceding command used a lock override by specifying a different device address to -d and a different feature_id to -f. The -V (verbose) option was given to show the lock override status.
The unlock command is used to release a lock:
$ opae-mem unlock [locked 0000:3b:00.0 0x20] unlock Please tell me the device / feature id to unlock. (-d 0000:3b:00.0 -f 0x20)
$ opae-mem unlock -d 0000:3b:00.0 -f 0x20 [locked 0000:3b:00.0 0x20] unlock [0000:3b:00.0] 0x20 OK
$ opae-mem lock lock Give me the device address and feature id.
"},{"location":"sw/fpga_tools/opae.io/opae.io/","title":"opae.io","text":""},{"location":"sw/fpga_tools/opae.io/opae.io/#synopsis","title":"SYNOPSIS","text":"opae.io ls [-v,--viddid VIDDID] [-s,--sub-viddid SUB_VIDDID] [--all] [--system-class] opae.io init [-d,--device PCI_ADDR] [USER[:GROUP]] opae.io release [-d,--device PCI_ADDR] opae.io [-d,--device PCI_ADDR] [-r,--region REGION] walk [--offset [OFFSET]] [-u,--show-uuid] [-D,--dump] [-c,--count COUNT] [-y,--delay DELAY] [-s,--safe] opae.io [-d,--device PCI_ADDR] [-r,--region REGION] dump [--offset [OFFSET]] [-o,--output OUTPUT] [-f,--format {bin,hex}] [-c,--count COUNT] opae.io [-d,--device PCI_ADDR] [-r,--region REGION] peek OFFSET opae.io [-d,--device PCI_ADDR] [-r,--region REGION] poke OFFSET VALUE opae.io [-d,--device PCI_ADDR] [-r,--region REGION] script SCRIPT ARG1 ARG2 ... ARGN opae.io [-d,--device PCI_ADDR] [-r,--region REGION]
"},{"location":"sw/fpga_tools/opae.io/opae.io/#description","title":"DESCRIPTION","text":"opae.io is an interactive Python environment packaged on top of libopaevfio.so, which provides user space access to PCIe devices via the vfio-pci driver. The main feature of opae.io is its built-in Python command interpreter, along with some Python bindings that provide a means to access Configuration and Status Registers (CSRs) that reside on the PCIe device.
Note: opae.io peek/poke will call fpgaEnumerate() internally and so a reset will be asserted on the AFU region. You can avoid initializing these registers by passing opae.io a python script directly.
opae.io has two operating modes: command line mode and interactive mode.
"},{"location":"sw/fpga_tools/opae.io/opae.io/#command-line-mode","title":"COMMAND LINE MODE","text":"To view the accelerator devices that are present on the system, opae.io provides the ls command option.
opae.io ls [-v,--viddid VIDDID] [-s,--sub-viddid SUB_VIDDID] [--all] [--system-class]
Each accelerator device is listed along with the PCIe address, the PCIe vendor/device ID, a brief description of the device, and the driver to which the device is currently bound.
Device filtering is available by providing a Vendor ID:Device ID pair, eg -v 8086:bcce. Further filtering can be done by providing a sub- Vendor ID:sub-Device ID pair, eg -s 8086:1771. The --all option provides a list of all of the PCIe devices in the system, which an be quite verbose. The --system-class option prints the PCIe database class of the accelerator device, rather than the product name.
opae.io provides an option to initialize a PCIe device for use with the vfio-pci driver. In order for the device CSRs to be accessed from user space, the device must first be bound to the vfio-pci driver. This is the job of the init command option.
opae.io init [-d,--device PCI_ADDR] [USER[:GROUP]]
The init command unbinds the specified device from its current driver and binds it to vfio-pci. This creates a new vfio group under /dev/vfio. This group path is then used by the libopaevfio.so library to interact with the device.
To release the PCIe device from vfio-pci and return it to use with its previous driver, the release command option is used.
opae.io release [-d,--device PCI_ADDR]
The release command option reverses the actions of the last init command, releasing the device from vfio-pci and binding it to the driver which was bound at the time the init command was issued.
The walk command option traverses and displays the Device Feature List of the given region.
opae.io walk [--offset [OFFSET]] [-u,--show-uuid] [-D,--dump] [-c,--count COUNT] [-y,--delay DELAY] [-s,--safe]
The various fields of each Device Feature Header are displayed. The --show-uuid option additionally displays the GUID for each feature. OFFSET can be used to specify the beginning of the DFL in the MMIO region. --dump displays the raw DFH contents in hex format. COUNT limits the number of DFH entries traversed. DELAY causes a pause between each printout. --safe examines each DFH offset for proper alignment.
The dump command provides a means to dump the MMIO space in ASCII hex or binary format.
opae.io dump [--offset [OFFSET]] [-o,--output OUTPUT] [-f,--format {bin,hex}] [-c,--count COUNT]
OFFSET specifies the starting MMIO offset. OUTPUT gives the name of a file to capture the dump output, where sys.stdout is used by default. --format allows changing the output format. COUNT specifies the number of qwords to dump.
The peek command option reads and displays a CSR value.
opae.io peek OFFSET
The poke command option writes a given value to a CSR.
opae.io poke OFFSET VALUE
opae.io can also execute Python scripts from the command line. These Python scripts may contain calls to the device built-in functions that are available during an interactive session. Refer to the description of interactive mode for details.
opae.io script myscript.py a b c
In order to enter the interactive mode of opae.io, simply invoke it and optionally pass the desired device address and MMIO region options.
opae.io [-d,--device PCI_ADDR] [-r,--region REGION]
"},{"location":"sw/fpga_tools/opae.io/opae.io/#interactive-mode","title":"INTERACTIVE MODE","text":"Upon entering interactive mode, opae.io begins a Python interpreter session and displays the command prompt shown below:
0000:3f:00.0[0]>>
The first portion of the prompt shows the address of the active PCIe device, here 0000:3f:00.0. The part in square brackets shows the active MMIO region, here [0].
The interpreter waits for a valid Python command, then attempts to execute the given command in the usual way. The only differences between the traditional Python command intepreter and opae.io are that opae.io provides 1) the notion of an active PCIe device and MMIO region and 2) several built-in functions and objects that allow manipulating the active device and MMIO region.
"},{"location":"sw/fpga_tools/opae.io/opae.io/#built-in-functions","title":"BUILT-IN FUNCTIONS","text":"The opae.io built-in functions assume an active device and MMIO region. Attempting to use the built-in functions without first opening a device and region will result in errors.
peek(OFFSET)
The peek built-in function reads and displays a CSR value from the active device and region, at the offset supplied by its argument.
0000:3f:00.0[0]>> peek(0x28) 0xdeadbeef
poke(OFFSET, VALUE)
The poke built-in function writes the given VALUE to the current MMIO region at the given OFFSET.
0000:3f:00.0[0]>> poke(0x28, 0xdeadbeef)
read_csr(OFFSET)
The read_csr built-in function returns the value of the CSR at the active MMIO region and the given OFFSET.
0000:3f:00.0[0]>> print('0x{:0x}'.format(read_csr(0x28))) 0xdeadbeef
write_csr(OFFSET, VALUE)
The write_csr built-in function writes the given VALUE to the current MMIO region at the given OFFSET.
0000:3f:00.0[0]>> write_csr(0x28, 0xdeadbeef)
device(PCI_ADDR)
The device built-in function allows changing the active PCIe device.
0000:3f:00.0[0]>> device('0000:2b:00.0') 0000:2b:00.0>>
region(REGION)
The region built-in function allows changing the active MMIO region.
0000:2b:00.0>> region(0) 0000:2b:00.0[0]>>
allocate_buffer(SIZE)
The allocate_buffer built-in function creates and returns a DMA buffer object. The underlying buffer will be SIZE bytes in length.
0000:2b:00.0[0]>> b1 = allocate_buffer(4096) 0000:2b:00.0[0]>> print(b1.size, '0x{:0x}'.format(b1.address), b1.io_address) 4096 0x7f9361c66000 0
version()
The version built-in function returns a tuple containing the four components used to identify the opae.io version:
0000:2b:00.0[0]>> print(version()) ('opae.io', 0, 2, 0)
"},{"location":"sw/fpga_tools/opae.io/opae.io/#built-in-objects","title":"BUILT-IN OBJECTS","text":"opae.io interactive mode provides two global objects corresponding to the current device and that device's current MMIO region. These objects are referred to by global variables the_device and the_region, respectively.
The device class:
the_device.descriptor() : method that returns the integer file descriptor of the VFIO container.
0000:2b:00.0[0]>> print(the_device.descriptor()) 5
the_device.repr() : method that is invoked when a device object is printed.
0000:2b:00.0[0]>> print(the_device) 0000:2b:00.0
the_device.allocate(SIZE) : method that allocates and returns a system_buffer object. The buffer will be mapped into the DMA space of the_device.
0000:2b:00.0[0]>> b1 = the_device.allocate(4096)
the_device.pci_address() : read-only property that returns the PCIe address of the_device.
0000:2b:00.0[0]>> print(the_device.pci_address) 0000:2b:00.0
the_device.num_regions : read-only property that returns the number of MMIO regions in the_device.
0000:2b:00.0[0]>> print(the_device.num_regions) 2
the_device.regions : read-only property that returns a list of the active MMIO regions of the_device:
0000:2b:00.0[0]>> print(the_device.regions) [0, 2]
The region class:
the_region.write32(OFFSET, VALUE) : method that writes a 32-bit VALUE to the CSR at OFFSET.
the_region.read32(OFFSET) : method that returns a 32-bit CSR at the given OFFSET.
0000:2b:00.0[0]>> the_region.write32(0x28, 0xdeadbeef) 0000:2b:00.0[0]>> print('0x{:0x}'.format(the_region.read32(0x28))) 0xdeadbeef
the_region.write64(OFFSET, VALUE): method that writes a 64-bit VALUE to the CSR at OFFSET.
the_region.read64(OFFSET): method that returns a 64-bit CSR at the given OFFSET.
0000:2b:00.0[0]>> the_region.write64(0x28, 0xbaddecaf) 0000:2b:00.0[0]>> print('0x{:0x}'.format(the_region.read64(0x28))) 0xbaddecaf
the_region.index(): method that returns the MMIO index of the_region.
0000:2b:00.0[0]>> print(the_region.index()) 0
the_region.repr(): method that is invoked when a region object is printed.
0000:2b:00.0[0]>> print(the_region) 0
the_region.len(): method that is invoked to determine the MMIO region size.
0000:2b:00.0[0]>> print(len(the_region)) 524288
The allocate_buffer() built-in function and the device.allocate() method return objects of type system_buffer.
The system_buffer class is as follows:
buf.size: read-only property that gives the buffer size.
0000:2b:00.0[0]>> print(b1.size) 4096
buf.address: read-only property that gives the buffer's user mode virtual address.
0000:2b:00.0[0]>> print('0x{:0x}'.format(b1.address)) 0x7f2c15d8200
buf.io_address: read-only property that gives the buffer's IO address.
0000:2b:00.0[0]>> print('0x{:0x}'.format(b1.io_address)) 0x0
buf.__getitem__ and buf.__setitem__: indexing get/set of 64-bit data item.
0000:2b:00.0[0]>> b1[0] = 0xdecafbad 0000:2b:00.0[0]>> print('0x{:0x}'.format(b1[0])) 0xdecafbad
buf.read8(OFFSET) buf.read16(OFFSET) buf.read32(OFFSET) buf.read64(OFFSET) : methods that read the given size data item from the given buffer OFFSET.
buf.fill8(VALUE) buf.fill16(VALUE) buf.fill32(VALUE) buf.fill64(VALUE) : methods that fill the buffer with the given VALUE, using the given size.
b1.compare(b2): method that compares buffers. The method returns the index of the first byte that miscompares, or the length of b1.
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/","title":"opaeuio","text":""},{"location":"sw/fpga_tools/opaeuio/opaeuio/#synopsis","title":"SYNOPSIS","text":"opaeuio [-h] [-i] [-r] [-d DRIVER] [-u USER] [-g GROUP] [-v] [device]
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/#description","title":"DESCRIPTION","text":"The opaeuio command enables the binding/unbinding of a DFL device to/from the dfl-uio-pdev device driver. See https://kernel.org/doc/html/v4.14/driver-api/uio-howto.html for a description of uio.
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/#options","title":"OPTIONS","text":"device The DFL device name, eg dfl_dev.10
-h, --help
Display command-line help and exit.\n
-i, --init
Specifies binding mode operation - initialize the given device for uio.\nUsed in conjunction with -u, -g, and -d.\n
-r, --release
Specifies unbinding mode operation - release the given device from uio.\n
-d DRIVER, --driver DRIVER
Specifies the device driver to bind to when binding to uio.\nThe default value is dfl-uio-pdev.\n
-u USER, --user USER
The user ID to assign when binding to uio. A new device node is created in\n/dev when the device is bound to uio. Use this option to specify\nthe new device owner.\n
-g GROUP, --group GROUP
The group ID to assign when binding to uio. Use this option to specify the\nnew device group for the device created in /dev.\n
-v, --version
Display script version information and exit.\n
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/#examples","title":"EXAMPLES","text":"opaeuio -h opaeuio -v sudo opaeuio -i -u lab -g labusers dfl_dev.10 sudo opaeuio -r dfl_dev.10
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/","title":"opaevfio","text":""},{"location":"sw/fpga_tools/opaevfio/opaevfio/#synopsis","title":"SYNOPSIS","text":"opaevfio [-h] [-i] [-r] [-d DRIVER] [-u USER] [-g GROUP] [-n] [-v] [addr]
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/#description","title":"DESCRIPTION","text":"The opaevfio command enables the binding/unbinding of a PCIe device to/from the vfio-pci device driver. See https://kernel.org/doc/Documentation/vfio.txt for a description of vfio-pci.
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/#options","title":"OPTIONS","text":"addr The PCIe address of the device in ssss:bb:dd.f format, eg 0000:7f:00.0
-h, --help
Display command-line help and exit.\n
-i, --init
Specifies binding mode operation - initialize the given addr for vfio.\nUsed in conjunction with -u, -g, and -n.\n
-r, --release
Specifies unbinding mode operation - release the given addr from vfio.\nUsed in conjunction with -d.\n
-d DRIVER, --driver DRIVER
Specifies the device driver to bind to when releasing from vfio.\nWhen omitted, the device is not rebound to a driver (default).\n
-u USER, --user USER
The user ID to assign when binding to vfio. A new device node is created in\n/dev/vfio when the device is bound to vfio-pci. Use this option to specify\nthe new device owner.\n
-g GROUP, --group GROUP
The group ID to assign when binding to vfio. Use this option to specify the\nnew device group for the device created in /dev/vfio.\n
-n, --no-sriov
Do not enable SR-IOV when binding to vfio. The default value for this option\nis FALSE, ie the script should specify SR-IOV functionality when binding to\nthe vfio-pci driver. When omitted, the modprobe command which loads the vfio-pci\ndriver will contain the `enable_sriov=1` option. When given, it will not.\n
-v, --version
Display script version information and exit.\n
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/#examples","title":"EXAMPLES","text":"opaevfio -h opaevfio -v sudo opaevfio -i -u lab -g labusers 0000:7f:00.0 sudo opaevfio -r 0000:7f:00.0
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/","title":"Pac hssi config","text":"# pac_hssi_config #\n\n## SYNOPSIS ##\n```console\npac_hssi_config.py [-h] subcommand [subarg] [bdf]\n
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#description","title":"DESCRIPTION","text":"The pac_hssi_config.py tool exercises the Ethernet 10 Gbps (10GbE) and 40GbE transceivers for designs using the Intel\u00ae Programmable Acceleration Card (PAC) with Intel Arria\u00ae 10 GX FPGA. This tool does not support the Intel Xeon\u00ae Processor with Integrated FPGA.
The two required arguments to the pac_hssi_config.py tool specify the subcommand and bus, device, and function (BDF) for the PCIe device under test. You must provide the BDF parameter for systems with more than one PCIe card.
.. note::\n If you do not provide the BDF when required, the command prints a list of valid BDFs for the system. You can also\n determine the BDF using the ``lspci`` command.\n
For usage help, type the following at a command prompt:
pac_hssi_config.py [-h|--help]
To configure the network ports, send data, and read statistics, use the following form of the pac_hssi_config.py script:
pac_hssi_config.py subcommand [subarg] [bdf]
Only a subset of subcommand arguments support subarg.
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#table-1-general-subcommands","title":"Table 1. General Subcommands","text":"Subcommand Subarg Description stat N/A Prints high speed serial interface (HSSI) controller statistics. eeprom N/A Reads the 128-bit unique board ID, MAC address, and board-specific IDs from EEPROM."},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#table-2-1040-gbe-traffic-generation-subcommands","title":"Table 2. 10/40 GbE Traffic Generation Subcommands","text":"Subcommand Subarg Description e10init and e40init N/A Initializes HSSI PHY to 10GbE or 40GbE mode. Clears statistics and enable internal HSSI transceiver loopback. e10loop and e40loop On/Off Turns on or off internal HSSI transceiver loopback. e10reset and e40reset On/Off Asserts or deasserts AFU reset. Clears packet statistics and disables internal HSSI transceiver loopback. e10send and e40send N/A Sends 1,000,000 1500-byte packets. For 10GbE sends packets on all four ports. 40GbE has a single port. e10stat and e40stat N/A Prints packet statistics. e10statclr and e40statclr N/A Clears packet statistics. Use this command after switching loopback modes to clear any transient statistics accumulated during the mode switch. The transceiver equalization eqwrite and eqread subcommands write and read transceiver equalization settings. These subcommands require you to specify the transceiver channel, the equalization setting, and the value (for writes). Use the following form for the eqwrite command:
pac_hssi_config.py eqwrite [transceiver channel number] [equalization setting] [equalization value] [bdf]
Use the following form for the eqreadcommand:
pac_hssi_config.py eqread [transceiver channel number] [equalization setting] [bdf]
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#table-3-transceiver-equalization-subcommands","title":"Table 3. Transceiver Equalization Subcommands","text":"Subcommand Channel Number Equalization Setting Value eqwrite 0-3 0 = Continuous time-linear equalization (CTLE) 1 = Variable gain amplifier (VGA) 2 = DCGAIN 3 = Pre-emphasis first post-tap 4 = Pre-emphasis second post-tap 5 = Pre-emphasis first pre-tap 6 = Pre-emphasis second pre-tap 7 = Differential output voltage (VOD) Specifies the value for the specified equalization setting. eqread 0-3 0 = Continuous time-linear equalization (CTLE) 1 = Variable gain amplifier (VGA) 2 = DCGAIN 3 = Pre-emphasis first post-tap 4 = Pre-emphasis second post-tap 5 = Pre-emphasis first pre-tap 6 = Pre-emphasis second pre-tap 7 = Differential output voltage (VOD) N/A For more information about reconfiguring transceiver analog parameter settings In Arria\u00ae 10 devices, refer to \"Changing PMA Analog Parameters\" in the Intel\u00ae Arria\u00ae 10 Transceiver PHY User Guide.
"},{"location":"sw/fpga_tools/packager/packager/","title":"packager","text":""},{"location":"sw/fpga_tools/packager/packager/#synopsis","title":"SYNOPSIS","text":"packager <cmd> [arguments]
"},{"location":"sw/fpga_tools/packager/packager/#description","title":"Description","text":"The packager provides tools that Accelerator Functional Unit (AFU) developers use to create Accelerator Function (AF) files. The AF file is the programming file for an AFU on Intel\u00ae FPGA platforms. The packager tool concatenates the metadata from the JSON file to a raw binary file (.rbf) that the Intel Quartus\u00ae Prime software generates.
The packager's only function is to create an AF file. Refer to Packager Command Syntax for more information about invoking the packager. The packager depends on a JSON file to describe AFU metadata. Refer to Accelerator Description File for more information about the JSON file.
The packager requires Python 2.7.1 and Python 2.7.3. The tool indicates if it is being called with a compatible of Python.
"},{"location":"sw/fpga_tools/packager/packager/#packager-command-syntax","title":"Packager Command Syntax","text":"The packager is a command line tool with the following syntax:
$ packager <cmd> [arguments]
The following table describes the <CMD> arguments:
Command Arguments Description create-gbs --rbf=<RBF_PATH> --afu=<AFU_JSON_PATH> --gbs=<GBS_PATH> --set-value=<key>.<value> Creates the AF file. The engineering name for this file is the green bit stream, abbreviated gbs. The --rbf and --afu arguments are required. <RBF_PATH> is the path to the RBF file for the AFU. The Quartus\u00ae Prime software generates this RBF by compiling the AFU design. <AFU_JSON_PATH> is the path to the Accelerator Description file. This is a JSON file that describes the metadata that create-gbs appends to the RBF. <GBS_PATH> is the path to the RBF file for the FPGA Interface Manager (FIM) that contains the FPGA interface unit and other interfaces. If you do not specify the --gbs, the command defaults to <rbf_name>.gbs. You can use the optional --set-value=<key>.<value> argument to set values for JSON metadata. To set more than one JSON value, list a series of <key>.<value> pairs. modify-gbs --gbs=<gbs_PATH> Modifies the AF file. The --input-gbsargument is required. If you do not provide the --output-gbs argument, modify-gbs overwrites the --input-gbs file. Use the --set-value=<key>.<value> argument to set values for JSON metadata. To set more than one JSON value, list a series of <key>.<value> pairs. gbs-info --input-gbs=<gbs_PATH> Prints information about the AF file. The --input-gbs argument is required. get-rbf --gbs=<GBS_PATH> --rbf=<RBF_PATH> Creates the RBF by extracting it from the AF file. The --gbsargument is required. If you do not specify the --rbf argument, the command defaults to <gbs_name.rbf . None, or any <CMD> --help Summarizes the <CMD> options. Typing packager --help gives a list of <CMD> values. Typing packager <CMD> --help provides detailed help for <CMD>"},{"location":"sw/fpga_tools/packager/packager/#examples","title":"Examples","text":"To generate an AF file, run:
$ packager create-gbs --rbf=<RBF_PATH> --afu=<AFU_JSON_PATH> --gbs=<GBS_PATH>
TIP: JSON files are very particular about syntax such as trailing commas. If you are getting errors, use jsonlint.com to validate that your JSON is formatted correctly.
To modify metadata in an existing AF, run the following command:
$ packager modify-gbs --input-gbs=<PATH_TO_GBS_TO_BE_MODIFIED> --outputgbs=<NAME_FOR_NEW_GBS> --set-value <key>:<value>
You can pass in a number of : pairs with --set-value to update values in an AF.
To print the metadata of an existing AF:
$ packager get-info --gbs=<GBS_PATH>
To extract the RBF from the AF:
$ packager get-rbf --gbs=<GBS_PATH> --rbf=<NAME_FOR_RBF>
"},{"location":"sw/fpga_tools/packager/packager/#accelerator-description-file","title":"Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Progammable Accelerator Engine (OPAE) uses this metadata during reconfiguration. Here is an example file:
{\n \"version\": 1,\n \"platform-name\": \"DCP\",\n \"afu-image\": {\n \"magic-no\": 488605312,\n \"interface-uuid\": \"01234567-89AB-CDEF-0123-456789ABCDEF\",\n \"power\": 0,\n \"accelerator-clusters\": [{\n \"name\": \"dma_test_afu\",\n \"total-contexts\": 1, \n \"accelerator-type-uuid\": \"331DB30C-9885-41EA-9081-F88B8F655CAA\"\n }\n ] \n }\n}\n
The packager stores these parameter values in the resultant AF. After reprogramming the AFU using partial reconfiguration (PR), the software driver reconfigures the PLLs by writing the clock-frequency-high and clock-frequency-low values (if present) over the PCIe\u00ae and CCI interfaces. .. note::
The JSON file format may change as the architecture evolves. Any changes to the current format trigger an update\nto the version number. \n
CATEGORY | NAME | TYPE | DESCRIPTION | MANDATORY ---------|------|------|-------------|:----------:| Per-AFU | version | Integer | Version of the metadata format. | Yes Per-AFU | magic-no (to be deprecated)| Integer | Magic no. Associated with the FPGA Interface Manager. | No Per-AFU | platform-name | String | Name of the platform for which the metadata is intended. The field value is \u201cDCP\u201d for Intel Acceleration Stack for FPGAs. | No Per-AFU | interface-uuid | UUID | Interface id associated with the FPGA Interface Manager. | Yes Per-AFU | power | Integer | Accelerator Function power consumption, in watts. Set to 0 for Intel Acceleration Stack for FPGAs platforms. | Yes Per-AFU | clock-frequency-low | Float | Clock frequency for 1st PLL (Clock network)1 in MHz. | No Per-AFU | clock-frequency-high | Float | Clock frequency for 2nd PLL (0 if absent) in MHz. | No Per-AFC Cluster | total-contexts | Integer | Number of AFCs in this cluster. Always be 1 in current architectures. | Yes Per-AFC Cluster | afc-type-uuid | UUID | AFC type = AFU ID in current architectures. | Yes Per-AFC Cluster | name | string | AFC name = AFU name in current architectures. | Yes
Date Intel Acceleration Stack Version Changes Made 2018.05.21 DCP 1.1 Beta (works with Quartus Prime Pro 17.1.1) Fixed typos."},{"location":"sw/fpga_tools/pci_device/pci_device/","title":"pci_device","text":""},{"location":"sw/fpga_tools/pci_device/pci_device/#synopsis","title":"SYNOPSIS","text":"pci_device [-h] [-E] device-filter [{aer,bind,plug,remove,rescan,topology,unbind,unplug,vf}]
"},{"location":"sw/fpga_tools/pci_device/pci_device/#description","title":"DESCRIPTION","text":"pci_device is a tool to aid in common operations for managing PCIe devices and drivers.
"},{"location":"sw/fpga_tools/pci_device/pci_device/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/pci_device/pci_device/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"`device filter`\n\nPCIe address of a device or vendor/device ID pair.\nThe PCIe address follows the format of [segment:]bus:device.function\nwhile the vendor/device ID pair follows the format [vendor ID]:[device ID]\nwhere at least one of these must be present.\n\n`{aer,bind,plug,remove,rescan,topology,unbind,unplug,vf}`\n\naction to perform on device\n\n`aer`\nPerform AER (Advanced Error Reporting) operations.\nThe aer action has its own sub-commands which are listed below:\n\n* `dump` sub-command will print out the AER error counters as reported\n by the sysfs files for the device.\n* `mask` can either print out the current AER mask bits or set them\n * If `show` or `print` (or nothing) is given after the `mask`\n command, it will show the current mask bits for AER.\nBy default output will be written in stdout but can be written to an\noutput file if `-o|--output FILENAME` argument is given.\n * If `all` is given after the `mask` command, it will mask all bits\n (by setting the values to 0xffffffff and 0xffffffff).\n * If `off` is given after the `mask` command, it will unmask all\n bits (by setting the values to 0x0 and 0x0).\n * If two numbers are present after the `mask` command, those two\n numbers will be used to set the mask bits.\nValues for setting the mask can also be read in from an input file if\n`-i|--input FILENAME` argument is given.\n\n_NOTE_: mask related operations require root privileges\n\n`bind`\n\nAssociate a device with its driver.\n\n`plug`\n\nRestore a device that was previously given to `pci_device <device> unplug`\n\n`remove`\n\nRemove the pci device from the pci bus\n\n`rescan`\n\nRescan the bus as identified by the bus component of the PCIe device address\n\n'topology`\n\nPrint the PCIe topology from the root port to the PCIe device.\nThis shows the PCIe tree rooted at the PCIe root port.\nEach line shows the the PCIe address, vendor ID, and device ID along with\nthe driver bound to the device. The indentation is used to show\nparent/child relationship of devices.\n\nThe line listing the target PCIe device as identified by the given PCIe\naddress will be highlighted in green while the endpoints will be\nhighlighted in cyan.\n\nThe example below shows the topology of an N3000 device with eight virtual\nfunctions created from one of the Ethernet controllers:\n\n```console\n[pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)] (pcieport)\n [pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:3c:09.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:3f:00.0), pci_id(0x8086, 0x0b30)] (dfl-pci)\n [pci_address(0000:3c:11.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:43:00.0), pci_id(0x8086, 0x0b32)] (no driver)\n [pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:3d:02.0), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:00.1), pci_id(0x8086, 0x0d58)] (i40e)\n [pci_address(0000:3d:02.7), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.5), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.3), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.1), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n [pci_address(0000:3d:02.6), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.4), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.2), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:41:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n [pci_address(0000:41:00.1), pci_id(0x8086, 0x0d58)] (i40e)\n\n```\n\n`unbind`\n\nUnbind the driver bound to the device.\n\n`unplug`\n\nRemove device from PCI bus in anticipation of a RSU event by configuring its root port and associated endpoints.\n\n`vf`\n\nCreate/destroy VFs (virtual functions) by setting the number here.\nThe number given here will be written to sriov_numvfs sysfs file triggering\nthe PCIe subsystem to create/destroy VFs so that the current number of VFs\nwill be equal to the given number. If the number given is outside of the total VFs supported, an error message will be displayed to indicate this.\n
"},{"location":"sw/fpga_tools/pci_device/pci_device/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"`-h, --help`\n\nshow this help message and exit\n\n`-E, --other-endpoints`\n\nperform action on peer PCIe devices\n
"},{"location":"sw/fpga_tools/pci_device/pci_device/#examples","title":"EXAMPLES","text":"pci_device 0000:3d:00.0 remove\npci_device 0000:3d:00.0 rescan\npci_device 3d:00.0 topology\npci_device :0b30 topology\npci_device :0b30 aer\npci_device :0b30 aer mask\npci_device :0b30 aer mask all\npci_device :0b30 aer mask -o mask.dat\npci_device :0b30 aer mask -i mask.dat\n
"},{"location":"sw/fpga_tools/rsu/rsu/","title":"rsu","text":""},{"location":"sw/fpga_tools/rsu/rsu/#synopsis","title":"SYNOPSIS","text":"rsu [-h] [-d] {bmc,bmcimg,retimer,fpga,sdm,fpgadefault} [PCIE_ADDR]\n
"},{"location":"sw/fpga_tools/rsu/rsu/#description","title":"DESCRIPTION","text":""},{"location":"sw/fpga_tools/rsu/rsu/#mode-1-rsu","title":"Mode 1: RSU","text":"rsu bmc --page=(user|factory) [PCIE_ADDR]\nrsu retimer [PCIE_ADDR]\nrsu fpga --page=(user1|user2|factory) [PCIE_ADDR]\nrsu sdm --type=(sr|pr|sr_cancel|pr_cancel) [PCIE_ADDR]\n
Perform RSU (remote system update) operation on PAC device given its PCIe address. An RSU operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either BMC, Retimer, SDM, (on devices that support these) or the FPGA.
"},{"location":"sw/fpga_tools/rsu/rsu/#mode-2-default-fpga-image","title":"Mode 2: Default FPGA Image","text":"rsu fpgadefault --page=(user1|user2|factory) --fallback=<csv> [PCIE_ADDR]\n
Set the default FPGA boot sequence. The --page option determines the primary FPGA boot image. The --fallback option allows a comma-separated list of values to specify fallback images.
"},{"location":"sw/fpga_tools/rsu/rsu/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"{bmc,bmcimg,retimer,fpga,sdm,fpgadefault}
type of RSU operation or set Default FPGA Image operation.
PCIE_ADDR PCIe address of device to do rsu (e.g. 04:00.0 or 0000:04:00.0)
"},{"location":"sw/fpga_tools/rsu/rsu/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help show this help message and exit
-d, --debug log debug statements
--force force rsu operation
"},{"location":"sw/fpga_tools/rsu/rsu/#example","title":"EXAMPLE","text":"# rsu bmc --page=user 25:00.0\n
Triggers a boot of the BMC image (user page) for the device with PCIe address 25:00.0.
# rsu bmc --page=factory 25:00.0\n
Triggers a factory boot of the BMC image for the device with PCIe address 25:00.0.
# rsu fpga --page=user2 25:00.0\n
Triggers a reconfiguration of the FPGA (user2 page) for the device with PCIe address 25:00.0.
# rsu --force fpga --page=user2 25:00.0\n
Forces a reconfiguration of the FPGA (user2 page) for the device with PCIe address 25:00.0. Default behavior is to not perform the rsu operation if DPC (downstream port containment) is not supported and AER (advanced error reporting) is also not supported. Using --force changes this behavior to perform rsu operation regardless but may result in a surprise removal of pci devices which may cause the Linux kernel to panic.
# rsu fpga --page=factory 25:00.0\n
Triggers a factory reconfiguration of the FPGA for the device with PCIe address 25:00.0.
# rsu sdm --type=sr 25:00.0\n
Triggers Static Region key programming for the device with PCIE address 25:00.0.
# rsu fpgadefault --page=factory --fallback=user1,user2 25:00.0\n
Sets the FPGA boot sequence to factory with fallbacks user1, user2.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/","title":"Super Remote System Update User Guide","text":".. toctree::\n.. highlight:: c\n.. highlight:: console\n
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#overview","title":"Overview","text":"Intel Programmable Acceleration Card (PAC) devices are comprised of multiple processors and controllers that execute firmware. Maintaining and updating these firmware images manually is error-prone and does not scale well within the Data Center. The solution described here is derived with the following goals in mind:
- The ability to update one or more (possibly all) firwmare images with a single package.
- The ability to complete all firmware updates within a stipulated time window.
- The ability to update each PAC in the server, all servers in a Data Center, and multiple Data Centers remotely.
- The ability to remotely initiate download of the package and its installation with a single command per server instance.
- The ability to roll back firmware to a previous revision.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#implementation","title":"Implementation","text":"A single package containing firmware images for all programmable parts on a PAC is delivered as an RPM, eg opae-super-rsu-n3000.M.m.p-r.noarch.rpm. The RPM revision will sequentially increase with every update.
Installing or upgrading the RPM invokes the complete update of all programmable parts on all PAC boards in the system.
The standard RPM dependency framework ensures that correct versions of dependecy packages opae-intel-fpga-driver and fpga-tools-extra are installed on the system.
Rolling back is achieved by uninstalling the current version and re-installing a previous version of the RPM.
.. note::
Note: once Secure Update is deployed, roll back restrictions shall be implemented to prevent\nrollback attacks.\n
RPM management on remote systems is standard practice, requiring no new infrastructure/training.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#details","title":"Details","text":"The post-install hook of the opae-super-rsu-n3000 RPM is leveraged to call out to the super-rsu Python script to update all PAC boards. super-rsu uses the manifest file packaged within opae-super-rsu-n3000 to associate a firmware image with its version. Each of the firmware images contained in opae-super-rsu-n3000 is placed on the target system in /usr/share/opae/n3000.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#algorithm","title":"Algorithm","text":" - Acquire the current firmware versions of all programmable parts.
- For each programmable image, if the installed version of firmware does not equal the version provided in the RPM manifest file, then update the firmware image, and set image_updated to True.
- After all updates, if image_updated, then initiate a safe reboot of all boards in the system.
- After safe reboot, verify that the reported firmware versions match those of the RPM manifest. If they do not match, then RPM installation exits with a failing status.
- Run board self test. If the self test fails, then the RPM installation exits with a failing status.
- If all of the above checks is successful, then RPM installation exits with a success status.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#dependencies","title":"Dependencies","text":" - The standard Python package for the distro (version 2.7).
- The opae-intel-fpga-driver RPM. (version determined by opae-super-rsu-n3000)
- The opae-tools-extra RPM. (version determined by opae-super-rsu-n3000)
"},{"location":"sw/fpga_tools/userclk/userclk/","title":"userclk","text":""},{"location":"sw/fpga_tools/userclk/userclk/#synopsis","title":"SYNOPSIS","text":"userclk [-hv] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR] [-H <User clock high frequency>] -L <User clock low frequency>]
"},{"location":"sw/fpga_tools/userclk/userclk/#description","title":"DESCRIPTION","text":"userclk sets the frequency range for an AFU.
"},{"location":"sw/fpga_tools/userclk/userclk/#examples","title":"EXAMPLES","text":"./userclk -B 0x5e -H 400 -L 200
Sets AFU frequency.
"},{"location":"sw/fpga_tools/userclk/userclk/#options","title":"OPTIONS","text":"-v,--version
Prints version information and exits.
-S,--segment
FPGA segment number.
-B,--bus
FPGA Bus number.
-D,--device
FPGA Device number.
-F,--function
FPGA function number.
-H,--freq-high
User clock high frequency.
-L,--freq-low
User clock low frequency.
Date Intel Acceleration Stack Version Changes Made 2018.05.21 DCP 1.1 Beta (works with Quartus Prime Pro 17.1.1) Fixed typos."},{"location":"sw/fpga_tools/vabtool/vabtool/","title":"vabtool","text":""},{"location":"sw/fpga_tools/vabtool/vabtool/#synopsis","title":"SYNOPSIS","text":"vabtool [-r RETRIES] [-d] [-y] [-v] <ACTION>
Where ACTION is defined as one of the following:
vabtool sr_key_provision PCIE_ADDRESS SR_RKH_FILE FPGA_IMG_FILE vabtool sr_status PCIE_ADDRESS vabtool pr_key_provision PCIE_ADDRESS PR_AUTH_CERT_FILE PR_RKH_FILE vabtool pr_status PCIE_ADDRESS vabtool sr_key_cancel PCIE_ADDRESS SR_RKH_CANCEL_FILE vabtool sr_cancel_status PCIE_ADDRESS vabtool pr_key_cancel PCIE_ADDRESS PR_RKH_CANCEL_FILE vabtool pr_cancel_status PCIE_ADDRESS
"},{"location":"sw/fpga_tools/vabtool/vabtool/#description","title":"DESCRIPTION","text":"The vabtool command helps perform Vendor Authenticated Boot provisioning of Static Region and Partial Reconfiguration Region key hashes and helps perform SR and PR hash cancellation and status reporting.
"},{"location":"sw/fpga_tools/vabtool/vabtool/#options","title":"OPTIONS","text":"-r RETRIES, --retries RETRIES
Specifies the number of times a failed SR or PR key provision is to be\nretried. The default value for RETRIES is 3.\n
-d, --dry-run
Don't execute the actual fpgasupdate and rsu commands, but only print\nthe commands that would be executed during a normal run of the script.\n
-y, --yes
The tool will respond with an automatic Yes answer to all confirmation\nprompts posed by the sub-tools.\n
-v, --version
Display script version information and exit.\n
"},{"location":"sw/fpga_tools/vabtool/vabtool/#examples","title":"EXAMPLES","text":"sudo vabtool -y sr_key_provision 0000:bc:00.0 my_sr_rkh.bin my_fpga.bin sudo vabtool sr_status 0000:bc:00.0 sudo vabtool -y pr_key_provision 0000:bc:00.0 pr_auth_cert.bin my_pr_rkh.bin sudo vabtool pr_status 0000:bc:00.0 sudo vabtool sr_key_cancel 0000:bc:00.0 my_sr_rhk_cancel.bin sudo vabtool sr_cancel_status 0000:bc:00.0 sudo vabtool pr_key_cancel 0000:bc:00.0 my_pr_rhk_cancel.bin sudo vabtool pr_cancel_status 0000:bc:00.0
"},{"location":"sw/install_guide/installation_guide/","title":"OPAE Installation Guide","text":""},{"location":"sw/install_guide/installation_guide/#how-to-download-the-opae-sdk","title":"How to download the OPAE SDK","text":"OPAE SDK releases are available on GitHub. Source code for the OPAE DFL device driver for Linux is also available on GitHub.
"},{"location":"sw/install_guide/installation_guide/#install-the-fedora","title":"Install the Fedora","text":"Download the Fedora (x86_64 version) installation file in fedora, and install the Fedora in yourserver. You can choose Fedora Workstation or Fedora server.
"},{"location":"sw/install_guide/installation_guide/#build-the-kernel-and-dfl-drivers","title":"Build the kernel and DFL drivers","text":"For building the OPAE kernel and kernel driver, the kernel development environment is required. So before you build the kernel, you must install the required packages. Run the following commands:
$ sudo dnf install gcc gcc-c++ make kernel-headers kernel-devel elfutils-libelf-devel ncurses-devel openssl-devel bison flex\n
Download the OPAE upstream kernel tree from github, for example download from fpga-ofs-dev-5.15-lts branch.
$ git clone https://github.com/OPAE/linux-dfl.git -b fpga-ofs-dev-5.15-lts\n
Configure the kernel.
$ cd linux-dfl\n$ cp /boot/config-`uname -r` .config\n$ cat configs/dfl-config >> .config\n$ echo 'CONFIG_LOCALVERSION=\"-dfl\"' >> .config\n$ echo 'CONFIG_LOCALVERSION_AUTO=y' >> .config\n$ sed -i -r 's/CONFIG_SYSTEM_TRUSTED_KEYS=.*/CONFIG_SYSTEM_TRUSTED_KEYS=\"\"/' .config\n$ sed -i '/^CONFIG_DEBUG_INFO_BTF/ s/./#&/' .config\n$ echo 'CONFIG_DEBUG_ATOMIC_SLEEP=y' >> .config\n$ make olddefconfig\n
Compile and install the new kernel.
$ make -j $(nproc)\n$ sudo make modules_install -j $(nproc)\n$ sudo make install\n
Build linux DFL Kernel instructions please also refer to: https://github.com/OPAE/linux-dfl/wiki/Build-the-linux-dfl-kernel
When install finished, reboot your system. When the system login again, verify the kernel version is correct. For example:
[figo@localhost linux-dfl]$ uname -a\nLinux localhost.localdomain 5.15.lts-dfl-g73e16386cda0 #6 SMP Mon Jun 13 21:21:31 -04 2022 x86_64 x86_64 x86_64\n
And also you can check the OPAE dfl drivers have auto-loaded.
[figo@localhost linux-dfl]$ lsmod | grep fpga\nifpga_sec_mgr 20480 1 intel_m10_bmc_secure\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n[figo@localhost linux-dfl]$ lsmod | grep dfl\ndfl_eth_group 36864 0\ndfl_fme_region 20480 0\ndfl_emif 16384 0\ndfl_n3000_nios 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 1\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 7 dfl_pci,dfl_fme,dfl_fme_br,dfl_eth_group,dfl_n3000_nios,dfl_afu,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n
"},{"location":"sw/install_guide/installation_guide/#build-the-opae-sdk","title":"Build the OPAE-SDK","text":"Before you build the OPAE SDK, you must install the required packages. Run the following commands:
"},{"location":"sw/install_guide/installation_guide/#rocky-linux-85","title":"Rocky Linux 8.5","text":"# dnf install -y 'dnf-command(config-manager)'\n# dnf config-manager --set-enabled powertools\n# dnf install -y epel-release\n# dnf check-update\n# dnf upgrade -y\n# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel linuxptp numactl-devel\n# python3 -m pip install jsonschema virtualenv pyyaml\n
"},{"location":"sw/install_guide/installation_guide/#fedora","title":"Fedora","text":"# dnf check-update\n# dnf upgrade -y\n# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc g++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel libedit-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel cli11-devel spdlog-devel linuxptp numactl-devel\n# pip3 install jsonschema virtualenv pyyaml\n
"},{"location":"sw/install_guide/installation_guide/#ubuntu-2204","title":"Ubuntu 22.04","text":"# apt-get update\n# apt-get upgrade -y\n# apt-get install -y python3 python3-pip python3-dev git gcc g++ make cmake uuid-dev libjson-c-dev libhwloc-dev libtbb-dev libedit-dev libudev-dev linuxptp pandoc devscripts debhelper doxygen libnuma-dev\n# pip3 install jsonschema virtualenv pyyaml pybind11\n
"},{"location":"sw/install_guide/installation_guide/#rhel-82","title":"RHEL 8.2","text":"Register and enable Red Hat subscription to install any packages on the system.
# subscription-manager register --proxy=PROXY --username=USER --password=PASSWORD --auto-attach\n
Set the RHEL version and install packages. Set proxy name and port number.
# subscription-manager release --set=8.2 --proxy proxy-name.com:port number\n# subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\n# dnf upgrade -y\n# dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n# dnf install -y python3 python3-pip python3-devel gdb vim git gcc gcc-c++ make cmake libuuid-devel rpm-build systemd-devel nmap\n# dnf install -y python3-jsonschema json-c-devel tbb-devel rpmdevtools libcap-devel numactl-devel\n# dnf check-update || true\n# dnf install -y spdlog-devel cli11-devel python3-pyyaml python3-pybind11 hwloc-devel libedit-devel\n# python3 -m pip install --user jsonschema virtualenv pudb pyyaml\n
Install the latest version of cmake on top of the outdated cmake package from the package manager.
# cd cmake-3.25.1/\n# ./bootstrap --prefix=/usr\n# make\n# make install\n# which cmake\n/usr/bin/cmake\n
"},{"location":"sw/install_guide/installation_guide/#create-opae-sdk-packages","title":"Create opae-sdk packages","text":"Download the OPAE-SDK source code from github. For example, download from Master branch.
$ git clone https://github.com/OPAE/opae-sdk.git\n
Compile and build the OPAE-SDK RPMs (Fedora, Rocky, RHEL 8.2).
$ cd opae-sdk/packaging/opae/rpm\n$ ./create fedora\n
Note that if you find that your distribution has changed package names such that there is a conflict when building RPMs, you can install all of the build dependencies so that the SDK compiles and then build the RPMs in unrestricted mode:
$ cd opae-sdk/packaging/opae/rpm\n$ ./create unrestricted\n
After a successful compile, there are 3 rpm packages generated (Fedora, Rocky, RHEL8.2). For example:
opae-2.1.0-1.fc34.x86_64.rpm\nopae-devel-2.1.0-1.fc34.x86_64.rpm\nopae-extra-tools-2.1.0-1.fc34.x86_64.rpm\n
Compile and build the OPAE-SDK deb packages (Ubuntu 22.04).
$ cd opae-sdk/packaging/opae/deb\n$ ./create\n
After a successful compile, there are 3 deb packages generated (Ubuntu 22.04). For example:
opae_2.1.1-1_amd64.deb \nopae-devel_2.1.1-1_amd64.deb \nopae-extra-tools_2.1.1-1_amd64.deb\n
"},{"location":"sw/install_guide/installation_guide/#opae-sdk-installation-with-rpmdeb-packages","title":"OPAE SDK installation with rpm/deb packages","text":"The rpm packages generated in the previous step can be installed using these commands:
$ sudo dnf install ./*.rpm\n
The deb packages generated in the previous step can be installed using these commands:
$ sudo dpkg -i ./*.deb\n
When you installed the rpms, you can run fpgainfo command to check the FPGA FME infomation. For example:
[figo@localhost install_guide]$ fpgainfo fme\nBoard Management Controller, MAX10 NIOS FW version: D.2.1.24\nBoard Management Controller, MAX10 Build version: D.2.0.7\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:08:00.0\nDevice Id : 0x0B30\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x2300011001030F\nBitstream Version : 0.2.3\nPr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb\nBoot Page : user\n
To uninstall the OPAE rpms, you can use this commands
$ dnf list installed | grep opae\n$ sudo dnf remove opae*.x86_64\n
To uninstall the OPAE deb, you can use this commands
$ dpkg -l | grep opae\n$ dpkg -r opae-extra-tools:amd64\n$ dpkg -r opae-devel:amd64\n$ dpkg -r opae\n
"},{"location":"sw/install_guide/installation_guide/#fpga-device-access-permissions","title":"FPGA Device Access Permissions","text":"Access to FPGA accelerators and devices is controlled using file access permissions on the Intel\u00ae FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
$ sudo chmod a+rw /dev/dfl-port.0\n
"},{"location":"sw/install_guide/installation_guide/#memlock-limit","title":"Memlock limit","text":"Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
$ ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"sw/install_guide/installation_guide/#hugepage-settings","title":"Hugepage Settings","text":"Users need to configure system hugepages to reserve 2MB-hugepages or 1GB-hugepages. For example, the 'hello_fpga' sample requires several 2MB-hugepages. And the fpgadiag tool requires several 1GB-hugepages.
The command below reserves 20 2M-hugepages:
$ sudo sh -c 'echo 20 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages'\n
The command below reserves 4 1GB-hugepages:
$ sudo sh -c 'echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'\n
For x86_64 architecture processors, user can use following command to find out avaiable hugepage sizes:
$ grep pse /proc/cpuinfo | uniq\nflags : ... pse ...\n
If this commands returns a non-empty string, 2MB pages are supported.
$ grep pse /proc/cpuinfo | uniq\nflags : ... pdpe1gb ...\n
If this commands returns a non-empty string, 1GB pages are supported.
"},{"location":"sw/pyopae/python_bindings/","title":"OPAE Python Bindings","text":"OPAE (Open Programmable Acceleration Engine) now includes Python bindings for interacting with FPGA resources. The OPAE Python API is built on top of the OPAE C++ Core API and its object model. Because of this, developing OPAE applications in Python is very similar to developing OPAE applications in C++ which significantly reduces the learning curve required to adapt to the Python API. While the object model remains the same, some static factory functions in the OPAE C++ Core API have been moved to module level methods in the OPAE Python API with the exception of the properties class. The goal of the OPAE Python API is to enable fast prototyping, test automation, infrastructure managment, and an easy to use framework for FPGA resource interactions that don\u2019t rely on software algorithms with a high runtime complexity.
Currently, the only Python package that is part of OPAE is opae.fpga
"},{"location":"sw/pyopae/python_bindings/#implementation","title":"Implementation","text":"The OPAE Python API is implemented by creating a Python extension using pybind11 <http://pybind11.readthedocs.io/en/stable>. This extension is created by using the pybind11 API which relies mostly on macros and compile time introspection to define the module initialization point as well as type converters between OPAE C++ Core types and OPAE Python types.
"},{"location":"sw/pyopae/python_bindings/#benefits","title":"Benefits","text":"The major benefits of using pybind11 for developing the OPAE Python API include, but are not limited to, the following features of pybind11:
- Uses C++ 11 standard library although it can use C++ 14 or C++17.
- Automatic conversions of shared_ptr types
- Built-in support for numpy and Eigen numerical libraries
- Interoperable with the Python C API
"},{"location":"sw/pyopae/python_bindings/#runtime-requirements","title":"Runtime Requirements","text":"Because opae.fpga is built on top of the opae-cxx-core API, it does require that the runtime libraries for both opae-cxx-core and opae-c be installed on the system (as well as any other libraries they depend on). Those libraries can be installed using the opae-libs package (from either RPM or DEB format - depending on your Linux distribution).
"},{"location":"sw/pyopae/python_bindings/#installation","title":"Installation","text":""},{"location":"sw/pyopae/python_bindings/#python-wheels","title":"Python Wheels","text":"The preferred method of installation is to use a binary wheel package for your version of Python.
The following table lists example names for different Python versions and platforms.
| Python Version | Python ABI | Linux Platform | Package Name | |\u2014\u2014\u2014\u2014\u2014-|\u2014\u2014\u2014\u2014\u2014\u2013|\u2014\u2014\u2014\u2014\u2014-|\u2014\u2014\u2014\u2014\u2013| | 2.7 | CPython w/ UCS4 | x86_64 | opae.fpga.-cp27-cp27mu-linux_x86_64.whl | | 3.4 | CPython w/ UCS4 | x86_64 | opae.fpga.-cp34-cp34mu-linux_x86_64.whl | | 3.6 | CPython w/ UCS4 | x86_64 | opae.fpga.-cp36-cp36mu-linux_x86_64.whl |
opae.fpga is currently not available in the Python Package Index but once it does become available, one should be able to install using pip by simply typing the following:
> pip install --user opae.fpga\n
"},{"location":"sw/pyopae/python_bindings/#installing-from-source","title":"Installing From Source","text":"In addition to the runtime libraries mentioned above, installing from source does require that the OPAE header files be installed as well as those header files for pybind11. The former can be installed with the opae-devel package and the latter can be installed by installing pybind11 Python module.
"},{"location":"sw/pyopae/python_bindings/#example-installation","title":"Example Installation","text":"The following example shows how to build from source by installing the prerequisites before running the setup.py file.
>sudo yum install opae-libs-<release>.x86_64.rpm\n>sudo yum install opae-devel-<release>.x86_64.rpm\n>pip install --user pybind11\n>pip install --user opae.fpga-<release>.tar.gz\n
NOTE: The pip examples above use the --user flag to avoid requiring root permissions. Those packages will be installed in the user\u2019s site-packages directory found in the user\u2019s .local directory.
"},{"location":"sw/pyopae/python_bindings/#example-scripts","title":"Example Scripts","text":"The following example is an implementation of the sample, hello_fpga.c, which is designed to configure the NLB0 diagnostic accelerator for a simple loopback.
import time\nfrom opae import fpga\n\nNLB0 = \"d8424dc4-a4a3-c413-f89e-433683f9040b\"\nCTL = 0x138\nCFG = 0x140\nNUM_LINES = 0x130\nSRC_ADDR = 0x0120\nDST_ADDR = 0x0128\nDSM_ADDR = 0x0110\nDSM_STATUS = 0x40\n\ndef cl_align(addr):\n return addr >> 6\n\ntokens = fpga.enumerate(type=fpga.ACCELERATOR, guid=NLB0)\nassert tokens, \"Could not enumerate accelerator: {}\".format(NlB0)\n\nwith fpga.open(tokens[0], fpga.OPEN_SHARED) as handle:\n src = fpga.allocate_shared_buffer(handle, 4096)\n dst = fpga.allocate_shared_buffer(handle, 4096)\n dsm = fpga.allocate_shared_buffer(handle, 4096)\n handle.write_csr32(CTL, 0)\n handle.write_csr32(CTL, 1)\n handle.write_csr64(DSM_ADDR, dsm.io_address())\n handle.write_csr64(SRC_ADDR, cl_align(src.io_address())) # cacheline-aligned\n handle.write_csr64(DST_ADDR, cl_align(dst.io_address())) # cacheline-aligned\n handle.write_csr32(CFG, 0x42000)\n handle.write_csr32(NUM_LINES, 4096/64)\n handle.write_csr32(CTL, 3)\n while dsm[DSM_STATUS] & 0x1 == 0:\n time.sleep(0.001)\n handle.write_csr32(CTL, 7)\n
This example shows how one might reprogram (Partial Reconfiguration) an accelerator on a given bus, 0x5e, using a bitstream file, m0.gbs.
from opae import fpga\n\nBUS = 0x5e\nGBS = 'm0.gbs'\ntokens = fpga.enumerate(type=fpga.DEVICE, bus=BUS)\nassert tokens, \"Could not enumerate device on bus: {}\".format(BUS)\nwith open(GBS, 'rb') as fd, fpga.open(tokens[0]) as device:\n device.reconfigure(0, fd)\n
"},{"location":"sw/tod/tod/","title":"Time of Day (ToD)","text":""},{"location":"sw/tod/tod/#synopsis","title":"SYNOPSIS","text":"The Intel FPGA ToD driver in the kernel space exposes ToD IP as PHC (PTP Hardware Clock) device to the Linux PTP (Precision Time Protocol) stack to synchronize the system clock to its ToD information. The phc2sys utility of Linux PTP stack is used to access ToD information and synchronize the system clock.
Install the Linux PTP utilities:
# sudo yum install linuxptp\n
phc_ctl and phc2sys utilities (linuxptp package) are used to control the PHC device and synchronize the system clock to its ToD information.
phc_ctl: directly controls PHC device clock.
Usage: phc_ctl [options] <device> -- [command]\n device ethernet or ptp clock device\nOptions:\n -l [num] set the logging level to 'num'\n -q do not print messages to the syslog\n -Q do not print messages to stdout\n -v prints the software version and exits\n -h prints this message and exits\nCommands:\n specify commands with arguments. Can specify multiple commands to be executed in order. \n Seconds are read as double precision floating point values.\n set [seconds] set PHC time (defaults to time on CLOCK_REALTIME)\n get get PHC time\n adj <seconds> adjust PHC time by offset\n freq [ppb] adjust PHC frequency (default returns current offset)\n cmp compare PHC offset to CLOCK_REALTIME\n caps display device capabilities (default if no command given)\n wait <seconds> pause between commands\n This command may be useful for sanity checking whether the PHC clock is\n running as expected.\n The arguments specified in seconds are read as double precision floating point\n values, and will scale to nanoseconds. This means providing a value of 5.5 means 5\n and one half seconds. This allows specifying fairly precise values for time.\n
phc2sys: synchronize two clocks.
Usage: phc2sys [options]\nAutomatic configuration:\n -a turn on autoconfiguration\n -r synchronize system (realtime) clock\n repeat -r to consider it also as a time source\nManual configuration:\n -c [dev|name] slave clock (CLOCK_REALTIME)\n -d [dev] master PPS device\n -s [dev|name] master clock\n -O [offset] slave-master time offset (0)\n -w wait for ptp4l\nOptions:\n -f [file] configuration file\n -E [pi|linreg] clock servo (pi)\n -P [kp] proportional constant (0.7)\n -I [ki] integration constant (0.3)\n -S [step] step threshold (disabled)\n -F [step] step threshold only on start (0.00002)\n -R [rate] slave clock update rate in HZ (1.0)\n -N [num] number of master clock readings per update (5)\n -L [limit] sanity frequency limit in ppb (200000000)\n -M [num] NTP SHM segment number (0)\n -u [num] number of clock updates in summary stats (0)\n -n [num] domain number (0)\n -x apply leap seconds by servo instead of kernel\n -z [path] server address for UDS (/var/run/ptp4l)\n -l [num] set the logging level to 'num' (6)\n -t [tag] add tag to log messages\n -m print messages to stdout\n -q do not print messages to the syslog\n -v prints the software version and exits\n -h prints this message and exits\n
"},{"location":"sw/tod/tod/#description","title":"DESCRIPTION","text":"The phc2sys utility is used to synchronize the system clock to the PTP Hardware Clock (PHC) or ToD clock. The phc_ctl utility is used to directly control PHC clock device.
"},{"location":"sw/tod/tod/#configuring-the-ptp-service","title":"Configuring the PTP service","text":" - Install the linuxptp package:
# sudo yum install linuxptp\n
- Check PTP device is created successfully by the ToD driver.
ToD driver registering as PHC device (clock_name: dfl_tod) to the Linux PTP stack and exposing to the Linux kernel to synchronize the system clock to its ToD information.
# cat /sys/class/ptp/ptp0/clock_name\ndfl_tod\n
- Configure phc2sys service on a system:
The phc2sys service is configured in the /etc/sysconfig/phc2sys configuration file. Define start-up option for phc2sys daemon in /etc/sysconfig/phc2sys. The master clock is /dev/ptp0 device and the slave clock is system clock/CLOCK_REALTIME:
OPTIONS=\"-s /dev/ptp0 -c CLOCK_REALTIME -r -O 0 -R 16\"\n
-
Start phc2sys service:
# service phc2sys start\n
-
Stop phc2sys service:
# service phc2sys stop\n
"},{"location":"sw/tod/tod/#examples","title":"Examples","text":""},{"location":"sw/tod/tod/#using-phc_ctl-utility","title":"using phc_ctl utility","text":"Read the current clock time from the PHC clock device:
# sudo phc_ctl /dev/ptp0 get\n
Set the PHC clock time to CLOCK_REALTIME:
# sudo phc_ctl /dev/ptp0 set\n
Set PHC clock time to 0:
# sudo phc_ctl /dev/ptp0 set 0.0\n
Set PHC clock time to 0 and wait for 10 sec and read the clock time:
# sudo phc_ctl /dev/ptp0 set 0.0 wait 10.0 get\n
Set and compare PHC clock time to CLOCK_REALTIME:
# sudo phc_ctl /dev/ptp0 set cmp\n
Read the PHC device capabilities:
# sudo phc_ctl /dev/ptp0 caps\n
"},{"location":"sw/tod/tod/#using-phc2sys-utility","title":"using phc2sys utility","text":"To synchronize the system clock to the PHC clock:
# sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME -r -O 0 -R 16 -m\nphc2sys[7896.789]: CLOCK_REALTIME phc offset -1259509 s0 freq -31462 delay 1338\nphc2sys[7896.852]: CLOCK_REALTIME phc offset -1261498 s1 freq -63144 delay 1328\nphc2sys[7896.914]: CLOCK_REALTIME phc offset -15 s2 freq -63159 delay 1328\nphc2sys[7896.977]: CLOCK_REALTIME phc offset -19 s2 freq -63167 delay 1327\nphc2sys[7897.039]: CLOCK_REALTIME phc offset -35 s2 freq -63189 delay 1328\nphc2sys[7897.102]: CLOCK_REALTIME phc offset -37 s2 freq -63201 delay 1331\nphc2sys[7897.165]: CLOCK_REALTIME phc offset -30 s2 freq -63205 delay 1328\nphc2sys[7897.227]: CLOCK_REALTIME phc offset -50 s2 freq -63234 delay 1331\nphc2sys[7897.290]: CLOCK_REALTIME phc offset -50 s2 freq -63249 delay 1329\nphc2sys[7897.353]: CLOCK_REALTIME phc offset -62 s2 freq -63276 delay 1334\nphc2sys[7897.415]: CLOCK_REALTIME phc offset -53 s2 freq -63286 delay 1335\nphc2sys[7897.478]: CLOCK_REALTIME phc offset -46 s2 freq -63295 delay 1325\nphc2sys[7897.541]: CLOCK_REALTIME phc offset -57 s2 freq -63320 delay 1334\nphc2sys[7897.603]: CLOCK_REALTIME phc offset -39 s2 freq -63319 delay 1327\nphc2sys[7897.666]: CLOCK_REALTIME phc offset -31 s2 freq -63323 delay 1328\nphc2sys[7897.728]: CLOCK_REALTIME phc offset -48 s2 freq -63349 delay 1327\nphc2sys[7897.791]: CLOCK_REALTIME phc offset -42 s2 freq -63357 delay 1323\nphc2sys[7897.854]: CLOCK_REALTIME phc offset -41 s2 freq -63369 delay 1324\nphc2sys[7897.916]: CLOCK_REALTIME phc offset -44 s2 freq -63384 delay 1328\nphc2sys[7897.979]: CLOCK_REALTIME phc offset -13 s2 freq -63366 delay 1327\nphc2sys[7898.042]: CLOCK_REALTIME phc offset -16 s2 freq -63373 delay 1327\nphc2sys[7898.104]: CLOCK_REALTIME phc offset -19 s2 freq -63381 delay 1328\nphc2sys[7898.167]: CLOCK_REALTIME phc offset -16 s2 freq -63384 delay 1327\nphc2sys[7898.229]: CLOCK_REALTIME phc offset 3 s2 freq -63370 delay 1327\nphc2sys[7898.292]: CLOCK_REALTIME phc offset 16 s2 freq -63356 delay 1325\nphc2sys[7898.355]: CLOCK_REALTIME phc offset 10 s2 freq -63357 delay 1319\nphc2sys[7898.417]: CLOCK_REALTIME phc offset 23 s2 freq -63341 delay 1327\nphc2sys[7898.480]: CLOCK_REALTIME phc offset 13 s2 freq -63344 delay 1335\nphc2sys[7898.543]: CLOCK_REALTIME phc offset 23 s2 freq -63330 delay 1323\nphc2sys[7898.605]: CLOCK_REALTIME phc offset 29 s2 freq -63317 delay 1312\nphc2sys[7898.668]: CLOCK_REALTIME phc offset 22 s2 freq -63315 delay 1324\nphc2sys[7898.730]: CLOCK_REALTIME phc offset 42 s2 freq -63289 delay 1325\nphc2sys[7898.793]: CLOCK_REALTIME phc offset 29 s2 freq -63289 delay 1333\nphc2sys[7898.856]: CLOCK_REALTIME phc offset 34 s2 freq -63276 delay 1327\nphc2sys[7898.918]: CLOCK_REALTIME phc offset 21 s2 freq -63278 delay 1331\nphc2sys[7898.981]: CLOCK_REALTIME phc offset 22 s2 freq -63271 delay 1335\nphc2sys[7899.044]: CLOCK_REALTIME phc offset 30 s2 freq -63256 delay 1327\nphc2sys[7899.106]: CLOCK_REALTIME phc offset 30 s2 freq -63247 delay 1328\nphc2sys[7899.169]: CLOCK_REALTIME phc offset 37 s2 freq -63231 delay 1333\nphc2sys[7899.232]: CLOCK_REALTIME phc offset 29 s2 freq -63228 delay 1331\nphc2sys[7899.294]: CLOCK_REALTIME phc offset 24 s2 freq -63225 delay 1330\n
"},{"location":"opae-code/annotated/","title":"Class List","text":"Here are the classes, structs, unions and interfaces with brief descriptions:
- struct _fpga_token_header Internal token type header.
- struct _opae_hash_map Hash map object.
- struct _opae_hash_map_item List link item.
- struct cache_line
- struct config
- struct fpga_error_info
- struct fpga_metric Metric struct.
- struct fpga_metric_info Metric info struct.
- struct fpga_version Semantic version.
- struct mem_alloc
- struct mem_link Provides an API for allocating/freeing a logical address space.
- struct metric_threshold
- union metric_value Metric value union.
- namespace opae
- namespace fpga
- namespace types
- class busy busy exception
- namespace detail
- class error An error object represents an error register for a resource.
- class event Wraps fpga event routines in OPAE C.
- struct type_t C++ struct that is interchangeable with fpga_event_type enum.
- class except Generic OPAE exception.
- class exception exception exception
- struct guid_t Representation of the guid member of a properties object.
- class handle An allocated accelerator resource.
- class invalid_param invalid_param exception
- class no_access no_access exception
- class no_daemon no_daemon exception
- class no_driver no_driver exception
- class no_memory no_memory exception
- class not_found not_found exception
- class not_supported not_supported exception
- class properties Wraps an OPAE fpga_properties object.
- struct pvalue Wraps OPAE properties defined in the OPAE C API by associating an
fpga_properties reference with the getters and setters defined for a property. - class reconf_error reconf_error exception
- class shared_buffer Host/AFU shared memory blocks.
- class src_location Identify a particular line in a source file.
- class sysobject Wraps the OPAE fpga_object primitive.
- class token Wraps the OPAE fpga_token primitive.
- class version
- struct opae_uio OPAE UIO device abstraction.
- struct opae_uio_device_region MMIO region info.
- struct opae_vfio OPAE VFIO device abstraction.
- struct opae_vfio_buffer System DMA buffer.
- struct opae_vfio_device VFIO device.
- struct opae_vfio_device_irq Interrupt info.
- struct opae_vfio_device_region MMIO region info.
- struct opae_vfio_group VFIO group.
- struct opae_vfio_iova_range IO Virtual Address Range.
- struct opae_vfio_sparse_info MMIO sparse-mappable region info.
- struct ras_inject_error
- namespace std
- struct threshold Threshold struct.
"},{"location":"opae-code/files/","title":"File List","text":"Here is a list of all files with brief descriptions:
- dir docs
- dir sw
- dir include
- dir opae
- file access.h Functions to acquire, release, and reset OPAE FPGA resources.
- file buffer.h Functions for allocating and sharing system memory with an FPGA accelerator.
- dir cxx
- file core.h
- dir core
- file errors.h
- file events.h
- file except.h
- file handle.h
- file properties.h
- file pvalue.h
- file shared_buffer.h
- file sysobject.h
- file token.h
- file version.h
- file enum.h APIs for resource enumeration and managing tokens.
- file error.h Functions for reading and clearing errors in resources.
- file event.h Functions for registering events and managing the lifecycle for
fpga_event_handle s. - file fpga.h FPGA API.
- file hash_map.h A general-purpose hybrid array/list hash map implementation.
- file init.h Initialization routine.
- file log.h
- file manage.h Functions for managing FPGA configurations.
- file mem_alloc.h
- file metrics.h Functions for Discover/ Enumerates metrics and retrieves values.
- file mmio.h Functions for mapping and accessing MMIO space.
- file properties.h Functions for examining and manipulating
fpga_properties objects. - file sysobject.h Functions to read/write from system objects.
- file types.h Type definitions for FPGA API.
- file types_enum.h Definitions of enumerated types for the OPAE C API.
- file uio.h APIs to manage a PCIe device via UIO.
- file umsg.h FPGA UMsg API.
- file userclk.h Functions for setting and get afu user clock.
- file utils.h Utility functions and macros for the FPGA API.
- file version.h
- file vfio.h APIs to manage a PCIe device via vfio-pci.
- dir samples
- dir hello_events
- file hello_events.c A code sample of using OPAE event API.
- dir hello_fpga
- file hello_fpga.c A code sample illustrates the basic usage of the OPAE C API.
"},{"location":"opae-code/struct__fpga__token__header/","title":"Struct _fpga_token_header","text":"ClassList > _fpga_token_header
Internal token type header. More...
#include <types.h>
"},{"location":"opae-code/struct__fpga__token__header/#public-attributes","title":"Public Attributes","text":"Type Name uint8_t bus uint8_t device uint16_t device_id uint8_t function fpga_guid guid fpga_interface interface uint64_t magic uint64_t object_id fpga_objtype objtype uint16_t segment uint16_t subsystem_device_id uint16_t subsystem_vendor_id uint16_t vendor_id"},{"location":"opae-code/struct__fpga__token__header/#detailed-description","title":"Detailed Description","text":"Each plugin (dfl: libxfpga.so, vfio: libopae-v.so) implements its own proprietary token type. This header must appear at offset zero within that structure.
eg, see lib/plugins/xfpga/types_int.h:struct _fpga_token and lib/plugins/vfio/opae_vfio.h:struct _vfio_token.
"},{"location":"opae-code/struct__fpga__token__header/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/struct__fpga__token__header/#variable-bus","title":"variable bus","text":"uint8_t _fpga_token_header::bus;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-device","title":"variable device","text":"uint8_t _fpga_token_header::device;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-device_id","title":"variable device_id","text":"uint16_t _fpga_token_header::device_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-function","title":"variable function","text":"uint8_t _fpga_token_header::function;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-guid","title":"variable guid","text":"fpga_guid _fpga_token_header::guid;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-interface","title":"variable interface","text":"fpga_interface _fpga_token_header::interface;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-magic","title":"variable magic","text":"uint64_t _fpga_token_header::magic;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-object_id","title":"variable object_id","text":"uint64_t _fpga_token_header::object_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-objtype","title":"variable objtype","text":"fpga_objtype _fpga_token_header::objtype;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-segment","title":"variable segment","text":"uint16_t _fpga_token_header::segment;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-subsystem_device_id","title":"variable subsystem_device_id","text":"uint16_t _fpga_token_header::subsystem_device_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-subsystem_vendor_id","title":"variable subsystem_vendor_id","text":"uint16_t _fpga_token_header::subsystem_vendor_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-vendor_id","title":"variable vendor_id","text":"uint16_t _fpga_token_header::vendor_id;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/struct__opae__hash__map/","title":"Struct _opae_hash_map","text":"ClassList > _opae_hash_map
Hash map object. More...
#include <hash_map.h>
"},{"location":"opae-code/struct__opae__hash__map/#public-attributes","title":"Public Attributes","text":"Type Name opae_hash_map_item ** buckets void * cleanup_context Optional second parameter to key_cleanup and value_cleanup. int flags uint32_t hash_seed void(* key_cleanup (optional) int(* key_compare (required) uint32_t(* key_hash uint32_t num_buckets void(* value_cleanup (optional)"},{"location":"opae-code/struct__opae__hash__map/#detailed-description","title":"Detailed Description","text":"This structure defines the internals of the hash map. Each of the parameters supplied to opae_hash_map_init() is stored in the structure. All parameters are required, except key_cleanup and value_cleanup, which may optionally be NULL.
"},{"location":"opae-code/struct__opae__hash__map/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/struct__opae__hash__map/#variable-buckets","title":"variable buckets","text":"opae_hash_map_item** _opae_hash_map::buckets;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-cleanup_context","title":"variable cleanup_context","text":"void* _opae_hash_map::cleanup_context;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-flags","title":"variable flags","text":"int _opae_hash_map::flags;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-hash_seed","title":"variable hash_seed","text":"uint32_t _opae_hash_map::hash_seed;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-key_cleanup","title":"variable key_cleanup","text":"void(* _opae_hash_map::key_cleanup) (void *key, void *context);\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-key_compare","title":"variable key_compare","text":"int(* _opae_hash_map::key_compare) (void *keya, void *keyb);\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-key_hash","title":"variable key_hash","text":"uint32_t(* _opae_hash_map::key_hash) (uint32_t num_buckets, uint32_t hash_seed, void *key);\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-num_buckets","title":"variable num_buckets","text":"uint32_t _opae_hash_map::num_buckets;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-value_cleanup","title":"variable value_cleanup","text":"void(* _opae_hash_map::value_cleanup) (void *value, void *context);\n
The documentation for this class was generated from the following file docs/sw/include/opae/hash_map.h
"},{"location":"opae-code/struct__opae__hash__map__item/","title":"Struct _opae_hash_map_item","text":"ClassList > _opae_hash_map_item
List link item. More...
#include <hash_map.h>
"},{"location":"opae-code/struct__opae__hash__map__item/#public-attributes","title":"Public Attributes","text":"Type Name void * key struct _opae_hash_map_item * next void * value"},{"location":"opae-code/struct__opae__hash__map__item/#detailed-description","title":"Detailed Description","text":"This structure provides the association between key and value. When the supplied hash function for keys A and B returns the same bucket index, both A and B can co-exist on the same list rooted at the bucket index.
"},{"location":"opae-code/struct__opae__hash__map__item/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/struct__opae__hash__map__item/#variable-key","title":"variable key","text":"void* _opae_hash_map_item::key;\n
"},{"location":"opae-code/struct__opae__hash__map__item/#variable-next","title":"variable next","text":"struct _opae_hash_map_item* _opae_hash_map_item::next;\n
"},{"location":"opae-code/struct__opae__hash__map__item/#variable-value","title":"variable value","text":"void* _opae_hash_map_item::value;\n
The documentation for this class was generated from the following file docs/sw/include/opae/hash_map.h
"},{"location":"opae-code/structcache__line/","title":"Struct cache_line","text":"ClassList > cache_line
"},{"location":"opae-code/structcache__line/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t uint"},{"location":"opae-code/structcache__line/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structcache__line/#variable-uint","title":"variable uint","text":"uint32_t cache_line::uint[16];\n
The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/hello_fpga.c
"},{"location":"opae-code/structconfig/","title":"Struct config","text":"ClassList > config
"},{"location":"opae-code/structconfig/#public-attributes","title":"Public Attributes","text":"Type Name int open_flags int run_n3000"},{"location":"opae-code/structconfig/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structconfig/#variable-open_flags","title":"variable open_flags","text":"int config::open_flags;\n
"},{"location":"opae-code/structconfig/#variable-run_n3000","title":"variable run_n3000","text":"int config::run_n3000;\n
The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/hello_fpga.c
"},{"location":"opae-code/structfpga__error__info/","title":"Struct fpga_error_info","text":"ClassList > fpga_error_info
#include <types.h>
"},{"location":"opae-code/structfpga__error__info/#public-attributes","title":"Public Attributes","text":"Type Name bool can_clear name of the error char name"},{"location":"opae-code/structfpga__error__info/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__error__info/#variable-can_clear","title":"variable can_clear","text":"bool fpga_error_info::can_clear;\n
"},{"location":"opae-code/structfpga__error__info/#variable-name","title":"variable name","text":"char fpga_error_info::name[FPGA_ERROR_NAME_MAX];\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structfpga__metric/","title":"Struct fpga_metric","text":"ClassList > fpga_metric
Metric struct.
#include <types.h>
"},{"location":"opae-code/structfpga__metric/#public-attributes","title":"Public Attributes","text":"Type Name bool isvalid uint64_t metric_num metric_value value"},{"location":"opae-code/structfpga__metric/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__metric/#variable-isvalid","title":"variable isvalid","text":"bool fpga_metric::isvalid;\n
"},{"location":"opae-code/structfpga__metric/#variable-metric_num","title":"variable metric_num","text":"uint64_t fpga_metric::metric_num;\n
"},{"location":"opae-code/structfpga__metric/#variable-value","title":"variable value","text":"metric_value fpga_metric::value;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structfpga__metric__info/","title":"Struct fpga_metric_info","text":"ClassList > fpga_metric_info
Metric info struct.
#include <types.h>
"},{"location":"opae-code/structfpga__metric__info/#public-attributes","title":"Public Attributes","text":"Type Name char group_name enum fpga_metric_datatype metric_datatype fpga_guid metric_guid char metric_name uint64_t metric_num enum fpga_metric_type metric_type char metric_units char qualifier_name"},{"location":"opae-code/structfpga__metric__info/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__metric__info/#variable-group_name","title":"variable group_name","text":"char fpga_metric_info::group_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_datatype","title":"variable metric_datatype","text":"enum fpga_metric_datatype fpga_metric_info::metric_datatype;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_guid","title":"variable metric_guid","text":"fpga_guid fpga_metric_info::metric_guid;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_name","title":"variable metric_name","text":"char fpga_metric_info::metric_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_num","title":"variable metric_num","text":"uint64_t fpga_metric_info::metric_num;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_type","title":"variable metric_type","text":"enum fpga_metric_type fpga_metric_info::metric_type;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_units","title":"variable metric_units","text":"char fpga_metric_info::metric_units[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structfpga__metric__info/#variable-qualifier_name","title":"variable qualifier_name","text":"char fpga_metric_info::qualifier_name[FPGA_METRIC_STR_SIZE];\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structfpga__version/","title":"Struct fpga_version","text":"ClassList > fpga_version
Semantic version. More...
#include <types.h>
"},{"location":"opae-code/structfpga__version/#public-attributes","title":"Public Attributes","text":"Type Name uint8_t major Major version. uint8_t minor Minor version. uint16_t patch Revision or patchlevel."},{"location":"opae-code/structfpga__version/#detailed-description","title":"Detailed Description","text":"Data structure for expressing version identifiers following the semantic versioning scheme. Used in various properties for tracking component versions.
"},{"location":"opae-code/structfpga__version/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__version/#variable-major","title":"variable major","text":"uint8_t fpga_version::major;\n
"},{"location":"opae-code/structfpga__version/#variable-minor","title":"variable minor","text":"uint8_t fpga_version::minor;\n
"},{"location":"opae-code/structfpga__version/#variable-patch","title":"variable patch","text":"uint16_t fpga_version::patch;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structmem__alloc/","title":"Struct mem_alloc","text":"ClassList > mem_alloc
#include <mem_alloc.h>
"},{"location":"opae-code/structmem__alloc/#public-attributes","title":"Public Attributes","text":"Type Name struct mem_link allocated struct mem_link free"},{"location":"opae-code/structmem__alloc/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structmem__alloc/#variable-allocated","title":"variable allocated","text":"struct mem_link mem_alloc::allocated;\n
"},{"location":"opae-code/structmem__alloc/#variable-free","title":"variable free","text":"struct mem_link mem_alloc::free;\n
The documentation for this class was generated from the following file docs/sw/include/opae/mem_alloc.h
"},{"location":"opae-code/structmem__link/","title":"Struct mem_link","text":"ClassList > mem_link
Provides an API for allocating/freeing a logical address space. More...
#include <mem_alloc.h>
"},{"location":"opae-code/structmem__link/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t address struct mem_link * next struct mem_link * prev uint64_t size"},{"location":"opae-code/structmem__link/#detailed-description","title":"Detailed Description","text":"There is no interaction with any OS memory allocation infrastructure, whether malloc, mmap, etc. The \"address ranges\" tracked by this allocator are arbitrary 64-bit integers. The allocator simply provides the bookeeping logic that ensures that a unique address with the appropriate size is returned for each allocation request, and that an allocation can be freed, ie released back to the available pool of logical address space for future allocations. The memory backing the allocator's internal data structures is managed by malloc()/free().
"},{"location":"opae-code/structmem__link/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structmem__link/#variable-address","title":"variable address","text":"uint64_t mem_link::address;\n
"},{"location":"opae-code/structmem__link/#variable-next","title":"variable next","text":"struct mem_link* mem_link::next;\n
"},{"location":"opae-code/structmem__link/#variable-prev","title":"variable prev","text":"struct mem_link* mem_link::prev;\n
"},{"location":"opae-code/structmem__link/#variable-size","title":"variable size","text":"uint64_t mem_link::size;\n
The documentation for this class was generated from the following file docs/sw/include/opae/mem_alloc.h
"},{"location":"opae-code/structmetric__threshold/","title":"Struct metric_threshold","text":"ClassList > metric_threshold
#include <types.h>
"},{"location":"opae-code/structmetric__threshold/#public-attributes","title":"Public Attributes","text":"Type Name threshold hysteresis threshold lower_c_threshold threshold lower_nc_threshold threshold lower_nr_threshold char metric_name threshold upper_c_threshold threshold upper_nc_threshold threshold upper_nr_threshold"},{"location":"opae-code/structmetric__threshold/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structmetric__threshold/#variable-hysteresis","title":"variable hysteresis","text":"threshold metric_threshold::hysteresis;\n
"},{"location":"opae-code/structmetric__threshold/#variable-lower_c_threshold","title":"variable lower_c_threshold","text":"threshold metric_threshold::lower_c_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-lower_nc_threshold","title":"variable lower_nc_threshold","text":"threshold metric_threshold::lower_nc_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-lower_nr_threshold","title":"variable lower_nr_threshold","text":"threshold metric_threshold::lower_nr_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-metric_name","title":"variable metric_name","text":"char metric_threshold::metric_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structmetric__threshold/#variable-upper_c_threshold","title":"variable upper_c_threshold","text":"threshold metric_threshold::upper_c_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-upper_nc_threshold","title":"variable upper_nc_threshold","text":"threshold metric_threshold::upper_nc_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-upper_nr_threshold","title":"variable upper_nr_threshold","text":"threshold metric_threshold::upper_nr_threshold;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/unionmetric__value/","title":"Union metric_value","text":"ClassList > metric_value
Metric value union.
#include <types.h>
"},{"location":"opae-code/unionmetric__value/#public-attributes","title":"Public Attributes","text":"Type Name bool bvalue double dvalue float fvalue uint64_t ivalue"},{"location":"opae-code/unionmetric__value/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/unionmetric__value/#variable-bvalue","title":"variable bvalue","text":"bool metric_value::bvalue;\n
"},{"location":"opae-code/unionmetric__value/#variable-dvalue","title":"variable dvalue","text":"double metric_value::dvalue;\n
"},{"location":"opae-code/unionmetric__value/#variable-fvalue","title":"variable fvalue","text":"float metric_value::fvalue;\n
"},{"location":"opae-code/unionmetric__value/#variable-ivalue","title":"variable ivalue","text":"uint64_t metric_value::ivalue;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/namespaceopae/","title":"Namespace opae","text":"Namespace List > opae
"},{"location":"opae-code/namespaceopae/#namespaces","title":"Namespaces","text":"Type Name namespace fpga The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/namespaceopae_1_1fpga/","title":"Namespace opae::fpga","text":"Namespace List > opae > fpga
"},{"location":"opae-code/namespaceopae_1_1fpga/#namespaces","title":"Namespaces","text":"Type Name namespace types The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types/","title":"Namespace opae::fpga::types","text":"Namespace List > opae > fpga > types
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types/#namespaces","title":"Namespaces","text":"Type Name namespace detail"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types/#classes","title":"Classes","text":"Type Name class busy busy exception class error An error object represents an error register for a resource. class event Wraps fpga event routines in OPAE C. class except Generic OPAE exception. class exception exception exception struct guid_t Representation of the guid member of a properties object. class handle An allocated accelerator resource. class invalid_param invalid_param exception class no_access no_access exception class no_daemon no_daemon exception class no_driver no_driver exception class no_memory no_memory exception class not_found not_found exception class not_supported not_supported exception class properties Wraps an OPAE fpga_properties object. struct pvalue <typename T>Wraps OPAE properties defined in the OPAE C API by associating an fpga_properties reference with the getters and setters defined for a property. class reconf_error reconf_error exception class shared_buffer Host/AFU shared memory blocks. class src_location Identify a particular line in a source file. class sysobject Wraps the OPAE fpga_object primitive. class token Wraps the OPAE fpga_token primitive. class version The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/","title":"Class opae::fpga::types::busy","text":"ClassList > opae > fpga > types > busy
busy exception More...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-functions","title":"Public Functions","text":"Type Name busy (src_location loc) noexceptbusy constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#detailed-description","title":"Detailed Description","text":"busy tracks the source line of origin for exceptions thrown when the error code FPGA_BUSY is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#function-busy","title":"function busy","text":"busy constructor
inline opae::fpga::types::busy::busy (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/","title":"Namespace opae::fpga::types::detail","text":"Namespace List > opae > fpga > types > detail
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-types","title":"Public Types","text":"Type Name typedef bool(* exception_fn typedef function pointer that returns bool if result is FPGA_OK"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-attributes","title":"Public Static Attributes","text":"Type Name exception_fn opae_exceptions = = { is_ok<opae::fpga::types::invalid_param>, is_ok<opae::fpga::types::busy>, is_ok<opae::fpga::types::exception>, is_ok<opae::fpga::types::not_found>, is_ok<opae::fpga::types::no_memory>, is_ok<opae::fpga::types::not_supported>, is_ok<opae::fpga::types::no_driver>, is_ok<opae::fpga::types::no_daemon>, is_ok<opae::fpga::types::no_access>, is_ok<opae::fpga::types::reconf_error>}"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-functions","title":"Public Functions","text":"Type Name constexpr bool is_ok (fpga_result result, const opae::fpga::types::src_location & loc) is_ok is a template function that throws an excpetion of its template argument type if the result code is not FPGA_OK."},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-functions","title":"Public Static Functions","text":"Type Name void assert_fpga_ok (fpga_result result, const opae::fpga::types::src_location & loc)"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#typedef-exception_fn","title":"typedef exception_fn","text":"typedef bool(* opae::fpga::types::detail::exception_fn) (fpga_result, const opae::fpga::types::src_location &loc);\n
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#variable-opae_exceptions","title":"variable opae_exceptions","text":"exception_fn opae::fpga::types::detail::opae_exceptions[12];\n
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#function-is_ok","title":"function is_ok","text":"is_ok is a template function that throws an excpetion of its template argument type if the result code is not FPGA_OK.
template<typename T typename T>\nconstexpr bool opae::fpga::types::detail::is_ok (\nfpga_result result,\nconst opae::fpga::types::src_location & loc\n)
Otherwise it returns true.
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#function-assert_fpga_ok","title":"function assert_fpga_ok","text":"static inline void opae::fpga::types::detail::assert_fpga_ok (\nfpga_result result,\nconst opae::fpga::types::src_location & loc\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/","title":"Class opae::fpga::types::error","text":"ClassList > opae > fpga > types > error
An error object represents an error register for a resource. More...
#include <errors.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< error > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-functions","title":"Public Functions","text":"Type Name fpga_error_info c_type () constGet the C data structure. bool can_clear () Indicates whether an error register can be cleared. error (const error & e) = delete std::string name () Get the error register name. error & operator= (const error & e) = delete uint64_t read_value () Read the raw value contained in the associated error register. ~error ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-static-functions","title":"Public Static Functions","text":"Type Name error::ptr_t get (token::ptr_t tok, uint32_t num) Factory function for creating an error object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#detailed-description","title":"Detailed Description","text":"This is used to read out the raw value in the register. No parsing is done by this class.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<error> opae::fpga::types::error::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-c_type","title":"function c_type","text":"Get the C data structure.
inline fpga_error_info opae::fpga::types::error::c_type () const\n
Returns:
The fpga_error_info that contains the name and the can_clear boolean.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-can_clear","title":"function can_clear","text":"Indicates whether an error register can be cleared.
inline bool opae::fpga::types::error::can_clear ()
Returns:
A boolean value indicating if the error register can be cleared.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-error-12","title":"function error [\u00bd]","text":"opae::fpga::types::error::error (\nconst error & e\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-name","title":"function name","text":"Get the error register name.
inline std::string opae::fpga::types::error::name ()
Returns:
A std::string object set to the error name.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-operator","title":"function operator=","text":"error & opae::fpga::types::error::operator= (\nconst error & e\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-read_value","title":"function read_value","text":"Read the raw value contained in the associated error register.
uint64_t opae::fpga::types::error::read_value ()
Returns:
A 64-bit value (unparsed) read from the error register
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-error","title":"function ~error","text":"inline opae::fpga::types::error::~error ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-get","title":"function get","text":"Factory function for creating an error object.
static error::ptr_t opae::fpga::types::error::get (\ntoken::ptr_t tok,\nuint32_t num\n)
Parameters:
tok The token object representing a resource. num The index of the error register. This must be lower than the num_errors property of the resource.
Returns:
A shared_ptr containing the error object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/","title":"Class opae::fpga::types::event","text":"ClassList > opae > fpga > types > event
Wraps fpga event routines in OPAE C.
#include <events.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#classes","title":"Classes","text":"Type Name struct type_t C++ struct that is interchangeable with fpga_event_type enum."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< event > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-functions","title":"Public Functions","text":"Type Name fpga_event_handle get () Get the fpga_event_handle contained in this object. operator fpga_event_handle () Coversion operator for converting to fpga_event_handle objects. int os_object () constGet OS Object from the event object. virtual ~event () Destroy event and associated resources."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-static-functions","title":"Public Static Functions","text":"Type Name event::ptr_t register_event (handle::ptr_t h, event::type_t t, int flags=0) Factory function to create event objects."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<event> opae::fpga::types::event::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-get","title":"function get","text":"Get the fpga_event_handle contained in this object.
inline fpga_event_handle opae::fpga::types::event::get ()
Returns:
The fpga_event_handle contained in this object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-operator-fpga_event_handle","title":"function operator fpga_event_handle","text":"Coversion operator for converting to fpga_event_handle objects.
opae::fpga::types::event::operator fpga_event_handle ()
Returns:
The fpga_event_handle contained in this object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-os_object","title":"function os_object","text":"Get OS Object from the event object.
int opae::fpga::types::event::os_object () const\n
Get an OS specific object from the event which can be used to subscribe for events. On Linux, the object corresponds to a file descriptor that can be used with select/poll/epoll calls.
Returns:
An integer object representing the OS object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-event","title":"function ~event","text":"virtual opae::fpga::types::event::~event ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-register_event","title":"function register_event","text":"Factory function to create event objects.
static event::ptr_t opae::fpga::types::event::register_event (\nhandle::ptr_t h,\nevent::type_t t,\nint flags=0\n)
Parameters:
h A shared ptr of a resource handle t The resource type flags Event registration flags passed on to fpgaRegisterEvent
Returns:
A shared ptr to an event object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/events.h
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/","title":"Struct opae::fpga::types::event::type_t","text":"ClassList > opae > fpga > types > event > type_t
C++ struct that is interchangeable with fpga_event_type enum.
#include <events.h>
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-static-attributes","title":"Public Static Attributes","text":"Type Name constexpr fpga_event_type error = = FPGA_EVENT_ERROR constexpr fpga_event_type interrupt = = FPGA_EVENT_INTERRUPT constexpr fpga_event_type power_thermal = = FPGA_EVENT_POWER_THERMAL"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-functions","title":"Public Functions","text":"Type Name operator fpga_event_type () type_t (fpga_event_type c_type)"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#variable-error","title":"variable error","text":"constexpr fpga_event_type opae::fpga::types::event::type_t::error;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#variable-interrupt","title":"variable interrupt","text":"constexpr fpga_event_type opae::fpga::types::event::type_t::interrupt;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#variable-power_thermal","title":"variable power_thermal","text":"constexpr fpga_event_type opae::fpga::types::event::type_t::power_thermal;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#function-operator-fpga_event_type","title":"function operator fpga_event_type","text":"inline opae::fpga::types::event::type_t::operator fpga_event_type ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#function-type_t","title":"function type_t","text":"inline opae::fpga::types::event::type_t::type_t (\nfpga_event_type c_type\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/events.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/","title":"Class opae::fpga::types::except","text":"ClassList > opae > fpga > types > except
Generic OPAE exception. More...
#include <except.h>
Inherits the following classes: std::exception
Inherited by the following classes: opae::fpga::types::busy, opae::fpga::types::exception, opae::fpga::types::invalid_param, opae::fpga::types::no_access, opae::fpga::types::no_daemon, opae::fpga::types::no_driver, opae::fpga::types::no_memory, opae::fpga::types::not_found, opae::fpga::types::not_supported, opae::fpga::types::reconf_error
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-static-attributes","title":"Public Static Attributes","text":"Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-functions","title":"Public Functions","text":"Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#protected-attributes","title":"Protected Attributes","text":"Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#detailed-description","title":"Detailed Description","text":"An except tracks the source line of origin and an optional fpga_result. If no fpga_result is given, then FPGA_EXCEPTION is used.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-max_except","title":"variable MAX_EXCEPT","text":"const std::size_t opae::fpga::types::except::MAX_EXCEPT;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-except-13","title":"function except [\u2153]","text":"except constructor The fpga_result value is FPGA_EXCEPTION.
opae::fpga::types::except::except (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-except-23","title":"function except [\u2154]","text":"except constructor
opae::fpga::types::except::except (\nfpga_result res,\nsrc_location loc\n) noexcept\n
Parameters:
res The fpga_result value associated with this exception. loc Location where the exception was constructed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-except-33","title":"function except [3/3]","text":"except constructor
opae::fpga::types::except::except (\nfpga_result res,\nconst char * msg,\nsrc_location loc\n) noexcept\n
Parameters:
res The fpga_result value associated with this exception. msg The error message as a string loc Location where the exception was constructed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-operator-fpga_result","title":"function operator fpga_result","text":"inline opae::fpga::types::except::operator fpga_result () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-what","title":"function what","text":"virtual const char * opae::fpga::types::except::what () noexcept override const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#protected-attributes-documentation","title":"Protected Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-buf_","title":"variable buf_","text":"char opae::fpga::types::except::buf_[MAX_EXCEPT];\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-loc_","title":"variable loc_","text":"src_location opae::fpga::types::except::loc_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-msg_","title":"variable msg_","text":"const char* opae::fpga::types::except::msg_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-res_","title":"variable res_","text":"fpga_result opae::fpga::types::except::res_;\n
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/","title":"Class opae::fpga::types::exception","text":"ClassList > opae > fpga > types > exception
exception exception More...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-functions","title":"Public Functions","text":"Type Name exception (src_location loc) noexceptexception constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#detailed-description","title":"Detailed Description","text":"exception tracks the source line of origin for exceptions thrown when the error code FPGA_EXCEPTION is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#function-exception","title":"function exception","text":"exception constructor
inline opae::fpga::types::exception::exception (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/","title":"Struct opae::fpga::types::guid_t","text":"ClassList > opae > fpga > types > guid_t
Representation of the guid member of a properties object.
#include <pvalue.h>
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#public-functions","title":"Public Functions","text":"Type Name const uint8_t * c_type () constReturn a raw pointer to the guid. guid_t (fpga_properties * p) Construct the guid_t given its containing fpga_properties. void invalidate () Invalidate the cached local copy of the guid. bool is_set () constTracks whether the cached local copy of the guid is valid. operator uint8_t * () Return a raw pointer to the guid. guid_t & operator= (fpga_guid g) Assign from fpga_guid Sets the guid field of the associated properties object using the OPAE properties API. bool operator== (const fpga_guid & g) Compare contents with an fpga_guid. void parse (const char * str) Convert a string representation of a guid to binary. void update () Update the local cached copy of the guid."},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-c_type","title":"function c_type","text":"inline const uint8_t * opae::fpga::types::guid_t::c_type () const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-guid_t","title":"function guid_t","text":"inline opae::fpga::types::guid_t::guid_t (\nfpga_properties * p\n)
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-invalidate","title":"function invalidate","text":"inline void opae::fpga::types::guid_t::invalidate ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-is_set","title":"function is_set","text":"inline bool opae::fpga::types::guid_t::is_set () const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-operator-uint8_t","title":"function operator uint8_t *","text":"Return a raw pointer to the guid.
inline opae::fpga::types::guid_t::operator uint8_t * ()
Return value:
nullptr if the guid could not be queried.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-operator","title":"function operator=","text":"Assign from fpga_guid Sets the guid field of the associated properties object using the OPAE properties API.
inline guid_t & opae::fpga::types::guid_t::operator= (\nfpga_guid g\n)
Parameters:
g The given fpga_guid.
Returns:
a reference to this guid_t.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-operator_1","title":"function operator==","text":"Compare contents with an fpga_guid.
inline bool opae::fpga::types::guid_t::operator== (\nconst fpga_guid & g\n)
Return value:
The result of memcmp of the two objects.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-parse","title":"function parse","text":"Convert a string representation of a guid to binary.
inline void opae::fpga::types::guid_t::parse (\nconst char * str\n)
Parameters:
str The guid string.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-update","title":"function update","text":"inline void opae::fpga::types::guid_t::update ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#friends-documentation","title":"Friends Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#friend-operator","title":"friend operator<<","text":"inline std::ostream & opae::fpga::types::guid_t::operator<< (\nstd::ostream & ostr,\nconst guid_t & g\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/pvalue.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/","title":"Class opae::fpga::types::handle","text":"ClassList > opae > fpga > types > handle
An allocated accelerator resource. More...
#include <handle.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< handle > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-functions","title":"Public Functions","text":"Type Name fpga_handle c_type () constRetrieve the underlying OPAE handle. fpga_result close () Close an accelerator resource (if opened) token::ptr_t get_token () constRetrieve the token corresponding to this handle object. handle (const handle &) = delete uint8_t * mmio_ptr (uint64_t offset, uint32_t csr_space=0) constRetrieve a pointer to the MMIO region. operator fpga_handle () constRetrieve the underlying OPAE handle. handle & operator= (const handle &) = delete uint32_t read_csr32 (uint64_t offset, uint32_t csr_space=0) constRead 32 bits from a CSR belonging to a resource associated with a handle. uint64_t read_csr64 (uint64_t offset, uint32_t csr_space=0) constRead 64 bits from a CSR belonging to a resource associated with a handle. void reconfigure (uint32_t slot, const uint8_t * bitstream, size_t size, int flags) Load a bitstream into an FPGA slot. virtual void reset () Reset the accelerator identified by this handle. void write_csr32 (uint64_t offset, uint32_t value, uint32_t csr_space=0) Write 32 bit to a CSR belonging to a resource associated with a handle. void write_csr512 (uint64_t offset, const void * value, uint32_t csr_space=0) Write 512 bits to a CSR belonging to a resource associated with a handle. void write_csr64 (uint64_t offset, uint64_t value, uint32_t csr_space=0) Write 64 bits to a CSR belonging to a resource associated with a handle. virtual ~handle ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-static-functions","title":"Public Static Functions","text":"Type Name handle::ptr_t open (fpga_token token, int flags) Open an accelerator resource, given a raw fpga_token. handle::ptr_t open (token::ptr_t token, int flags) Open an accelerator resource, given a token object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#detailed-description","title":"Detailed Description","text":"Represents an accelerator resource that has been allocated by OPAE. Depending on the type of resource, its register space may be read/written using a handle object.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<handle> opae::fpga::types::handle::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-c_type","title":"function c_type","text":"inline fpga_handle opae::fpga::types::handle::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-close","title":"function close","text":"Close an accelerator resource (if opened)
fpga_result opae::fpga::types::handle::close ()
Returns:
fpga_result indication the result of closing the handle or FPGA_EXCEPTION if handle is not opened
Note:
This is available for explicitly closing a handle. The destructor for handle will call close.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-get_token","title":"function get_token","text":"token::ptr_t opae::fpga::types::handle::get_token () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-handle-12","title":"function handle [\u00bd]","text":"opae::fpga::types::handle::handle (\nconst handle &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-mmio_ptr","title":"function mmio_ptr","text":"Retrieve a pointer to the MMIO region.
uint8_t * opae::fpga::types::handle::mmio_ptr (\nuint64_t offset,\nuint32_t csr_space=0\n) const\n
Parameters:
offset The byte offset to add to MMIO base. csr_space The desired CSR space. Default is 0.
Returns:
MMIO base + offset
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-operator-fpga_handle","title":"function operator fpga_handle","text":"inline opae::fpga::types::handle::operator fpga_handle () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-operator","title":"function operator=","text":"handle & opae::fpga::types::handle::operator= (\nconst handle &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-read_csr32","title":"function read_csr32","text":"Read 32 bits from a CSR belonging to a resource associated with a handle.
uint32_t opae::fpga::types::handle::read_csr32 (\nuint64_t offset,\nuint32_t csr_space=0\n) const\n
Parameters:
offset The register offset csr_space The CSR space to read from. Default is 0.
Returns:
The 32-bit value read from the CSR
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-read_csr64","title":"function read_csr64","text":"Read 64 bits from a CSR belonging to a resource associated with a handle.
uint64_t opae::fpga::types::handle::read_csr64 (\nuint64_t offset,\nuint32_t csr_space=0\n) const\n
Parameters:
offset The register offset csr_space The CSR space to read from. Default is 0.
Returns:
The 64-bit value read from the CSR
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-reconfigure","title":"function reconfigure","text":"Load a bitstream into an FPGA slot.
void opae::fpga::types::handle::reconfigure (\nuint32_t slot,\nconst uint8_t * bitstream,\nsize_t size,\nint flags\n)
Parameters:
slot The slot number to program bitstream The bitstream binary data size The size of the bitstream flags Flags that control behavior of reconfiguration. Value of 0 indicates no flags. FPGA_RECONF_FORCE indicates that the bitstream is programmed into the slot without checking if the resource is currently in use.
Exception:
- invalid_param if the handle is not an FPGA device handle or if the other parameters are not valid.
exception if an internal error is encountered. busy if the accelerator for the given slot is in use. - reconf_error if errors are reported by the driver (CRC or protocol errors).
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-reset","title":"function reset","text":"virtual void opae::fpga::types::handle::reset ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-write_csr32","title":"function write_csr32","text":"Write 32 bit to a CSR belonging to a resource associated with a handle.
void opae::fpga::types::handle::write_csr32 (\nuint64_t offset,\nuint32_t value,\nuint32_t csr_space=0\n)
Parameters:
offset The register offset. value The 32-bit value to write to the register. csr_space The CSR space to read from. Default is 0.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-write_csr512","title":"function write_csr512","text":"Write 512 bits to a CSR belonging to a resource associated with a handle.
void opae::fpga::types::handle::write_csr512 (\nuint64_t offset,\nconst void * value,\nuint32_t csr_space=0\n)
Parameters:
offset The register offset. value Pointer to the 512-bit value to write to the register. csr_space The CSR space to read from. Default is 0.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-write_csr64","title":"function write_csr64","text":"Write 64 bits to a CSR belonging to a resource associated with a handle.
void opae::fpga::types::handle::write_csr64 (\nuint64_t offset,\nuint64_t value,\nuint32_t csr_space=0\n)
Parameters:
offset The register offset. value The 64-bit value to write to the register. csr_space The CSR space to read from. Default is 0.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-handle","title":"function ~handle","text":"virtual opae::fpga::types::handle::~handle ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-open-12","title":"function open [\u00bd]","text":"Open an accelerator resource, given a raw fpga_token.
static handle::ptr_t opae::fpga::types::handle::open (\nfpga_token token,\nint flags\n)
Parameters:
token A token describing the accelerator resource to be allocated.flags The flags parameter to fpgaOpen().
Returns:
pointer to the mmio base + offset for the given csr space
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-open-22","title":"function open [2/2]","text":"Open an accelerator resource, given a token object.
static handle::ptr_t opae::fpga::types::handle::open (\ntoken::ptr_t token,\nint flags\n)
Parameters:
token A token object describing the accelerator resource to be allocated.flags The flags parameter to fpgaOpen().
Returns:
shared ptr to a handle object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/handle.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/","title":"Class opae::fpga::types::invalid_param","text":"ClassList > opae > fpga > types > invalid_param
invalid_param exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-functions","title":"Public Functions","text":"Type Name invalid_param (src_location loc) noexceptinvalid_param constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#detailed-description","title":"Detailed Description","text":"invalid_param tracks the source line of origin for exceptions thrown when the error code FPGA_INVALID_PARAM is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#function-invalid_param","title":"function invalid_param","text":"invalid_param constructor
inline opae::fpga::types::invalid_param::invalid_param (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/","title":"Class opae::fpga::types::no_access","text":"ClassList > opae > fpga > types > no_access
no_access exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-functions","title":"Public Functions","text":"Type Name no_access (src_location loc) noexceptno_access constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#detailed-description","title":"Detailed Description","text":"no_access tracks the source line of origin for exceptions thrown when the error code FPGA_NO_ACCESS is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#function-no_access","title":"function no_access","text":"no_access constructor
inline opae::fpga::types::no_access::no_access (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/","title":"Class opae::fpga::types::no_daemon","text":"ClassList > opae > fpga > types > no_daemon
no_daemon exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-functions","title":"Public Functions","text":"Type Name no_daemon (src_location loc) noexceptno_daemon constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#detailed-description","title":"Detailed Description","text":"no_daemon tracks the source line of origin for exceptions thrown when the error code FPGA_NO_DAEMON is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#function-no_daemon","title":"function no_daemon","text":"no_daemon constructor
inline opae::fpga::types::no_daemon::no_daemon (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/","title":"Class opae::fpga::types::no_driver","text":"ClassList > opae > fpga > types > no_driver
no_driver exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-functions","title":"Public Functions","text":"Type Name no_driver (src_location loc) noexceptno_driver constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#detailed-description","title":"Detailed Description","text":"no_driver tracks the source line of origin for exceptions thrown when the error code FPGA_NO_DRIVER is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#function-no_driver","title":"function no_driver","text":"no_driver constructor
inline opae::fpga::types::no_driver::no_driver (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/","title":"Class opae::fpga::types::no_memory","text":"ClassList > opae > fpga > types > no_memory
no_memory exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-functions","title":"Public Functions","text":"Type Name no_memory (src_location loc) noexceptno_memory constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#detailed-description","title":"Detailed Description","text":"no_memory tracks the source line of origin for exceptions thrown when the error code FPGA_NO_MEMORY is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#function-no_memory","title":"function no_memory","text":"no_memory constructor
inline opae::fpga::types::no_memory::no_memory (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/","title":"Class opae::fpga::types::not_found","text":"ClassList > opae > fpga > types > not_found
not_found exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-functions","title":"Public Functions","text":"Type Name not_found (src_location loc) noexceptnot_found constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#detailed-description","title":"Detailed Description","text":"not_found tracks the source line of origin for exceptions thrown when the error code FPGA_NOT_FOUND is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#function-not_found","title":"function not_found","text":"not_found constructor
inline opae::fpga::types::not_found::not_found (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/","title":"Class opae::fpga::types::not_supported","text":"ClassList > opae > fpga > types > not_supported
not_supported exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-functions","title":"Public Functions","text":"Type Name not_supported (src_location loc) noexceptnot_supported constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#detailed-description","title":"Detailed Description","text":"not_supported tracks the source line of origin for exceptions thrown when the error code FPGA_NOT_SUPPORTED is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#function-not_supported","title":"function not_supported","text":"not_supported constructor
inline opae::fpga::types::not_supported::not_supported (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/","title":"Class opae::fpga::types::properties","text":"ClassList > opae > fpga > types > properties
Wraps an OPAE fpga_properties object. More...
#include <properties.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< properties > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-attributes","title":"Public Attributes","text":"Type Name pvalue< fpga_accelerator_state > accelerator_state pvalue< uint64_t > bbs_id pvalue< fpga_version > bbs_version pvalue< uint8_t > bus pvalue< uint64_t > capabilities pvalue< uint8_t > device pvalue< uint16_t > device_id pvalue< uint8_t > function guid_t guid pvalue< fpga_interface > interface pvalue< uint64_t > local_memory_size pvalue< char * > model pvalue< uint32_t > num_errors pvalue< uint32_t > num_interrupts pvalue< uint32_t > num_mmio pvalue< uint32_t > num_slots pvalue< uint64_t > object_id pvalue< fpga_token > parent pvalue< uint16_t > segment pvalue< uint8_t > socket_id pvalue< uint16_t > subsystem_device_id pvalue< uint16_t > subsystem_vendor_id pvalue< fpga_objtype > type pvalue< uint16_t > vendor_id"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-attributes","title":"Public Static Attributes","text":"Type Name const std::vector< properties::ptr_t > none An empty vector of properties."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-functions","title":"Public Functions","text":"Type Name fpga_properties c_type () constGet the underlying fpga_properties object. operator fpga_properties () constGet the underlying fpga_properties object. properties & operator= (const properties & p) = delete properties (const properties & p) = delete ~properties ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-functions","title":"Public Static Functions","text":"Type Name properties::ptr_t get () Create a new properties object. properties::ptr_t get (fpga_guid guid_in) Create a new properties object from a guid. properties::ptr_t get (fpga_objtype objtype) Create a new properties object from an fpga_objtype. properties::ptr_t get (std::shared_ptr< token > t) Retrieve the properties for a given token object. properties::ptr_t get (fpga_token t) Retrieve the properties for a given fpga_token. properties::ptr_t get (std::shared_ptr< handle > h) Retrieve the properties for a given handle object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#detailed-description","title":"Detailed Description","text":"properties are information describing an accelerator resource that is identified by its token. The properties are used during enumeration to narrow the search for an accelerator resource, and after enumeration to provide the configuration of that resource.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<properties> opae::fpga::types::properties::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-accelerator_state","title":"variable accelerator_state","text":"pvalue<fpga_accelerator_state> opae::fpga::types::properties::accelerator_state;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-bbs_id","title":"variable bbs_id","text":"pvalue<uint64_t> opae::fpga::types::properties::bbs_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-bbs_version","title":"variable bbs_version","text":"pvalue<fpga_version> opae::fpga::types::properties::bbs_version;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-bus","title":"variable bus","text":"pvalue<uint8_t> opae::fpga::types::properties::bus;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-capabilities","title":"variable capabilities","text":"pvalue<uint64_t> opae::fpga::types::properties::capabilities;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-device","title":"variable device","text":"pvalue<uint8_t> opae::fpga::types::properties::device;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-device_id","title":"variable device_id","text":"pvalue<uint16_t> opae::fpga::types::properties::device_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-function","title":"variable function","text":"pvalue<uint8_t> opae::fpga::types::properties::function;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-guid","title":"variable guid","text":"guid_t opae::fpga::types::properties::guid;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-interface","title":"variable interface","text":"pvalue<fpga_interface> opae::fpga::types::properties::interface;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-local_memory_size","title":"variable local_memory_size","text":"pvalue<uint64_t> opae::fpga::types::properties::local_memory_size;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-model","title":"variable model","text":"pvalue<char *> opae::fpga::types::properties::model;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_errors","title":"variable num_errors","text":"pvalue<uint32_t> opae::fpga::types::properties::num_errors;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_interrupts","title":"variable num_interrupts","text":"pvalue<uint32_t> opae::fpga::types::properties::num_interrupts;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_mmio","title":"variable num_mmio","text":"pvalue<uint32_t> opae::fpga::types::properties::num_mmio;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_slots","title":"variable num_slots","text":"pvalue<uint32_t> opae::fpga::types::properties::num_slots;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-object_id","title":"variable object_id","text":"pvalue<uint64_t> opae::fpga::types::properties::object_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-parent","title":"variable parent","text":"pvalue<fpga_token> opae::fpga::types::properties::parent;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-segment","title":"variable segment","text":"pvalue<uint16_t> opae::fpga::types::properties::segment;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-socket_id","title":"variable socket_id","text":"pvalue<uint8_t> opae::fpga::types::properties::socket_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-subsystem_device_id","title":"variable subsystem_device_id","text":"pvalue<uint16_t> opae::fpga::types::properties::subsystem_device_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-subsystem_vendor_id","title":"variable subsystem_vendor_id","text":"pvalue<uint16_t> opae::fpga::types::properties::subsystem_vendor_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-type","title":"variable type","text":"pvalue<fpga_objtype> opae::fpga::types::properties::type;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-vendor_id","title":"variable vendor_id","text":"pvalue<uint16_t> opae::fpga::types::properties::vendor_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-none","title":"variable none","text":"An empty vector of properties.
const std::vector<properties::ptr_t> opae::fpga::types::properties::none;\n
Useful for enumerating based on a \"match all\" criteria.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-c_type","title":"function c_type","text":"inline fpga_properties opae::fpga::types::properties::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-operator-fpga_properties","title":"function operator fpga_properties","text":"inline opae::fpga::types::properties::operator fpga_properties () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-operator","title":"function operator=","text":"properties & opae::fpga::types::properties::operator= (\nconst properties & p\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-properties-12","title":"function properties [\u00bd]","text":"opae::fpga::types::properties::properties (\nconst properties & p\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-properties","title":"function ~properties","text":"opae::fpga::types::properties::~properties ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-16","title":"function get [\u2159]","text":"Create a new properties object.
static properties::ptr_t opae::fpga::types::properties::get ()
Returns:
A properties smart pointer.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-26","title":"function get [2/6]","text":"Create a new properties object from a guid.
static properties::ptr_t opae::fpga::types::properties::get (\nfpga_guid guid_in\n)
Parameters:
guid_in A guid to set in the properties
Returns:
A properties smart pointer with its guid initialized to guid_in
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-36","title":"function get [3/6]","text":"Create a new properties object from an fpga_objtype.
static properties::ptr_t opae::fpga::types::properties::get (\nfpga_objtype objtype\n)
Parameters:
objtype An object type to set in the properties
Returns:
A properties smart pointer with its object type set to objtype.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-46","title":"function get [4/6]","text":"Retrieve the properties for a given token object.
static properties::ptr_t opae::fpga::types::properties::get (\nstd::shared_ptr< token > t\n)
Parameters:
t A token identifying the accelerator resource.
Returns:
A properties smart pointer for the given token.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-56","title":"function get [\u215a]","text":"Retrieve the properties for a given fpga_token.
static properties::ptr_t opae::fpga::types::properties::get (\nfpga_token t\n)
Parameters:
t An fpga_token identifying the accelerator resource.
Returns:
A properties smart pointer for the given fpga_token.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-66","title":"function get [6/6]","text":"Retrieve the properties for a given handle object.
static properties::ptr_t opae::fpga::types::properties::get (\nstd::shared_ptr< handle > h\n)
Parameters:
h A handle identifying the accelerator resource.
Returns:
A properties smart pointer for the given handle.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/properties.h
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/","title":"Struct opae::fpga::types::pvalue","text":"template <typename T typename T>
ClassList > opae > fpga > types > pvalue
Wraps OPAE properties defined in the OPAE C API by associating an fpga_properties reference with the getters and setters defined for a property.More...
#include <pvalue.h>
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-types","title":"Public Types","text":"Type Name typedef std::conditional< std::is_same< T, char * >::value, typename std::string, T >::type copy_t Define the type of our copy variable For char* types use std::string as the copy. typedef std::conditional< std::is_same< T, char * >::value, fpga_result(*)(fpga_properties, T), fpga_result(*)(fpga_properties, T *)>::type getter_t Define getter function as getter_t For char* types, do not use T* as the second argument but instead use T. typedef fpga_result(* setter_t Define the setter function as setter_t."},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-functions","title":"Public Functions","text":"Type Name fpga_result get_value (T & value) const void invalidate () Invalidate the cached local copy of the pvalue. bool is_set () constTracks whether the cached local copy of the pvalue is valid. operator copy_t () Implicit converter operator - calls the wrapped getter. pvalue< T > & operator= (const T & v) Overload of = operator that calls the wrapped setter. bool operator== (const T & other) Compare a property for equality with a value. pvalue () pvalue (fpga_properties * p, getter_t g, setter_t s) pvalue contructor that takes in a reference to fpga_properties and corresponding accessor methods for a property void update () void update () Template specialization of char* type property updater."},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#detailed-description","title":"Detailed Description","text":"Template parameters:
T The type of the property value being wrapped
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#typedef-copy_t","title":"typedef copy_t","text":"typedef std::conditional<std::is_same<T, char *>::value, typename std::string, T>::type opae::fpga::types::pvalue< T >::copy_t;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#typedef-getter_t","title":"typedef getter_t","text":"typedef std::conditional< std::is_same<T, char *>::value, fpga_result (*)(fpga_properties, T), fpga_result (*)(fpga_properties, T *)>::type opae::fpga::types::pvalue< T >::getter_t;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#typedef-setter_t","title":"typedef setter_t","text":"typedef fpga_result(* opae::fpga::types::pvalue< T >::setter_t) (fpga_properties, T);\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-get_value","title":"function get_value","text":"inline fpga_result opae::fpga::types::pvalue::get_value (\nT & value\n) const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-invalidate","title":"function invalidate","text":"inline void opae::fpga::types::pvalue::invalidate ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-is_set","title":"function is_set","text":"inline bool opae::fpga::types::pvalue::is_set () const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-operator-copy_t","title":"function operator copy_t","text":"Implicit converter operator - calls the wrapped getter.
inline opae::fpga::types::pvalue::operator copy_t ()
Returns:
The property value after calling the getter or a default value of the value type
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-operator","title":"function operator=","text":"Overload of = operator that calls the wrapped setter.
inline pvalue < T > & opae::fpga::types::pvalue::operator= (\nconst T & v\n)
Parameters:
v The value to set
Returns:
A reference to itself
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-operator_1","title":"function operator==","text":"Compare a property for equality with a value.
inline bool opae::fpga::types::pvalue::operator== (\nconst T & other\n)
Parameters:
other The value being compared to
Returns:
Whether or not the property is equal to the value
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-pvalue-12","title":"function pvalue [\u00bd]","text":"inline opae::fpga::types::pvalue::pvalue ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-pvalue-22","title":"function pvalue [2/2]","text":"pvalue contructor that takes in a reference to fpga_properties and corresponding accessor methods for a property
inline opae::fpga::types::pvalue::pvalue (\nfpga_properties * p,\ngetter_t g,\nsetter_t s\n)
Parameters:
p A reference to an fpga_properties g The getter function s The setter function
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-update-12","title":"function update [\u00bd]","text":"inline void opae::fpga::types::pvalue::update ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-update-22","title":"function update [2/2]","text":"Template specialization of char* type property updater.
inline void opae::fpga::types::pvalue::update ()
Returns:
The result of the property getter function.
## Friends Documentation\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#friend-operator","title":"friend operator<<","text":"Stream overalod operator.
inline std::ostream & opae::fpga::types::pvalue::operator<< (\nstd::ostream & ostr,\nconst pvalue < T > & p\n)
Parameters:
ostr The output stream p A reference to a pvalue<T> object
Returns:
The stream operator after streaming the property value
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/pvalue.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/","title":"Class opae::fpga::types::reconf_error","text":"ClassList > opae > fpga > types > reconf_error
reconf_error exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-functions","title":"Public Functions","text":"Type Name reconf_error (src_location loc) noexceptreconf_error constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#detailed-description","title":"Detailed Description","text":"reconf_error tracks the source line of origin for exceptions thrown when the error code FPGA_RECONF_ERROR is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#function-reconf_error","title":"function reconf_error","text":"reconf_error constructor
inline opae::fpga::types::reconf_error::reconf_error (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/","title":"Class opae::fpga::types::shared_buffer","text":"ClassList > opae > fpga > types > shared_buffer
Host/AFU shared memory blocks. More...
#include <shared_buffer.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< shared_buffer > ptr_t typedef std::size_t size_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-functions","title":"Public Functions","text":"Type Name uint8_t * c_type () constRetrieve the virtual address of the buffer base. int compare (ptr_t other, size_t len) constCompare this shared_buffer (the first len bytes) to that held in other, using memcmp(). void fill (int c) Write c to each byte location in the buffer. uint64_t io_address () constRetrieve the address of the buffer suitable for programming into the accelerator device. shared_buffer & operator= (const shared_buffer &) = delete handle::ptr_t owner () constRetrieve the handle smart pointer associated with this buffer. T read (size_t offset) constRead a T-sized block of memory at the given location. void release () Disassociate the shared_buffer object from the resource used to create it. shared_buffer (const shared_buffer &) = delete size_t size () constRetrieve the length of the buffer in bytes. void write (const T & value, size_t offset) Write a T-sized block of memory to the given location. uint64_t wsid () constRetrieve the underlying buffer's workspace id. virtual ~shared_buffer () shared_buffer destructor."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-static-functions","title":"Public Static Functions","text":"Type Name shared_buffer::ptr_t allocate (handle::ptr_t handle, size_t len, bool read_only=false) shared_buffer factory method - allocate ashared_buffer . shared_buffer::ptr_t attach (handle::ptr_t handle, uint8_t * base, size_t len, bool read_only=false) Attach a pre-allocated buffer to a shared_buffer object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-attributes","title":"Protected Attributes","text":"Type Name handle::ptr_t handle_ uint64_t io_address_ size_t len_ uint8_t * virt_ uint64_t wsid_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-functions","title":"Protected Functions","text":"Type Name shared_buffer (handle::ptr_t handle, size_t len, uint8_t * virt, uint64_t wsid, uint64_t io_address)"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#detailed-description","title":"Detailed Description","text":"shared_buffer abstracts a memory block that may be shared between the host cpu and an accelerator. The block may be allocated by the shared_buffer class itself (see allocate), or it may be allocated elsewhere and then attached to a shared_buffer object via attach.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<shared_buffer> opae::fpga::types::shared_buffer::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#typedef-size_t","title":"typedef size_t","text":"typedef std::size_t opae::fpga::types::shared_buffer::size_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-c_type","title":"function c_type","text":"Retrieve the virtual address of the buffer base.
inline uint8_t * opae::fpga::types::shared_buffer::c_type () const\n
Note:
Instances of a shared buffer can only be created using either 'allocate' or 'attach' static factory function. Because these functions return a shared pointer (std::shared_ptr) to the instance, references to an instance are counted automatically by design of the shared_ptr class. Calling 'c_type()' function is provided to get access to the raw data but isn't used in tracking its reference count. Assigning this to a variable should be done in limited scopes as this variable can be defined in an outer scope and may outlive the shared_buffer object. Once the reference count in the shared_ptr reaches zero, the shared_buffer object will be released and deallocated, turning any variables assigned from a call to 'c_type()' into dangling pointers.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-compare","title":"function compare","text":"int opae::fpga::types::shared_buffer::compare (\nptr_t other,\nsize_t len\n) const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-fill","title":"function fill","text":"void opae::fpga::types::shared_buffer::fill (\nint c\n)
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-io_address","title":"function io_address","text":"inline uint64_t opae::fpga::types::shared_buffer::io_address () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-operator","title":"function operator=","text":"shared_buffer & opae::fpga::types::shared_buffer::operator= (\nconst shared_buffer &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-owner","title":"function owner","text":"inline handle::ptr_t opae::fpga::types::shared_buffer::owner () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-read","title":"function read","text":"Read a T-sized block of memory at the given location.
template<typename T typename T>\ninline T opae::fpga::types::shared_buffer::read (\nsize_t offset\n) const\n
Parameters:
offset The byte offset from the start of the buffer.
Returns:
A T from buffer base + offset.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-release","title":"function release","text":"Disassociate the shared_buffer object from the resource used to create it.
void opae::fpga::types::shared_buffer::release ()
If the buffer was allocated using the allocate function then the buffer is freed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-shared_buffer-12","title":"function shared_buffer [\u00bd]","text":"opae::fpga::types::shared_buffer::shared_buffer (\nconst shared_buffer &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-size","title":"function size","text":"inline size_t opae::fpga::types::shared_buffer::size () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-write","title":"function write","text":"Write a T-sized block of memory to the given location.
template<typename T typename T>\ninline void opae::fpga::types::shared_buffer::write (\nconst T & value,\nsize_t offset\n)
Parameters:
value The value to write. offset The byte offset from the start of the buffer.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-wsid","title":"function wsid","text":"inline uint64_t opae::fpga::types::shared_buffer::wsid () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-shared_buffer","title":"function ~shared_buffer","text":"virtual opae::fpga::types::shared_buffer::~shared_buffer ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-allocate","title":"function allocate","text":"shared_buffer factory method - allocate ashared_buffer .
static shared_buffer::ptr_t opae::fpga::types::shared_buffer::allocate (\nhandle::ptr_t handle,\nsize_t len,\nbool read_only=false\n)
Parameters:
handle The handle used to allocate the buffer. len The length in bytes of the requested buffer.
Returns:
A valid shared_buffer smart pointer on success, or an empty smart pointer on failure.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-attach","title":"function attach","text":"Attach a pre-allocated buffer to a shared_buffer object.
static shared_buffer::ptr_t opae::fpga::types::shared_buffer::attach (\nhandle::ptr_t handle,\nuint8_t * base,\nsize_t len,\nbool read_only=false\n)
Parameters:
handle The handle used to attach the buffer. base The base of the pre-allocated memory. len The size of the pre-allocated memory, which must be a multiple of the page size.
Returns:
A valid shared_buffer smart pointer on success, or an empty smart pointer on failure.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-attributes-documentation","title":"Protected Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-handle_","title":"variable handle_","text":"handle::ptr_t opae::fpga::types::shared_buffer::handle_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-io_address_","title":"variable io_address_","text":"uint64_t opae::fpga::types::shared_buffer::io_address_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-len_","title":"variable len_","text":"size_t opae::fpga::types::shared_buffer::len_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-virt_","title":"variable virt_","text":"uint8_t* opae::fpga::types::shared_buffer::virt_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-wsid_","title":"variable wsid_","text":"uint64_t opae::fpga::types::shared_buffer::wsid_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-functions-documentation","title":"Protected Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-shared_buffer-22","title":"function shared_buffer [2/2]","text":"opae::fpga::types::shared_buffer::shared_buffer (\nhandle::ptr_t handle,\nsize_t len,\nuint8_t * virt,\nuint64_t wsid,\nuint64_t io_address\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/shared_buffer.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/","title":"Class opae::fpga::types::src_location","text":"ClassList > opae > fpga > types > src_location
Identify a particular line in a source file.
#include <except.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#public-functions","title":"Public Functions","text":"Type Name const char * file () noexcept constRetrieve the file name component of the location. const char * fn () noexcept constRetrieve the function name component of the location. int line () noexcept constRetrieve the line number component of the location. src_location & operator= (const src_location & other) noexcept src_location (const char * file, const char * fn, int line) noexceptsrc_location constructor src_location (const src_location & other) noexcept"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-file","title":"function file","text":"const char * opae::fpga::types::src_location::file () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-fn","title":"function fn","text":"inline const char * opae::fpga::types::src_location::fn () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-line","title":"function line","text":"inline int opae::fpga::types::src_location::line () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-operator","title":"function operator=","text":"src_location & opae::fpga::types::src_location::operator= (\nconst src_location & other\n) noexcept\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-src_location-12","title":"function src_location [\u00bd]","text":"src_location constructor
opae::fpga::types::src_location::src_location (\nconst char * file,\nconst char * fn,\nint line\n) noexcept\n
Parameters:
file The source file name, typically FILE. fn The current function, typically func. line The current line number, typically LINE.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-src_location-22","title":"function src_location [2/2]","text":"opae::fpga::types::src_location::src_location (\nconst src_location & other\n) noexcept\n
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/","title":"Class opae::fpga::types::sysobject","text":"ClassList > opae > fpga > types > sysobject
Wraps the OPAE fpga_object primitive. More...
#include <sysobject.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< sysobject > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-functions","title":"Public Functions","text":"Type Name std::vector< uint8_t > bytes (int flags=0) constGet all raw bytes from the object. std::vector< uint8_t > bytes (uint32_t offset, uint32_t size, int flags=0) constGet a subset of raw bytes from the object. fpga_object c_type () constRetrieve the underlying fpga_object primitive. sysobject::ptr_t get (const std::string & name, int flags=0) Get a sysobject from an object. sysobject::ptr_t get (int index) Get a sysobject from a container object. operator fpga_object () constRetrieve the underlying fpga_object primitive. sysobject & operator= (const sysobject & o) = delete uint64_t read64 (int flags=0) constRead a 64-bit value from an FPGA object. uint32_t size () constGet the size (in bytes) of the object. sysobject () = delete sysobject (const sysobject & o) = delete enum fpga_sysobject_type type () constGet the object type (attribute or container) void write64 (uint64_t value, int flags=0) constWrite 64-bit value to an FPGA object. virtual ~sysobject ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-static-functions","title":"Public Static Functions","text":"Type Name sysobject::ptr_t get (token::ptr_t t, const std::string & name, int flags=0) Get a sysobject from a token. sysobject::ptr_t get (handle::ptr_t h, const std::string & name, int flags=0) Get a sysobject from a handle."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#detailed-description","title":"Detailed Description","text":"sysobject's are created from a call to fpgaTokenGetObject, fpgaHandleGetObject, or fpgaObjectGetObject
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<sysobject> opae::fpga::types::sysobject::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-bytes-12","title":"function bytes [\u00bd]","text":"Get all raw bytes from the object.
std::vector< uint8_t > opae::fpga::types::sysobject::bytes (\nint flags=0\n) const\n
Parameters:
flags Flags that control how object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data.
Returns:
A vector of all bytes in the object.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-bytes-22","title":"function bytes [2/2]","text":"Get a subset of raw bytes from the object.
std::vector< uint8_t > opae::fpga::types::sysobject::bytes (\nuint32_t offset,\nuint32_t size,\nint flags=0\n) const\n
Parameters:
offset The bytes offset for the start of the returned vector. size The number of bytes for the returned vector. flags Flags that control how object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data.
Returns:
A vector of size bytes in the object starting at offset.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-c_type","title":"function c_type","text":"inline fpga_object opae::fpga::types::sysobject::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-14","title":"function get [\u00bc]","text":"Get a sysobject from an object.
sysobject::ptr_t opae::fpga::types::sysobject::get (\nconst std::string & name,\nint flags=0\n)
This will be read-write if its parent was created from a handle..
Parameters:
name An identifier representing an object belonging to this object. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects. Flags are defaulted to 0 meaning no flags.
Returns:
A shared_ptr to a sysobject instance.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-24","title":"function get [2/4]","text":"Get a sysobject from a container object.
sysobject::ptr_t opae::fpga::types::sysobject::get (\nint index\n)
This will be read-write if its parent was created from a handle..
Parameters:
index An index number to get.
Returns:
A shared_ptr to a sysobject instance.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-operator-fpga_object","title":"function operator fpga_object","text":"inline opae::fpga::types::sysobject::operator fpga_object () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-operator","title":"function operator=","text":"sysobject & opae::fpga::types::sysobject::operator= (\nconst sysobject & o\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-read64","title":"function read64","text":"Read a 64-bit value from an FPGA object.
uint64_t opae::fpga::types::sysobject::read64 (\nint flags=0\n) const\n
The value is assumed to be in string format and will be parsed. See flags below for changing that behavior.
Parameters:
flags Flags that control how the object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data. If FPGA_OBJECT_RAW is used, then the data will be read as raw bytes into the uint64_t pointer variable. Flags are defaulted to 0 meaning no flags.
Returns:
A 64-bit value from the object.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-size","title":"function size","text":"Get the size (in bytes) of the object.
uint32_t opae::fpga::types::sysobject::size () const\n
Returns:
The number of bytes that the object occupies in memory.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-sysobject-13","title":"function sysobject [\u2153]","text":"opae::fpga::types::sysobject::sysobject () = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-sysobject-23","title":"function sysobject [\u2154]","text":"opae::fpga::types::sysobject::sysobject (\nconst sysobject & o\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-type","title":"function type","text":"enum fpga_sysobject_type opae::fpga::types::sysobject::type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-write64","title":"function write64","text":"Write 64-bit value to an FPGA object.
void opae::fpga::types::sysobject::write64 (\nuint64_t value,\nint flags=0\n) const\n
The value will be converted to string before writing. See flags below for changing that behavior.
Parameters:
value The value to write to the object. flags Flags that control how the object is written If FPGA_OBJECT_RAW is used, then the value will be written as raw bytes. Flags are defaulted to 0 meaning no flags.
Note:
This operation will force a sync operation to update its cached buffer
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-sysobject","title":"function ~sysobject","text":"virtual opae::fpga::types::sysobject::~sysobject ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-34","title":"function get [\u00be]","text":"Get a sysobject from a token.
static sysobject::ptr_t opae::fpga::types::sysobject::get (\ntoken::ptr_t t,\nconst std::string & name,\nint flags=0\n)
This will be read-only.
Parameters:
t Token object representing a resource. name An identifier representing an object belonging to a resource represented by the token. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
A shared_ptr to a sysobject instance.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-44","title":"function get [4/4]","text":"Get a sysobject from a handle.
static sysobject::ptr_t opae::fpga::types::sysobject::get (\nhandle::ptr_t h,\nconst std::string & name,\nint flags=0\n)
This will be read-write.
Parameters:
h Handle object representing an open resource. name An identifier representing an object belonging to a resource represented by the handle. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
A shared_ptr to a sysobject instance.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/sysobject.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/","title":"Class opae::fpga::types::token","text":"ClassList > opae > fpga > types > token
Wraps the OPAE fpga_token primitive. More...
#include <token.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< token > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-functions","title":"Public Functions","text":"Type Name fpga_token c_type () constRetrieve the underlying fpga_token primitive. ptr_t get_parent () constRetrieve the parent token, or an empty pointer if there is none. operator fpga_token () constRetrieve the underlying fpga_token primitive. ~token ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-static-functions","title":"Public Static Functions","text":"Type Name std::vector< token::ptr_t > enumerate (const std::vector< properties::ptr_t > & props) Obtain a vector of token smart pointers for given search criteria."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#detailed-description","title":"Detailed Description","text":"token's are created from an enumeration operation that uses properties describing an accelerator resource as search criteria.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<token> opae::fpga::types::token::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-c_type","title":"function c_type","text":"inline fpga_token opae::fpga::types::token::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-get_parent","title":"function get_parent","text":"ptr_t opae::fpga::types::token::get_parent () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-operator-fpga_token","title":"function operator fpga_token","text":"inline opae::fpga::types::token::operator fpga_token () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-token","title":"function ~token","text":"opae::fpga::types::token::~token ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-enumerate","title":"function enumerate","text":"Obtain a vector of token smart pointers for given search criteria.
static std::vector< token::ptr_t > opae::fpga::types::token::enumerate (\nconst std::vector< properties::ptr_t > & props\n)
Parameters:
props The search criteria.
Returns:
A set of known tokens that match the search.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/token.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/","title":"Class opae::fpga::types::version","text":"ClassList > opae > fpga > types > version
#include <version.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#public-static-functions","title":"Public Static Functions","text":"Type Name std::string as_string () Get the package version information as a string. fpga_version as_struct () Get the package version information as a struct. std::string build () Get the package build information as a string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#function-as_string","title":"function as_string","text":"Get the package version information as a string.
static std::string opae::fpga::types::version::as_string ()
Returns:
The package version as an std::string object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#function-as_struct","title":"function as_struct","text":"Get the package version information as a struct.
static fpga_version opae::fpga::types::version::as_struct ()
Returns:
The package version as an fpga_version struct
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#function-build","title":"function build","text":"Get the package build information as a string.
static std::string opae::fpga::types::version::build ()
Returns:
The package build as an std::string object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/version.h
"},{"location":"opae-code/structopae__uio/","title":"Struct opae_uio","text":"ClassList > opae_uio
OPAE UIO device abstraction. More...
#include <uio.h>
"},{"location":"opae-code/structopae__uio/#public-attributes","title":"Public Attributes","text":"Type Name int device_fd char device_path struct opae_uio_device_region * regions"},{"location":"opae-code/structopae__uio/#detailed-description","title":"Detailed Description","text":"This structure is used to interact with the OPAE UIO API. Each UIO device has a file descriptor that is used to mmap its regions into user address space. Each device also has one or more MMIO regions.
"},{"location":"opae-code/structopae__uio/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__uio/#variable-device_fd","title":"variable device_fd","text":"int opae_uio::device_fd;\n
"},{"location":"opae-code/structopae__uio/#variable-device_path","title":"variable device_path","text":"char opae_uio::device_path[OPAE_UIO_PATH_MAX];\n
"},{"location":"opae-code/structopae__uio/#variable-regions","title":"variable regions","text":"struct opae_uio_device_region* opae_uio::regions;\n
The documentation for this class was generated from the following file docs/sw/include/opae/uio.h
"},{"location":"opae-code/structopae__uio__device__region/","title":"Struct opae_uio_device_region","text":"ClassList > opae_uio_device_region
MMIO region info. More...
#include <uio.h>
"},{"location":"opae-code/structopae__uio__device__region/#public-attributes","title":"Public Attributes","text":"Type Name struct opae_uio_device_region * next uint32_t region_index size_t region_page_offset uint8_t * region_ptr size_t region_size"},{"location":"opae-code/structopae__uio__device__region/#detailed-description","title":"Detailed Description","text":"Describes a range of mappable MMIO.
"},{"location":"opae-code/structopae__uio__device__region/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__uio__device__region/#variable-next","title":"variable next","text":"struct opae_uio_device_region* opae_uio_device_region::next;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_index","title":"variable region_index","text":"uint32_t opae_uio_device_region::region_index;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_page_offset","title":"variable region_page_offset","text":"size_t opae_uio_device_region::region_page_offset;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_ptr","title":"variable region_ptr","text":"uint8_t* opae_uio_device_region::region_ptr;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_size","title":"variable region_size","text":"size_t opae_uio_device_region::region_size;\n
The documentation for this class was generated from the following file docs/sw/include/opae/uio.h
"},{"location":"opae-code/structopae__vfio/","title":"Struct opae_vfio","text":"ClassList > opae_vfio
OPAE VFIO device abstraction. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio/#public-attributes","title":"Public Attributes","text":"Type Name opae_hash_map cont_buffers Map of allocated DMA buffers. char * cont_device \"/dev/vfio/vfio\" int cont_fd Container file descriptor. char * cont_pciaddr PCIe address, eg 0000:00:00.0. struct opae_vfio_iova_range * cont_ranges List of IOVA ranges. struct opae_vfio_device device The VFIO device. struct opae_vfio_group group The VFIO device group. struct mem_alloc iova_alloc Allocator for IOVA space. pthread_mutex_t lock For thread safety."},{"location":"opae-code/structopae__vfio/#detailed-description","title":"Detailed Description","text":"This structure is used to interact with the OPAE VFIO API. It tracks data related to the VFIO container, group, and device. A mutex is provided for thread safety.
"},{"location":"opae-code/structopae__vfio/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio/#variable-cont_buffers","title":"variable cont_buffers","text":"opae_hash_map opae_vfio::cont_buffers;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_device","title":"variable cont_device","text":"char* opae_vfio::cont_device;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_fd","title":"variable cont_fd","text":"int opae_vfio::cont_fd;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_pciaddr","title":"variable cont_pciaddr","text":"char* opae_vfio::cont_pciaddr;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_ranges","title":"variable cont_ranges","text":"struct opae_vfio_iova_range* opae_vfio::cont_ranges;\n
"},{"location":"opae-code/structopae__vfio/#variable-device","title":"variable device","text":"struct opae_vfio_device opae_vfio::device;\n
"},{"location":"opae-code/structopae__vfio/#variable-group","title":"variable group","text":"struct opae_vfio_group opae_vfio::group;\n
"},{"location":"opae-code/structopae__vfio/#variable-iova_alloc","title":"variable iova_alloc","text":"struct mem_alloc opae_vfio::iova_alloc;\n
"},{"location":"opae-code/structopae__vfio/#variable-lock","title":"variable lock","text":"pthread_mutex_t opae_vfio::lock;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__buffer/","title":"Struct opae_vfio_buffer","text":"ClassList > opae_vfio_buffer
System DMA buffer. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__buffer/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t buffer_iova Buffer IOVA address. uint8_t * buffer_ptr Buffer virtual address. size_t buffer_size Buffer size. int flags See opae_vfio_buffer_flags."},{"location":"opae-code/structopae__vfio__buffer/#detailed-description","title":"Detailed Description","text":"Describes a system memory address space that is capable of DMA.
"},{"location":"opae-code/structopae__vfio__buffer/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__buffer/#variable-buffer_iova","title":"variable buffer_iova","text":"uint64_t opae_vfio_buffer::buffer_iova;\n
"},{"location":"opae-code/structopae__vfio__buffer/#variable-buffer_ptr","title":"variable buffer_ptr","text":"uint8_t* opae_vfio_buffer::buffer_ptr;\n
"},{"location":"opae-code/structopae__vfio__buffer/#variable-buffer_size","title":"variable buffer_size","text":"size_t opae_vfio_buffer::buffer_size;\n
"},{"location":"opae-code/structopae__vfio__buffer/#variable-flags","title":"variable flags","text":"int opae_vfio_buffer::flags;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__device/","title":"Struct opae_vfio_device","text":"ClassList > opae_vfio_device
VFIO device. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__device/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t device_config_offset Offset of PCIe config space. int device_fd Device file descriptor. uint32_t device_num_irqs IRQ index count. uint32_t device_num_regions Total MMIO region count. struct opae_vfio_device_irq * irqs IRQ list pointer. struct opae_vfio_device_region * regions Region list pointer."},{"location":"opae-code/structopae__vfio__device/#detailed-description","title":"Detailed Description","text":"Each VFIO device has a file descriptor that is used to query information about the device MMIO regions and config space.
"},{"location":"opae-code/structopae__vfio__device/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__device/#variable-device_config_offset","title":"variable device_config_offset","text":"uint64_t opae_vfio_device::device_config_offset;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-device_fd","title":"variable device_fd","text":"int opae_vfio_device::device_fd;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-device_num_irqs","title":"variable device_num_irqs","text":"uint32_t opae_vfio_device::device_num_irqs;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-device_num_regions","title":"variable device_num_regions","text":"uint32_t opae_vfio_device::device_num_regions;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-irqs","title":"variable irqs","text":"struct opae_vfio_device_irq* opae_vfio_device::irqs;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-regions","title":"variable regions","text":"struct opae_vfio_device_region* opae_vfio_device::regions;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__device__irq/","title":"Struct opae_vfio_device_irq","text":"ClassList > opae_vfio_device_irq
Interrupt info. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__device__irq/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t count Number of IRQs at this index. int32_t * event_fds Event file descriptors. uint32_t flags Flags. uint32_t index The IRQ index. int32_t * masks IRQ masks. struct opae_vfio_device_irq * next Pointer to next in list."},{"location":"opae-code/structopae__vfio__device__irq/#detailed-description","title":"Detailed Description","text":"Describes an interrupt capability.
"},{"location":"opae-code/structopae__vfio__device__irq/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__device__irq/#variable-count","title":"variable count","text":"uint32_t opae_vfio_device_irq::count;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-event_fds","title":"variable event_fds","text":"int32_t* opae_vfio_device_irq::event_fds;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-flags","title":"variable flags","text":"Flags.
uint32_t opae_vfio_device_irq::flags;\n
See struct vfio_irq_info.
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-index","title":"variable index","text":"uint32_t opae_vfio_device_irq::index;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-masks","title":"variable masks","text":"int32_t* opae_vfio_device_irq::masks;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-next","title":"variable next","text":"struct opae_vfio_device_irq* opae_vfio_device_irq::next;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__device__region/","title":"Struct opae_vfio_device_region","text":"ClassList > opae_vfio_device_region
MMIO region info. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__device__region/#public-attributes","title":"Public Attributes","text":"Type Name struct opae_vfio_device_region * next Pointer to next in list. uint32_t region_index Region index, from 0. uint8_t * region_ptr Virtual address of region. size_t region_size Size of region. struct opae_vfio_sparse_info * region_sparse For sparse regions."},{"location":"opae-code/structopae__vfio__device__region/#detailed-description","title":"Detailed Description","text":"Describes a range of mappable MMIO.
"},{"location":"opae-code/structopae__vfio__device__region/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__device__region/#variable-next","title":"variable next","text":"struct opae_vfio_device_region* opae_vfio_device_region::next;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_index","title":"variable region_index","text":"uint32_t opae_vfio_device_region::region_index;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_ptr","title":"variable region_ptr","text":"uint8_t* opae_vfio_device_region::region_ptr;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_size","title":"variable region_size","text":"size_t opae_vfio_device_region::region_size;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_sparse","title":"variable region_sparse","text":"struct opae_vfio_sparse_info* opae_vfio_device_region::region_sparse;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__group/","title":"Struct opae_vfio_group","text":"ClassList > opae_vfio_group
VFIO group. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__group/#public-attributes","title":"Public Attributes","text":"Type Name char * group_device Full path to the group device. int group_fd File descriptor for the group device."},{"location":"opae-code/structopae__vfio__group/#detailed-description","title":"Detailed Description","text":"Each device managed by vfio-pci belongs to a VFIO group rooted at /dev/vfio/N, where N is the group number.
"},{"location":"opae-code/structopae__vfio__group/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__group/#variable-group_device","title":"variable group_device","text":"char* opae_vfio_group::group_device;\n
"},{"location":"opae-code/structopae__vfio__group/#variable-group_fd","title":"variable group_fd","text":"int opae_vfio_group::group_fd;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__iova__range/","title":"Struct opae_vfio_iova_range","text":"ClassList > opae_vfio_iova_range
IO Virtual Address Range. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__iova__range/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t end End of this range of offsets. struct opae_vfio_iova_range * next Pointer to next in list. uint64_t start Start of this range of offsets."},{"location":"opae-code/structopae__vfio__iova__range/#detailed-description","title":"Detailed Description","text":"A range of allocatable IOVA offsets. Used for mapping DMA buffers.
"},{"location":"opae-code/structopae__vfio__iova__range/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__iova__range/#variable-end","title":"variable end","text":"uint64_t opae_vfio_iova_range::end;\n
"},{"location":"opae-code/structopae__vfio__iova__range/#variable-next","title":"variable next","text":"struct opae_vfio_iova_range* opae_vfio_iova_range::next;\n
"},{"location":"opae-code/structopae__vfio__iova__range/#variable-start","title":"variable start","text":"uint64_t opae_vfio_iova_range::start;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__sparse__info/","title":"Struct opae_vfio_sparse_info","text":"ClassList > opae_vfio_sparse_info
MMIO sparse-mappable region info. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__sparse__info/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t index Region index, from 0. struct opae_vfio_sparse_info * next Pointer to next in list. uint32_t offset Offset of sparse region. uint8_t * ptr Virtual address of sparse region. uint32_t size Size of sparse region."},{"location":"opae-code/structopae__vfio__sparse__info/#detailed-description","title":"Detailed Description","text":"Describes a range of sparse-mappable MMIO, for MMIO ranges that have non-contiguous addresses.
"},{"location":"opae-code/structopae__vfio__sparse__info/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__sparse__info/#variable-index","title":"variable index","text":"uint32_t opae_vfio_sparse_info::index;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-next","title":"variable next","text":"struct opae_vfio_sparse_info* opae_vfio_sparse_info::next;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-offset","title":"variable offset","text":"uint32_t opae_vfio_sparse_info::offset;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-ptr","title":"variable ptr","text":"uint8_t* opae_vfio_sparse_info::ptr;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-size","title":"variable size","text":"uint32_t opae_vfio_sparse_info::size;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structras__inject__error/","title":"Struct ras_inject_error","text":"ClassList > ras_inject_error
"},{"location":"opae-code/structras__inject__error/#public-attributes","title":"Public Attributes","text":"Type Name union ras_inject_error::@0 @1 uint64_t catastrophicr_error uint64_t csr uint64_t fatal_error uint64_t nonfatal_error uint64_t rsvd"},{"location":"opae-code/structras__inject__error/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structras__inject__error/#variable-1","title":"variable @1","text":"union ras_inject_error::@0 ras_inject_error::@1;\n
"},{"location":"opae-code/structras__inject__error/#variable-catastrophicr_error","title":"variable catastrophicr_error","text":"uint64_t ras_inject_error::catastrophicr_error;\n
"},{"location":"opae-code/structras__inject__error/#variable-csr","title":"variable csr","text":"uint64_t ras_inject_error::csr;\n
"},{"location":"opae-code/structras__inject__error/#variable-fatal_error","title":"variable fatal_error","text":"uint64_t ras_inject_error::fatal_error;\n
"},{"location":"opae-code/structras__inject__error/#variable-nonfatal_error","title":"variable nonfatal_error","text":"uint64_t ras_inject_error::nonfatal_error;\n
"},{"location":"opae-code/structras__inject__error/#variable-rsvd","title":"variable rsvd","text":"uint64_t ras_inject_error::rsvd;\n
The documentation for this class was generated from the following file docs/sw/samples/hello_events/hello_events.c
"},{"location":"opae-code/namespacestd/","title":"Namespace std","text":"Namespace List > std
The documentation for this class was generated from the following file [generated]
"},{"location":"opae-code/structthreshold/","title":"Struct threshold","text":"ClassList > threshold
Threshold struct.
#include <types.h>
"},{"location":"opae-code/structthreshold/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t is_valid char threshold_name double value"},{"location":"opae-code/structthreshold/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structthreshold/#variable-is_valid","title":"variable is_valid","text":"uint32_t threshold::is_valid;\n
"},{"location":"opae-code/structthreshold/#variable-threshold_name","title":"variable threshold_name","text":"char threshold::threshold_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structthreshold/#variable-value","title":"variable value","text":"double threshold::value;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/dir_49e56c817e5e54854c35e136979f97ca/","title":"Dir docs","text":"FileList > docs
"},{"location":"opae-code/dir_49e56c817e5e54854c35e136979f97ca/#directories","title":"Directories","text":"Type Name dir sw The documentation for this class was generated from the following file docs/
"},{"location":"opae-code/dir_55721a669a8e0900d975c02921addb49/","title":"Dir docs/sw","text":"FileList > docs > sw
"},{"location":"opae-code/dir_55721a669a8e0900d975c02921addb49/#directories","title":"Directories","text":"Type Name dir include dir samples The documentation for this class was generated from the following file docs/sw/
"},{"location":"opae-code/dir_97b4588afba69bf89bbe554642ac6431/","title":"Dir docs/sw/include","text":"FileList > docs > sw > include
"},{"location":"opae-code/dir_97b4588afba69bf89bbe554642ac6431/#directories","title":"Directories","text":"Type Name dir opae The documentation for this class was generated from the following file docs/sw/include/
"},{"location":"opae-code/dir_ade97cd9199f278c0723672dd8647ba4/","title":"Dir docs/sw/include/opae","text":"FileList > docs > sw > include > opae
"},{"location":"opae-code/dir_ade97cd9199f278c0723672dd8647ba4/#files","title":"Files","text":"Type Name file access.h Functions to acquire, release, and reset OPAE FPGA resources. file buffer.h Functions for allocating and sharing system memory with an FPGA accelerator. file enum.h APIs for resource enumeration and managing tokens. file error.h Functions for reading and clearing errors in resources. file event.h Functions for registering events and managing the lifecycle for fpga_event_handle s. file fpga.h FPGA API. file hash_map.h A general-purpose hybrid array/list hash map implementation. file init.h Initialization routine. file log.h file manage.h Functions for managing FPGA configurations. file mem_alloc.h file metrics.h Functions for Discover/ Enumerates metrics and retrieves values. file mmio.h Functions for mapping and accessing MMIO space. file properties.h Functions for examining and manipulating fpga_properties objects. file sysobject.h Functions to read/write from system objects. file types.h Type definitions for FPGA API. file types_enum.h Definitions of enumerated types for the OPAE C API. file uio.h APIs to manage a PCIe device via UIO. file umsg.h FPGA UMsg API. file userclk.h Functions for setting and get afu user clock. file utils.h Utility functions and macros for the FPGA API. file version.h file vfio.h APIs to manage a PCIe device via vfio-pci."},{"location":"opae-code/dir_ade97cd9199f278c0723672dd8647ba4/#directories","title":"Directories","text":"Type Name dir cxx The documentation for this class was generated from the following file docs/sw/include/opae/
"},{"location":"opae-code/access_8h/","title":"File access.h","text":"FileList > docs > sw > include > opae > access.h
Go to the source code of this file.
Functions to acquire, release, and reset OPAE FPGA resources.
#include <opae/types.h>
"},{"location":"opae-code/access_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaClose (fpga_handle handle) Close a previously opened FPGA object. fpga_result fpgaOpen (fpga_token token, fpga_handle * handle, int flags) Open an FPGA object. fpga_result fpgaReset (fpga_handle handle) Reset an FPGA object."},{"location":"opae-code/access_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/access_8h/#function-fpgaclose","title":"function fpgaClose","text":"Close a previously opened FPGA object.
fpga_result fpgaClose (\nfpga_handle handle\n)
Relinquishes ownership of a previously fpgaOpen()ed resource. This enables others to acquire ownership if the resource was opened exclusively. Also deallocates / unmaps MMIO and UMsg memory areas.
Parameters:
handle Handle to previously opened FPGA object
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to an acquired resource, or if handle is NULL. FPGA_EXCEPTION if an internal error occurred while accessing the handle.
"},{"location":"opae-code/access_8h/#function-fpgaopen","title":"function fpgaOpen","text":"Open an FPGA object.
fpga_result fpgaOpen (\nfpga_token token,\nfpga_handle * handle,\nint flags\n)
Acquires ownership of the FPGA resource referred to by 'token'.
Most often this will be used to open an accelerator object to directly interact with an accelerator function, or to open an FPGA object to perform management functions.
Parameters:
token Pointer to token identifying resource to acquire ownership of handle Pointer to preallocated memory to place a handle in. This handle will be used in subsequent API calls. flags One of the following flags: - FPGA_OPEN_SHARED allows the resource to be opened multiple times (not supported in ASE) Shared resources (including buffers) are released when all associated handles have been closed (either explicitly with fpgaClose() or by process termination).
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the resource for 'token' could not be found. FPGA_INVALID_PARAM if 'token' does not refer to a resource that can be opened, or if either argument is NULL or invalid. FPGA_EXCEPTION if an internal exception occurred while creating the handle. FPGA_NO_DRIVER if the driver is not loaded. FPGA_BUSY if trying to open a resource that has already been opened in exclusive mode. FPGA_NO_ACCESS if the current process' privileges are not sufficient to open the resource.
"},{"location":"opae-code/access_8h/#function-fpgareset","title":"function fpgaReset","text":"Reset an FPGA object.
fpga_result fpgaReset (\nfpga_handle handle\n)
Performs an accelerator reset.
Parameters:
handle Handle to previously opened FPGA object
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to an acquired resource or to a resource that cannot be reset. FPGA_EXCEPTION if an internal error occurred while trying to access the handle or resetting the resource.
The documentation for this class was generated from the following file docs/sw/include/opae/access.h
"},{"location":"opae-code/access_8h_source/","title":"File access.h","text":"File List > docs > sw > include > opae > access.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_ACCESS_H__\n#define __FPGA_ACCESS_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaOpen(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result fpgaClose(fpga_handle handle);\nfpga_result fpgaReset(fpga_handle handle);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_ACCESS_H__\n
"},{"location":"opae-code/buffer_8h/","title":"File buffer.h","text":"FileList > docs > sw > include > opae > buffer.h
Go to the source code of this file.
Functions for allocating and sharing system memory with an FPGA accelerator. More...
#include <opae/types.h>
"},{"location":"opae-code/buffer_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetIOAddress (fpga_handle handle, uint64_t wsid, uint64_t * ioaddr) Retrieve base IO address for buffer. fpga_result fpgaPrepareBuffer (fpga_handle handle, uint64_t len, void ** buf_addr, uint64_t * wsid, int flags) Prepare a shared memory buffer. fpga_result fpgaReleaseBuffer (fpga_handle handle, uint64_t wsid) Release a shared memory buffer."},{"location":"opae-code/buffer_8h/#detailed-description","title":"Detailed Description","text":"To share memory between a software application and an FPGA accelerator, these functions set up system components (e.g. an IOMMU) to allow accelerator access to a provided memory region.
There are a number of restrictions on what memory can be shared, depending on platform capabilities. Usually, FPGA accelerators to not have access to virtual address mappings of the CPU, so they can only access physical addresses. To support this, the OPAE C library on Linux uses hugepages to allocate large, contiguous pages of physical memory that can be shared with an accelerator. It also supports sharing memory that has already been allocated by an application, as long as that memory satisfies the requirements of being physically contigous and page-aligned.
"},{"location":"opae-code/buffer_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/buffer_8h/#function-fpgagetioaddress","title":"function fpgaGetIOAddress","text":"Retrieve base IO address for buffer.
fpga_result fpgaGetIOAddress (\nfpga_handle handle,\nuint64_t wsid,\nuint64_t * ioaddr\n)
This function is used to acquire the physical base address (on some platforms called IO Virtual Address or IOVA) for a shared buffer identified by wsid.
Note:
This function will disappear once the APIs for secure sharing of buffer addresses is implemented.
Parameters:
handle Handle to previously opened accelerator resource wsid Buffer handle / workspace ID referring to the buffer for which the IO address is requested ioaddr Pointer to memory where the IO address will be returned
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle. FPGA_NOT_FOUND if wsid does not refer to a previously shared buffer.
"},{"location":"opae-code/buffer_8h/#function-fpgapreparebuffer","title":"function fpgaPrepareBuffer","text":"Prepare a shared memory buffer.
fpga_result fpgaPrepareBuffer (\nfpga_handle handle,\nuint64_t len,\nvoid ** buf_addr,\nuint64_t * wsid,\nint flags\n)
Prepares a memory buffer for shared access between an accelerator and the calling process. This may either include allocation of physical memory, or preparation of already allocated memory for sharing. The latter case is indicated by supplying the FPGA_BUF_PREALLOCATED flag.
This function will ask the driver to pin the indicated memory (make it non-swappable), and program the IOMMU to allow access from the accelerator. If the buffer was not pre-allocated (flag FPGA_BUF_PREALLOCATED), the function will also allocate physical memory of the requested size and map the memory into the caller's process' virtual address space. It returns in 'wsid' an fpga_buffer object that can be used to program address registers in the accelerator for shared access to the memory.
When using FPGA_BUF_PREALLOCATED, the input len must be a non-zero multiple of the page size, else the function returns FPGA_INVALID_PARAM. When not using FPGA_BUF_PREALLOCATED, the input len is rounded up to the nearest multiple of page size.
Parameters:
handle Handle to previously opened accelerator resource len Length of the buffer to allocate/prepare in bytes buf_addr Virtual address of buffer. Contents may be NULL (OS will choose mapping) or non-NULL (OS will take contents as a hint for the virtual address). wsid Handle to the allocated/prepared buffer to be used with other functions flags Flags. FPGA_BUF_PREALLOCATED indicates that memory pointed at in '*buf_addr' is already allocated an mapped into virtual memory. FPGA_BUF_READ_ONLY pins pages with only read access from the FPGA.
Returns:
FPGA_OK on success. FPGA_NO_MEMORY if the requested memory could not be allocated. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
Note:
As a special case, when FPGA_BUF_PREALLOCATED is present in flags, if len == 0 and buf_addr == NULL, then the function returns FPGA_OK if pre-allocated buffers are supported. In this case, a return value other than FPGA_OK indicates that pre-allocated buffers are not supported.
"},{"location":"opae-code/buffer_8h/#function-fpgareleasebuffer","title":"function fpgaReleaseBuffer","text":"Release a shared memory buffer.
fpga_result fpgaReleaseBuffer (\nfpga_handle handle,\nuint64_t wsid\n)
Releases a previously prepared shared buffer. If the buffer was allocated using fpgaPrepareBuffer (FPGA_BUF_PREALLOCATED was not specified), this call will deallocate/free that memory. Otherwise, it will only be returned to it's previous state (pinned/unpinned, cached/non-cached).
Parameters:
handle Handle to previously opened accelerator resource wsid Handle to the allocated/prepared buffer
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
The documentation for this class was generated from the following file docs/sw/include/opae/buffer.h
"},{"location":"opae-code/buffer_8h_source/","title":"File buffer.h","text":"File List > docs > sw > include > opae > buffer.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_BUFFER_H__\n#define __FPGA_BUFFER_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaPrepareBuffer(fpga_handle handle,\nuint64_t len,\nvoid **buf_addr, uint64_t *wsid, int flags);\nfpga_result fpgaReleaseBuffer(fpga_handle handle, uint64_t wsid);\nfpga_result fpgaGetIOAddress(fpga_handle handle, uint64_t wsid,\nuint64_t *ioaddr);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_BUFFER_H__\n
"},{"location":"opae-code/dir_3731a7e5669218b938d292e51b4e531c/","title":"Dir docs/sw/include/opae/cxx","text":"FileList > cxx
"},{"location":"opae-code/dir_3731a7e5669218b938d292e51b4e531c/#files","title":"Files","text":"Type Name file core.h"},{"location":"opae-code/dir_3731a7e5669218b938d292e51b4e531c/#directories","title":"Directories","text":"Type Name dir core The documentation for this class was generated from the following file docs/sw/include/opae/cxx/
"},{"location":"opae-code/core_8h/","title":"File core.h","text":"FileList > cxx > core.h
Go to the source code of this file.
#include <opae/cxx/core/errors.h>#include <opae/cxx/core/events.h>#include <opae/cxx/core/except.h>#include <opae/cxx/core/handle.h>#include <opae/cxx/core/properties.h>#include <opae/cxx/core/pvalue.h>#include <opae/cxx/core/shared_buffer.h>#include <opae/cxx/core/token.h>#include <opae/cxx/core/version.h>
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core.h
"},{"location":"opae-code/core_8h_source/","title":"File core.h","text":"File List > cxx > core.h
Go to the documentation of this file.
// Copyright(c) 2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/errors.h>\n#include <opae/cxx/core/events.h>\n#include <opae/cxx/core/except.h>\n#include <opae/cxx/core/handle.h>\n#include <opae/cxx/core/properties.h>\n#include <opae/cxx/core/pvalue.h>\n#include <opae/cxx/core/shared_buffer.h>\n#include <opae/cxx/core/token.h>\n#include <opae/cxx/core/version.h>\n
"},{"location":"opae-code/dir_23b1b9d7ef54caa3fa7bb54d9bc2d47a/","title":"Dir docs/sw/include/opae/cxx/core","text":"FileList > core
"},{"location":"opae-code/dir_23b1b9d7ef54caa3fa7bb54d9bc2d47a/#files","title":"Files","text":"Type Name file errors.h file events.h file except.h file handle.h file properties.h file pvalue.h file shared_buffer.h file sysobject.h file token.h file version.h The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/
"},{"location":"opae-code/errors_8h/","title":"File errors.h","text":"FileList > core > errors.h
Go to the source code of this file.
#include <opae/cxx/core/token.h>#include <opae/types_enum.h>#include <memory>
"},{"location":"opae-code/errors_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/errors_8h/#classes","title":"Classes","text":"Type Name class error An error object represents an error register for a resource. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/errors_8h_source/","title":"File errors.h","text":"File List > core > errors.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/token.h>\n#include <opae/types_enum.h>\n#include <memory>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass error {\npublic:\ntypedef std::shared_ptr<error> ptr_t;\nerror(const error &e) = delete;\nerror &operator=(const error &e) = delete;\nstatic error::ptr_t get(token::ptr_t tok, uint32_t num);\nstd::string name() { return error_info_.name; }\nbool can_clear() { return error_info_.can_clear; }\nuint64_t read_value();\n~error() {}\nfpga_error_info c_type() const { return error_info_; }\nprivate:\nerror(token::ptr_t token, uint32_t num);\ntoken::ptr_t token_;\nfpga_error_info error_info_;\nuint32_t error_num_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/events_8h/","title":"File events.h","text":"FileList > core > events.h
Go to the source code of this file.
#include <opae/cxx/core/handle.h>#include <opae/types_enum.h>#include <memory>
"},{"location":"opae-code/events_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/events_8h/#classes","title":"Classes","text":"Type Name class event Wraps fpga event routines in OPAE C. struct type_t C++ struct that is interchangeable with fpga_event_type enum. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/events.h
"},{"location":"opae-code/events_8h_source/","title":"File events.h","text":"File List > core > events.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/handle.h>\n#include <opae/types_enum.h>\n#include <memory>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass event {\npublic:\ntypedef std::shared_ptr<event> ptr_t;\nvirtual ~event();\nstruct type_t {\ntype_t(fpga_event_type c_type) : type_(c_type) {}\noperator fpga_event_type() { return type_; }\nstatic constexpr fpga_event_type interrupt = FPGA_EVENT_INTERRUPT;\nstatic constexpr fpga_event_type error = FPGA_EVENT_ERROR;\nstatic constexpr fpga_event_type power_thermal = FPGA_EVENT_POWER_THERMAL;\nprivate:\nfpga_event_type type_;\n};\nfpga_event_handle get() { return event_handle_; }\noperator fpga_event_handle();\nstatic event::ptr_t register_event(handle::ptr_t h, event::type_t t,\nint flags = 0);\nint os_object() const;\nprivate:\nevent(handle::ptr_t h, event::type_t t, fpga_event_handle event_h);\nhandle::ptr_t handle_;\nevent::type_t type_;\nfpga_event_handle event_handle_;\nint os_object_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/except_8h/","title":"File except.h","text":"FileList > core > except.h
Go to the source code of this file.
#include <opae/types_enum.h>#include <cstdint>#include <exception>
"},{"location":"opae-code/except_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types namespace detail"},{"location":"opae-code/except_8h/#classes","title":"Classes","text":"Type Name class busy busy exception class except Generic OPAE exception. class exception exception exception class invalid_param invalid_param exception class no_access no_access exception class no_daemon no_daemon exception class no_driver no_driver exception class no_memory no_memory exception class not_found not_found exception class not_supported not_supported exception class reconf_error reconf_error exception class src_location Identify a particular line in a source file."},{"location":"opae-code/except_8h/#macros","title":"Macros","text":"Type Name define ASSERT_FPGA_OK \u00ae Macro to check of result is FPGA_OK If not, throw exception that corresponds to the result code. define OPAECXX_HERE opae::fpga::types::src_location(__FILE__, __func__, __LINE__)Construct a src_location object for the current source line."},{"location":"opae-code/except_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/except_8h/#define-assert_fpga_ok","title":"define ASSERT_FPGA_OK","text":"#define ASSERT_FPGA_OK (\nr\n) opae::fpga::types::detail::assert_fpga_ok( \\\n r, opae::fpga::types::src_location (__FILE__, __func__, __LINE__));\n
"},{"location":"opae-code/except_8h/#define-opaecxx_here","title":"define OPAECXX_HERE","text":"#define OPAECXX_HERE opae::fpga::types::src_location (__FILE__, __func__, __LINE__)\n
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/except_8h_source/","title":"File except.h","text":"File List > core > except.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/types_enum.h>\n#include <cstdint>\n#include <exception>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass src_location {\npublic:\nsrc_location(const char *file, const char *fn, int line) noexcept;\nsrc_location(const src_location &other) noexcept;\nsrc_location &operator=(const src_location &other) noexcept;\nconst char *file() const noexcept;\nconst char *fn() const noexcept { return fn_; }\nint line() const noexcept { return line_; }\nprivate:\nconst char *file_;\nconst char *fn_;\nint line_;\n};\n#define OPAECXX_HERE \\\n opae::fpga::types::src_location(__FILE__, __func__, __LINE__)\nclass except : public std::exception {\npublic:\nstatic const std::size_t MAX_EXCEPT = 256;\nexcept(src_location loc) noexcept;\nexcept(fpga_result res, src_location loc) noexcept;\nexcept(fpga_result res, const char *msg, src_location loc) noexcept;\nvirtual const char *what() const noexcept override;\noperator fpga_result() const noexcept { return res_; }\nprotected:\nfpga_result res_;\nconst char *msg_;\nsrc_location loc_;\nmutable char buf_[MAX_EXCEPT];\n};\nclass invalid_param : public except {\npublic:\ninvalid_param(src_location loc) noexcept\n: except(FPGA_INVALID_PARAM, \"failed with return code FPGA_INVALID_PARAM\",\nloc) {}\n};\nclass busy : public except {\npublic:\nbusy(src_location loc) noexcept\n: except(FPGA_BUSY, \"failed with return code FPGA_BUSY\", loc) {}\n};\nclass exception : public except {\npublic:\nexception(src_location loc) noexcept\n: except(FPGA_EXCEPTION, \"failed with return code FPGA_EXCEPTION\", loc) {}\n};\nclass not_found : public except {\npublic:\nnot_found(src_location loc) noexcept\n: except(FPGA_NOT_FOUND, \"failed with return code FPGA_NOT_FOUND\", loc) {}\n};\nclass no_memory : public except {\npublic:\nno_memory(src_location loc) noexcept\n: except(FPGA_NO_MEMORY, \"failed with return code FPGA_NO_MEMORY\", loc) {}\n};\nclass not_supported : public except {\npublic:\nnot_supported(src_location loc) noexcept\n: except(FPGA_NOT_SUPPORTED, \"failed with return code FPGA_NOT_SUPPORTED\",\nloc) {}\n};\nclass no_driver : public except {\npublic:\nno_driver(src_location loc) noexcept\n: except(FPGA_NO_DRIVER, \"failed with return code FPGA_NO_DRIVER\", loc) {}\n};\nclass no_daemon : public except {\npublic:\nno_daemon(src_location loc) noexcept\n: except(FPGA_NO_DAEMON, \"failed with return code FPGA_NO_DAEMON\", loc) {}\n};\nclass no_access : public except {\npublic:\nno_access(src_location loc) noexcept\n: except(FPGA_NO_ACCESS, \"failed with return code FPGA_NO_ACCESS\", loc) {}\n};\nclass reconf_error : public except {\npublic:\nreconf_error(src_location loc) noexcept\n: except(FPGA_RECONF_ERROR, \"failed with return code FPGA_RECONF_ERROR\",\nloc) {}\n};\nnamespace detail {\ntypedef bool (*exception_fn)(fpga_result,\nconst opae::fpga::types::src_location &loc);\ntemplate <typename T>\nconstexpr bool is_ok(fpga_result result,\nconst opae::fpga::types::src_location &loc) {\nreturn result == FPGA_OK ? true : throw T(loc);\n}\nstatic exception_fn opae_exceptions[12] = {\nis_ok<opae::fpga::types::invalid_param>,\nis_ok<opae::fpga::types::busy>,\nis_ok<opae::fpga::types::exception>,\nis_ok<opae::fpga::types::not_found>,\nis_ok<opae::fpga::types::no_memory>,\nis_ok<opae::fpga::types::not_supported>,\nis_ok<opae::fpga::types::no_driver>,\nis_ok<opae::fpga::types::no_daemon>,\nis_ok<opae::fpga::types::no_access>,\nis_ok<opae::fpga::types::reconf_error>};\nstatic inline void assert_fpga_ok(fpga_result result,\nconst opae::fpga::types::src_location &loc) {\nif (result > FPGA_OK && result <= FPGA_RECONF_ERROR)\n// our exception table above starts at invalid_param with index 0\n// but FPGA_INVALID_PARAM is actually enum 1 - let's account for that\nopae_exceptions[result - 1](result, loc);\n}\n} // end of namespace detail\n#define ASSERT_FPGA_OK(r) \\\n opae::fpga::types::detail::assert_fpga_ok( \\\n r, opae::fpga::types::src_location(__FILE__, __func__, __LINE__));\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/handle_8h/","title":"File handle.h","text":"FileList > core > handle.h
Go to the source code of this file.
#include <opae/cxx/core/token.h>#include <opae/enum.h>#include <opae/types.h>#include <memory>#include <vector>
"},{"location":"opae-code/handle_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/handle_8h/#classes","title":"Classes","text":"Type Name class handle An allocated accelerator resource. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/handle.h
"},{"location":"opae-code/handle_8h_source/","title":"File handle.h","text":"File List > core > handle.h
Go to the documentation of this file.
// Copyright(c) 2018-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/token.h>\n#include <opae/enum.h>\n#include <opae/types.h>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass handle {\npublic:\ntypedef std::shared_ptr<handle> ptr_t;\nhandle(const handle &) = delete;\nhandle &operator=(const handle &) = delete;\nvirtual ~handle();\nfpga_handle c_type() const { return handle_; }\noperator fpga_handle() const { return handle_; }\nvoid reconfigure(uint32_t slot, const uint8_t *bitstream, size_t size,\nint flags);\nuint32_t read_csr32(uint64_t offset, uint32_t csr_space = 0) const;\nvoid write_csr32(uint64_t offset, uint32_t value, uint32_t csr_space = 0);\nuint64_t read_csr64(uint64_t offset, uint32_t csr_space = 0) const;\nvoid write_csr64(uint64_t offset, uint64_t value, uint32_t csr_space = 0);\nvoid write_csr512(uint64_t offset, const void *value, uint32_t csr_space = 0);\nuint8_t *mmio_ptr(uint64_t offset, uint32_t csr_space = 0) const;\nstatic handle::ptr_t open(fpga_token token, int flags);\nstatic handle::ptr_t open(token::ptr_t token, int flags);\nvirtual void reset();\nfpga_result close();\ntoken::ptr_t get_token() const;\nprivate:\nhandle(fpga_handle h);\nfpga_handle handle_;\nfpga_token token_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/cxx_2core_2properties_8h/","title":"File properties.h","text":"FileList > core > properties.h
Go to the source code of this file.
#include <opae/cxx/core/pvalue.h>#include <opae/properties.h>#include <iostream>#include <map>#include <memory>#include <vector>
"},{"location":"opae-code/cxx_2core_2properties_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/cxx_2core_2properties_8h/#classes","title":"Classes","text":"Type Name class properties Wraps an OPAE fpga_properties object. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/properties.h
"},{"location":"opae-code/cxx_2core_2properties_8h_source/","title":"File properties.h","text":"File List > core > properties.h
Go to the documentation of this file.
// Copyright(c) 2018-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/pvalue.h>\n#include <opae/properties.h>\n#include <iostream>\n#include <map>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass token;\nclass handle;\nclass properties {\npublic:\ntypedef std::shared_ptr<properties> ptr_t;\nconst static std::vector<properties::ptr_t> none;\nproperties(const properties &p) = delete;\nproperties &operator=(const properties &p) = delete;\n~properties();\nfpga_properties c_type() const { return props_; }\noperator fpga_properties() const { return props_; }\nstatic properties::ptr_t get();\nstatic properties::ptr_t get(fpga_guid guid_in);\nstatic properties::ptr_t get(fpga_objtype objtype);\nstatic properties::ptr_t get(std::shared_ptr<token> t);\nstatic properties::ptr_t get(fpga_token t);\nstatic properties::ptr_t get(std::shared_ptr<handle> h);\nprivate:\nproperties(bool alloc_props = true);\nfpga_properties props_;\npublic:\npvalue<fpga_objtype> type;\npvalue<uint32_t> num_errors;\npvalue<uint16_t> segment;\npvalue<uint8_t> bus;\npvalue<uint8_t> device;\npvalue<uint8_t> function;\npvalue<uint8_t> socket_id;\npvalue<uint32_t> num_slots;\npvalue<uint64_t> bbs_id;\npvalue<fpga_version> bbs_version;\npvalue<uint16_t> vendor_id;\npvalue<uint16_t> device_id;\npvalue<uint16_t> subsystem_vendor_id;\npvalue<uint16_t> subsystem_device_id;\npvalue<char *> model;\npvalue<uint64_t> local_memory_size;\npvalue<uint64_t> capabilities;\npvalue<uint32_t> num_mmio;\npvalue<uint32_t> num_interrupts;\npvalue<fpga_accelerator_state> accelerator_state;\npvalue<uint64_t> object_id;\npvalue<fpga_token> parent;\npvalue<fpga_interface> interface;\nguid_t guid;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/pvalue_8h/","title":"File pvalue.h","text":"FileList > core > pvalue.h
Go to the source code of this file.
#include <opae/cxx/core/except.h>#include <opae/properties.h>#include <opae/utils.h>#include <uuid/uuid.h>#include <algorithm>#include <array>#include <cstring>#include <iostream>#include <type_traits>
"},{"location":"opae-code/pvalue_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/pvalue_8h/#classes","title":"Classes","text":"Type Name struct guid_t Representation of the guid member of a properties object. struct pvalue <typename T>Wraps OPAE properties defined in the OPAE C API by associating an fpga_properties reference with the getters and setters defined for a property. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/pvalue.h
"},{"location":"opae-code/pvalue_8h_source/","title":"File pvalue.h","text":"File List > core > pvalue.h
Go to the documentation of this file.
// Copyright(c) 2018-2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/except.h>\n#include <opae/properties.h>\n#include <opae/utils.h>\n#include <uuid/uuid.h>\n#include <algorithm>\n#include <array>\n#include <cstring>\n#include <iostream>\n#include <type_traits>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nstruct guid_t {\nguid_t(fpga_properties *p) : props_(p), is_set_(false) {}\nvoid update() {\nfpga_result res = fpgaPropertiesGetGUID(\n*props_, reinterpret_cast<fpga_guid *>(data_.data()));\nASSERT_FPGA_OK(res);\nis_set_ = true;\n}\noperator uint8_t *() {\nupdate();\nreturn data_.data();\n}\nconst uint8_t *c_type() const { return data_.data(); }\nguid_t &operator=(fpga_guid g) {\nis_set_ = false;\nASSERT_FPGA_OK(fpgaPropertiesSetGUID(*props_, g));\nis_set_ = true;\nuint8_t *begin = &g[0];\nuint8_t *end = begin + sizeof(fpga_guid);\nstd::copy(begin, end, data_.begin());\nreturn *this;\n}\nbool operator==(const fpga_guid &g) {\nreturn is_set() && (0 == std::memcmp(data_.data(), g, sizeof(fpga_guid)));\n}\nvoid parse(const char *str) {\nis_set_ = false;\nif (0 != uuid_parse(str, data_.data())) {\nthrow except(OPAECXX_HERE);\n}\nASSERT_FPGA_OK(fpgaPropertiesSetGUID(*props_, data_.data()));\nis_set_ = true;\n}\nfriend std::ostream &operator<<(std::ostream &ostr, const guid_t &g) {\nfpga_properties props = *g.props_;\nfpga_guid guid_value;\nfpga_result res;\nif ((res = fpgaPropertiesGetGUID(props, &guid_value)) == FPGA_OK) {\nchar guid_str[84];\nuuid_unparse(guid_value, guid_str);\nostr << guid_str;\n} else if (FPGA_NOT_FOUND == res) {\nstd::cerr << \"[guid_t::<<] GUID property not set\\n\";\n} else {\nASSERT_FPGA_OK(res);\n}\nreturn ostr;\n}\nbool is_set() const { return is_set_; }\nvoid invalidate() { is_set_ = false; }\nprivate:\nfpga_properties *props_;\nbool is_set_;\nstd::array<uint8_t, 16> data_;\n};\ntemplate <typename T>\nstruct pvalue {\ntypedef typename std::conditional<\nstd::is_same<T, char *>::value, fpga_result (*)(fpga_properties, T),\nfpga_result (*)(fpga_properties, T *)>::type getter_t;\ntypedef fpga_result (*setter_t)(fpga_properties, T);\ntypedef typename std::conditional<std::is_same<T, char *>::value,\ntypename std::string, T>::type copy_t;\npvalue() : props_(0), is_set_(false), get_(nullptr), set_(nullptr), copy_() {}\npvalue(fpga_properties *p, getter_t g, setter_t s)\n: props_(p), is_set_(false), get_(g), set_(s), copy_() {}\npvalue<T> &operator=(const T &v) {\nis_set_ = false;\nASSERT_FPGA_OK(set_(*props_, v));\nis_set_ = true;\ncopy_ = v;\nreturn *this;\n}\nbool operator==(const T &other) { return is_set() && (copy_ == other); }\nvoid update() {\nASSERT_FPGA_OK(get_(*props_, ©_));\nis_set_ = true;\n}\noperator copy_t() {\nupdate();\nreturn copy_;\n}\n// TODO: Remove this once all properties are tested\nfpga_result get_value(T &value) const { return get_(*props_, &value); }\nfriend std::ostream &operator<<(std::ostream &ostr, const pvalue<T> &p) {\nT value;\nfpga_properties props = *p.props_;\nfpga_result res;\nif ((res = p.get_(props, &value)) == FPGA_OK) {\nostr << +(value);\n} else if (FPGA_NOT_FOUND == res) {\nstd::cerr << \"property getter returned (\" << res << \") \"\n<< fpgaErrStr(res);\n} else {\nASSERT_FPGA_OK(res);\n}\nreturn ostr;\n}\nbool is_set() const { return is_set_; }\nvoid invalidate() { is_set_ = false; }\nprivate:\nfpga_properties *props_;\nbool is_set_;\ngetter_t get_;\nsetter_t set_;\ncopy_t copy_;\n};\ntemplate <>\ninline void pvalue<char *>::update() {\nchar buf[256];\nASSERT_FPGA_OK(get_(*props_, buf));\ncopy_.assign(buf);\nis_set_ = true;\n}\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/shared__buffer_8h/","title":"File shared_buffer.h","text":"FileList > core > shared_buffer.h
Go to the source code of this file.
#include <opae/buffer.h>#include <opae/cxx/core/except.h>#include <opae/cxx/core/handle.h>#include <chrono>#include <cstdint>#include <initializer_list>#include <memory>#include <thread>#include <vector>
"},{"location":"opae-code/shared__buffer_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/shared__buffer_8h/#classes","title":"Classes","text":"Type Name class shared_buffer Host/AFU shared memory blocks. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/shared_buffer.h
"},{"location":"opae-code/shared__buffer_8h_source/","title":"File shared_buffer.h","text":"File List > core > shared_buffer.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/buffer.h>\n#include <opae/cxx/core/except.h>\n#include <opae/cxx/core/handle.h>\n#include <chrono>\n#include <cstdint>\n#include <initializer_list>\n#include <memory>\n#include <thread>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass shared_buffer {\npublic:\ntypedef std::size_t size_t;\ntypedef std::shared_ptr<shared_buffer> ptr_t;\nshared_buffer(const shared_buffer &) = delete;\nshared_buffer &operator=(const shared_buffer &) = delete;\nvirtual ~shared_buffer();\nstatic shared_buffer::ptr_t allocate(handle::ptr_t handle, size_t len,\nbool read_only = false);\nstatic shared_buffer::ptr_t attach(handle::ptr_t handle, uint8_t *base,\nsize_t len, bool read_only = false);\nvoid release();\nvolatile uint8_t *c_type() const { return virt_; }\nhandle::ptr_t owner() const { return handle_; }\nsize_t size() const { return len_; }\nuint64_t wsid() const { return wsid_; }\nuint64_t io_address() const { return io_address_; }\nvoid fill(int c);\nint compare(ptr_t other, size_t len) const;\ntemplate <typename T>\nT read(size_t offset) const {\nif ((offset < len_) && (virt_ != nullptr)) {\nreturn *reinterpret_cast<T *>(virt_ + offset);\n} else if (offset >= len_) {\nthrow except(OPAECXX_HERE);\n} else {\nthrow except(OPAECXX_HERE);\n}\nreturn T();\n}\ntemplate <typename T>\nvoid write(const T &value, size_t offset) {\nif ((offset < len_) && (virt_ != nullptr)) {\n*reinterpret_cast<T *>(virt_ + offset) = value;\n} else if (offset >= len_) {\nthrow except(OPAECXX_HERE);\n} else {\nthrow except(OPAECXX_HERE);\n}\n}\nprotected:\nshared_buffer(handle::ptr_t handle, size_t len, uint8_t *virt, uint64_t wsid,\nuint64_t io_address);\nhandle::ptr_t handle_;\nsize_t len_;\nuint8_t *virt_;\nuint64_t wsid_;\nuint64_t io_address_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/cxx_2core_2sysobject_8h/","title":"File sysobject.h","text":"FileList > core > sysobject.h
Go to the source code of this file.
#include <opae/cxx/core/handle.h>#include <opae/cxx/core/token.h>#include <opae/types.h>#include <memory>#include <vector>
"},{"location":"opae-code/cxx_2core_2sysobject_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/cxx_2core_2sysobject_8h/#classes","title":"Classes","text":"Type Name class sysobject Wraps the OPAE fpga_object primitive. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/sysobject.h
"},{"location":"opae-code/cxx_2core_2sysobject_8h_source/","title":"File sysobject.h","text":"File List > core > sysobject.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/handle.h>\n#include <opae/cxx/core/token.h>\n#include <opae/types.h>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass sysobject {\npublic:\ntypedef std::shared_ptr<sysobject> ptr_t;\nsysobject() = delete;\nsysobject(const sysobject &o) = delete;\nsysobject &operator=(const sysobject &o) = delete;\nstatic sysobject::ptr_t get(token::ptr_t t, const std::string &name,\nint flags = 0);\nstatic sysobject::ptr_t get(handle::ptr_t h, const std::string &name,\nint flags = 0);\nsysobject::ptr_t get(const std::string &name, int flags = 0);\nsysobject::ptr_t get(int index);\nvirtual ~sysobject();\nuint32_t size() const;\nuint64_t read64(int flags = 0) const;\nvoid write64(uint64_t value, int flags = 0) const;\nstd::vector<uint8_t> bytes(int flags = 0) const;\nstd::vector<uint8_t> bytes(uint32_t offset, uint32_t size,\nint flags = 0) const;\nenum fpga_sysobject_type type() const;\nfpga_object c_type() const { return sysobject_; }\noperator fpga_object() const { return sysobject_; }\nprivate:\nsysobject(fpga_object sysobj, token::ptr_t token, handle::ptr_t hnd);\nfpga_object sysobject_;\ntoken::ptr_t token_;\nhandle::ptr_t handle_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/token_8h/","title":"File token.h","text":"FileList > core > token.h
Go to the source code of this file.
#include <opae/access.h>#include <opae/cxx/core/properties.h>#include <opae/enum.h>#include <opae/types.h>#include <memory>#include <vector>
"},{"location":"opae-code/token_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/token_8h/#classes","title":"Classes","text":"Type Name class token Wraps the OPAE fpga_token primitive. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/token.h
"},{"location":"opae-code/token_8h_source/","title":"File token.h","text":"File List > core > token.h
Go to the documentation of this file.
// Copyright(c) 2018-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/access.h>\n#include <opae/cxx/core/properties.h>\n#include <opae/enum.h>\n#include <opae/types.h>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass token {\npublic:\ntypedef std::shared_ptr<token> ptr_t;\nstatic std::vector<token::ptr_t> enumerate(\nconst std::vector<properties::ptr_t>& props);\n~token();\nfpga_token c_type() const { return token_; }\noperator fpga_token() const { return token_; }\nptr_t get_parent() const;\nprivate:\ntoken(fpga_token tok);\nfpga_token token_;\nfriend class handle;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/cxx_2core_2version_8h/","title":"File version.h","text":"FileList > core > version.h
Go to the source code of this file.
#include <opae/types.h>#include <string>
"},{"location":"opae-code/cxx_2core_2version_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/cxx_2core_2version_8h/#classes","title":"Classes","text":"Type Name class version The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/version.h
"},{"location":"opae-code/cxx_2core_2version_8h_source/","title":"File version.h","text":"File List > core > version.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/types.h>\n#include <string>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass version {\npublic:\nstatic fpga_version as_struct();\nstatic std::string as_string();\nstatic std::string build();\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/enum_8h/","title":"File enum.h","text":"FileList > docs > sw > include > opae > enum.h
Go to the source code of this file.
APIs for resource enumeration and managing tokens. More...
#include <opae/types.h>
"},{"location":"opae-code/enum_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaCloneToken (fpga_token src, fpga_token * dst) Clone a fpga_token object. fpga_result fpgaDestroyToken (fpga_token * token) Destroy a Token. fpga_result fpgaEnumerate (const fpga_properties * filters, uint32_t num_filters, fpga_token * tokens, uint32_t max_tokens, uint32_t * num_matches) Enumerate FPGA resources present in the system."},{"location":"opae-code/enum_8h/#detailed-description","title":"Detailed Description","text":"These APIs are the first step for any application using OPAE to discover resources that are present on the system. They allow selective enumeration (i.e. getting a list of resources that match a given list of criteria) and methods to manage the lifecycle of tokens generated by fpgaEnumerate().
"},{"location":"opae-code/enum_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/enum_8h/#function-fpgaclonetoken","title":"function fpgaCloneToken","text":"Clone a fpga_token object.
fpga_result fpgaCloneToken (\nfpga_token src,\nfpga_token * dst\n)
Creates a copy of an fpga_token object.
Note:
This call creates a new token object and allocates memory for it. It is the responsibility of the using application to free this memory after use by calling fpgaDestroyToken() for the cloned token.
Parameters:
src fpga_token object to copy dst New fpga_token object cloned from 'src'
Returns:
FPGA_OK on success
"},{"location":"opae-code/enum_8h/#function-fpgadestroytoken","title":"function fpgaDestroyToken","text":"Destroy a Token.
fpga_result fpgaDestroyToken (\nfpga_token * token\n)
This function destroys a token created by fpgaEnumerate() and frees the associated memory.
Note:
fpgaDestroyToken() requires the address of an fpga_token as previously created by fpgaEnumerate() or fpgaCloneToken(). Passing any other value results in undefined behavior.
Parameters:
token fpga_token to destroy
Returns:
FPGA_OK on success
"},{"location":"opae-code/enum_8h/#function-fpgaenumerate","title":"function fpgaEnumerate","text":"Enumerate FPGA resources present in the system.
fpga_result fpgaEnumerate (\nconst fpga_properties * filters,\nuint32_t num_filters,\nfpga_token * tokens,\nuint32_t max_tokens,\nuint32_t * num_matches\n)
This call allows the user to query the system for FPGA resources that match a certain set of criteria, e.g. all accelerators that are assigned to a host interface and available, all FPGAs of a specific type, etc.
fpgaEnumerate() will create a number of fpga_tokens to represent the matching resources and populate the array tokens with these tokens. The max_tokens argument can be used to limit the number of tokens allocated/returned by fpgaEnumerate(); i.e., the number of tokens in the returned tokens array will be either max_tokens or num_matches (the number of resources matching the filter), whichever is smaller. Use fpgaDestroyToken() to destroy tokens that are no longer needed.
To query the number of matches for a particular set of filters (e.g. to allocate a tokens array of the appropriate size), call fpgaEnumerate() with the parameter tokens set to NULL; this will only return the number of matches in num_matches.
Note:
fpgaEnumerate() will allocate memory for the created tokens returned in tokens. It is the responsibility of the using application to free this memory after use by calling fpgaDestroyToken() for each of the returned tokens.
Parameters:
filters Array of fpga_properties objects describing the properties of the objects that should be returned. A resource is considered matching if its properties match any one of the supplied filters. To match all FPGA resources, pass an empty filters object (one without any filter criteria set) or pass a NULL filters parameter with num_filters set to 0. num_filters Number of entries in the filters array, or 0 to match all FPGA resources when filters is NULL. tokens Pointer to an array of fpga_token variables to be populated. If NULL is supplied, fpgaEnumerate() will not create any tokens, but it will return the number of possible matches in num_match. max_tokens Maximum number of tokens that fpgaEnumerate() shall return (length of tokens array). There may be more or fewer matches than this number; num_matches is set to the number of actual matches. num_matches Number of resources matching the filter criteria. This number can be higher than the number of tokens returned in the tokens array (depending on the value of max_tokens).
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid pointers or objects are passed into the function. FPGA_NO_DRIVER if OPAE can't find the respective enumeration data structures usually provided by the driver. FPGA_NO_MEMORY if there was not enough memory to create tokens.
The documentation for this class was generated from the following file docs/sw/include/opae/enum.h
"},{"location":"opae-code/enum_8h_source/","title":"File enum.h","text":"File List > docs > sw > include > opae > enum.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_ENUM_H__\n#define __FPGA_ENUM_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens, uint32_t *num_matches);\nfpga_result fpgaCloneToken(fpga_token src, fpga_token *dst);\nfpga_result fpgaDestroyToken(fpga_token *token);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_ENUM_H__\n
"},{"location":"opae-code/error_8h/","title":"File error.h","text":"FileList > docs > sw > include > opae > error.h
Go to the source code of this file.
Functions for reading and clearing errors in resources. More...
#include <opae/types.h>
"},{"location":"opae-code/error_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaClearAllErrors (fpga_token token) Clear all error registers of a particular resource. fpga_result fpgaClearError (fpga_token token, uint32_t error_num) Clear error register. fpga_result fpgaGetErrorInfo (fpga_token token, uint32_t error_num, struct fpga_error_info * error_info) Get information about a particular error register. fpga_result fpgaReadError (fpga_token token, uint32_t error_num, uint64_t * value) Read error value."},{"location":"opae-code/error_8h/#detailed-description","title":"Detailed Description","text":"Many FPGA resources have the ability to track the occurrence of errors. This file provides functions to retrieve information about errors within resources.
"},{"location":"opae-code/error_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/error_8h/#function-fpgaclearallerrors","title":"function fpgaClearAllErrors","text":"Clear all error registers of a particular resource.
fpga_result fpgaClearAllErrors (\nfpga_token token\n)
This function will clear all error registers of the resource referenced by token, observing the necessary order of clearing errors, if any.
Parameters:
token Token to accelerator resource to query
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token, and FPGA_BUSY if error could not be cleared.
"},{"location":"opae-code/error_8h/#function-fpgaclearerror","title":"function fpgaClearError","text":"Clear error register.
fpga_result fpgaClearError (\nfpga_token token,\nuint32_t error_num\n)
This function will clear the error register error_num of the resource referenced by token.
Parameters:
token Token to accelerator resource to query error_num Number of error register to clear
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token, and FPGA_BUSY if error could not be cleared.
"},{"location":"opae-code/error_8h/#function-fpgageterrorinfo","title":"function fpgaGetErrorInfo","text":"Get information about a particular error register.
fpga_result fpgaGetErrorInfo (\nfpga_token token,\nuint32_t error_num,\nstruct fpga_error_info * error_info\n)
This function will populate a fpga_error_info struct with information about error number error_num of the resource referenced by token.
Parameters:
token Token to accelerator resource to query error_num Error register to retrieve information about error_info Pointer to memory to store information into
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token.
"},{"location":"opae-code/error_8h/#function-fpgareaderror","title":"function fpgaReadError","text":"Read error value.
fpga_result fpgaReadError (\nfpga_token token,\nuint32_t error_num,\nuint64_t * value\n)
This function will read the value of error register error_num of the resource referenced by token into the memory location pointed to by value.
Parameters:
token Token to accelerator resource to query error_num Number of error register to read value Pointer to memory to store error value into (64 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token.
The documentation for this class was generated from the following file docs/sw/include/opae/error.h
"},{"location":"opae-code/error_8h_source/","title":"File error.h","text":"File List > docs > sw > include > opae > error.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_ERROR_H__\n#define __FPGA_ERROR_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaReadError(fpga_token token, uint32_t error_num, uint64_t *value);\nfpga_result fpgaClearError(fpga_token token, uint32_t error_num);\nfpga_result fpgaClearAllErrors(fpga_token token);\nfpga_result fpgaGetErrorInfo(fpga_token token,\nuint32_t error_num,\nstruct fpga_error_info *error_info);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_ERROR_H__\n
"},{"location":"opae-code/event_8h/","title":"File event.h","text":"FileList > docs > sw > include > opae > event.h
Go to the source code of this file.
Functions for registering events and managing the lifecycle for fpga_event_handle s.More...
#include <opae/types.h>
"},{"location":"opae-code/event_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaCreateEventHandle (fpga_event_handle * event_handle) Initialize an event_handle. fpga_result fpgaDestroyEventHandle (fpga_event_handle * event_handle) Destroy an event_handle. fpga_result fpgaGetOSObjectFromEventHandle (const fpga_event_handle eh, int * fd) Get OS object from event handle. fpga_result fpgaRegisterEvent (fpga_handle handle, fpga_event_type event_type, fpga_event_handle event_handle, uint32_t flags) Register an FPGA event. fpga_result fpgaUnregisterEvent (fpga_handle handle, fpga_event_type event_type, fpga_event_handle event_handle) Unregister an FPGA event."},{"location":"opae-code/event_8h/#detailed-description","title":"Detailed Description","text":"OPAE provides an interface to asynchronous events that can be generated by different FPGA resources. The event API provides functions to register for these events; associated with every event a process has registered for is an fpga_event_handle, which encapsulates the OS-specific data structure for event objects. On Linux, an fpga_event_handle can be used as a file descriptor and passed to select(), poll(), epoll() and similar functions to wait for asynchronous events.
"},{"location":"opae-code/event_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/event_8h/#function-fpgacreateeventhandle","title":"function fpgaCreateEventHandle","text":"Initialize an event_handle.
fpga_result fpgaCreateEventHandle (\nfpga_event_handle * event_handle\n)
Platform independent way to initialize an event_handle used for notifications from the driver to application. For Linux, this function creates an eventfd and returns the eventfd file descriptor in *event_handle.
Parameters:
event_handle Pointer to event handle variable.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if event_handle is NULL. FPGA_NOT_SUPPORTED if platform does not support events.
"},{"location":"opae-code/event_8h/#function-fpgadestroyeventhandle","title":"function fpgaDestroyEventHandle","text":"Destroy an event_handle.
fpga_result fpgaDestroyEventHandle (\nfpga_event_handle * event_handle\n)
Destroy handle and free resources. On Linux this corresponds to closing the file descriptor pointed to by handle
Note:
fpgaDestroyEventHandle() requires the address of an event_handle as created by fpgaCreateEventHandle(). Passing any other value results in undefined behavior.
Parameters:
event_handle Pointer to handle to be destroyed
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if event_handle is NULL.
"},{"location":"opae-code/event_8h/#function-fpgagetosobjectfromeventhandle","title":"function fpgaGetOSObjectFromEventHandle","text":"Get OS object from event handle.
fpga_result fpgaGetOSObjectFromEventHandle (\nconst fpga_event_handle eh,\nint * fd\n)
Check validity of event handle, and get the OS object used to subscribe and unsubscribe to events. On Linux, the object corresponds to a file descriptor.
Parameters:
eh Event handle to get the descriptor value from fd integer to store the descriptor value
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if event_handle is invalid.
"},{"location":"opae-code/event_8h/#function-fpgaregisterevent","title":"function fpgaRegisterEvent","text":"Register an FPGA event.
fpga_result fpgaRegisterEvent (\nfpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle,\nuint32_t flags\n)
This function tells the driver that the caller is interested in notification for the event specified by the type and flags pair.
The event_handle points to an OS specific mechanism for event notification. An event_handle is associated with only a single event.
In case of user interrupts, the flags parameter will be used to specify the vector ID. The value of the flags parameter indicates the vector ID, no bit encoding is used.
Todo
define if calling fpgaRegisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored.
Parameters:
handle Handle to previously opened FPGA resource. event_type Type of event event_handle Handle to previously opened resource for event notification. flags Optional argument for specifying additional information about event. For example irq number for interrupt events.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to a resource supporting the requested event, or if event_handle is not valid. FPGA_EXCEPTION if an internal exception occurred while accessing the handle or the event_handle. On Linux: FPGA_NO_DAEMON if the driver does not support the requested event and there is no FPGA Daemon (fpgad) running to proxy it.
"},{"location":"opae-code/event_8h/#function-fpgaunregisterevent","title":"function fpgaUnregisterEvent","text":"Unregister an FPGA event.
fpga_result fpgaUnregisterEvent (\nfpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle\n)
This function tells the driver that the caller is no longer interested in notification for the event associated with the event_handle
The event_handle points to an OS specific mechanism for event notification. An event_handle is associated with only a single event.
Todo
define if calling fpgaUnregisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored.
Parameters:
handle Handle to previously opened FPGA resource. event_type Type of event to unregister. event_handle Handle to previously registered resource for event notification.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to a resource supporting the requested event, or if event_handle is not valid. FPGA_EXCEPTION if an internal error occurred accessing the handle or the event_handle.
The documentation for this class was generated from the following file docs/sw/include/opae/event.h
"},{"location":"opae-code/event_8h_source/","title":"File event.h","text":"File List > docs > sw > include > opae > event.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_EVENT_H__\n#define __FPGA_EVENT_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaCreateEventHandle(fpga_event_handle *event_handle);\nfpga_result fpgaDestroyEventHandle(fpga_event_handle *event_handle);\nfpga_result fpgaGetOSObjectFromEventHandle(const fpga_event_handle eh, int *fd);\nfpga_result fpgaRegisterEvent(fpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle,\nuint32_t flags);\nfpga_result fpgaUnregisterEvent(fpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_EVENT_H__\n
"},{"location":"opae-code/fpga_8h/","title":"File fpga.h","text":"FileList > docs > sw > include > opae > fpga.h
Go to the source code of this file.
FPGA API. More...
#include <opae/log.h>#include <opae/init.h>#include <opae/types.h>#include <opae/access.h>#include <opae/buffer.h>#include <opae/enum.h>#include <opae/event.h>#include <opae/manage.h>#include <opae/mmio.h>#include <opae/properties.h>#include <opae/umsg.h>#include <opae/utils.h>#include <opae/error.h>#include <opae/version.h>#include <opae/sysobject.h>#include <opae/userclk.h>#include <opae/metrics.h>
"},{"location":"opae-code/fpga_8h/#detailed-description","title":"Detailed Description","text":"This conveniently includes all APIs that a part of the OPAE release (base and extensions).
The documentation for this class was generated from the following file docs/sw/include/opae/fpga.h
"},{"location":"opae-code/fpga_8h_source/","title":"File fpga.h","text":"File List > docs > sw > include > opae > fpga.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_FPGA_H__\n#define __FPGA_FPGA_H__\n#include <opae/log.h>\n#include <opae/init.h>\n#include <opae/types.h>\n#include <opae/access.h>\n#include <opae/buffer.h>\n#include <opae/enum.h>\n#include <opae/event.h>\n#include <opae/manage.h>\n#include <opae/mmio.h>\n#include <opae/properties.h>\n#include <opae/umsg.h>\n#include <opae/utils.h>\n#include <opae/error.h>\n#include <opae/version.h>\n#include <opae/sysobject.h>\n#include <opae/userclk.h>\n#include <opae/metrics.h>\n#endif // __FPGA_FPGA_H__\n
"},{"location":"opae-code/hash__map_8h/","title":"File hash_map.h","text":"FileList > docs > sw > include > opae > hash_map.h
Go to the source code of this file.
A general-purpose hybrid array/list hash map implementation. More...
#include <stdint.h>#include <stdbool.h>#include <opae/types_enum.h>
"},{"location":"opae-code/hash__map_8h/#classes","title":"Classes","text":"Type Name struct _opae_hash_map Hash map object. struct _opae_hash_map_item List link item."},{"location":"opae-code/hash__map_8h/#public-types","title":"Public Types","text":"Type Name enum _opae_hash_map_flags Flags used to initialize a hash map. typedef struct _opae_hash_map opae_hash_map Hash map object. typedef enum _opae_hash_map_flags opae_hash_map_flags Flags used to initialize a hash map. typedef struct _opae_hash_map_item opae_hash_map_item List link item."},{"location":"opae-code/hash__map_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result opae_hash_map_add (opae_hash_map * hm, void * key, void * value) Map a key to a value. fpga_result opae_hash_map_destroy (opae_hash_map * hm) Tear down a hash map. fpga_result opae_hash_map_find (opae_hash_map * hm, void * key, void ** value) Retrieve the value for a given key. fpga_result opae_hash_map_init (opae_hash_map * hm, uint32_t num_buckets, uint32_t hash_seed, int flags, uint32_t(*)(uint32_t num_buckets, uint32_t hash_seed, void *key) key_hash, int(*)(void *keya, void *keyb) key_compare, void(*)(void *key, void *context) key_cleanup, void(*)(void *value, void *context) value_cleanup) Initialize a hash map. bool opae_hash_map_is_empty (opae_hash_map * hm) Determine whether a hash map is empty. fpga_result opae_hash_map_remove (opae_hash_map * hm, void * key) Remove a key/value association. int opae_u64_key_compare (void * keya, void * keyb) Convenience key comparison function for 64-bit values. uint32_t opae_u64_key_hash (uint32_t num_buckets, uint32_t hash_seed, void * key) Convenience hash function for arbitrary pointers/64-bit values."},{"location":"opae-code/hash__map_8h/#detailed-description","title":"Detailed Description","text":"Presents a generic interface for mapping key objects to value objects. Both keys and values may be arbitrary data structures. The user supplies the means by which the hash of values is generated and by which the keys are compared to each other.
"},{"location":"opae-code/hash__map_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/hash__map_8h/#enum-_opae_hash_map_flags","title":"enum _opae_hash_map_flags","text":"Flags used to initialize a hash map.
enum _opae_hash_map_flags {\nOPAE_HASH_MAP_UNIQUE_KEYSPACE = (1u << 0)\n};\n
OPAE_HASH_MAP_UNIQUE_KEYSPACE says that the user provides a guarantee that the key space is truly unique. In other words, when the provided hash function for keys A and B returns the same bucket index, the key comparison function when comparing A and B will never return a result saying that the keys are equal in value. This is helpful in situations where the key space is guaranteed to produce unique values, for example a memory allocator. When the key space is guaranteed to be unique, opae_hash_map_add() can implement a small performance improvement.
"},{"location":"opae-code/hash__map_8h/#typedef-opae_hash_map","title":"typedef opae_hash_map","text":"Hash map object.
typedef struct _opae_hash_map opae_hash_map;\n
This structure defines the internals of the hash map. Each of the parameters supplied to opae_hash_map_init() is stored in the structure. All parameters are required, except key_cleanup and value_cleanup, which may optionally be NULL.
"},{"location":"opae-code/hash__map_8h/#typedef-opae_hash_map_flags","title":"typedef opae_hash_map_flags","text":"Flags used to initialize a hash map.
typedef enum _opae_hash_map_flags opae_hash_map_flags;\n
OPAE_HASH_MAP_UNIQUE_KEYSPACE says that the user provides a guarantee that the key space is truly unique. In other words, when the provided hash function for keys A and B returns the same bucket index, the key comparison function when comparing A and B will never return a result saying that the keys are equal in value. This is helpful in situations where the key space is guaranteed to produce unique values, for example a memory allocator. When the key space is guaranteed to be unique, opae_hash_map_add() can implement a small performance improvement.
"},{"location":"opae-code/hash__map_8h/#typedef-opae_hash_map_item","title":"typedef opae_hash_map_item","text":"List link item.
typedef struct _opae_hash_map_item opae_hash_map_item;\n
This structure provides the association between key and value. When the supplied hash function for keys A and B returns the same bucket index, both A and B can co-exist on the same list rooted at the bucket index.
"},{"location":"opae-code/hash__map_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_add","title":"function opae_hash_map_add","text":"Map a key to a value.
fpga_result opae_hash_map_add (\nopae_hash_map * hm,\nvoid * key,\nvoid * value\n)
Inserts a mapping from key to value in the given hash map object. Subsequent calls to opae_hash_map_find() that are given the key will retrieve the value.
Parameters:
hm A pointer to the storage for the hash map object. key The hash map key. value The hash map value.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if hm is NULL, FPGA_NO_MEMORY if malloc() fails when allocating the list item, or FPGA_INVALID_PARAM if the key hash produced by key_hash is out of bounds.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_destroy","title":"function opae_hash_map_destroy","text":"Tear down a hash map.
fpga_result opae_hash_map_destroy (\nopae_hash_map * hm\n)
Given a hash map that was previously initialized by opae_hash_map_init(), destroy the hash map, releasing all keys, values, and the bucket array.
Parameters:
hm A pointer to the storage for the hash map object.
Returns:
FPGA_OK on success or FPGA_INVALID_PARAM is hm is NULL.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_find","title":"function opae_hash_map_find","text":"Retrieve the value for a given key.
fpga_result opae_hash_map_find (\nopae_hash_map * hm,\nvoid * key,\nvoid ** value\n)
Given a key that was previously passed to opae_hash_map_add(), retrieve its associated value.
Parameters:
hm A pointer to the storage for the hash map object. key The hash map key. value A pointer to receive the hash map value.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if hm is NULL or if the key hash produced by key_hash is out of bounds, or FPGA_NOT_FOUND if the given key was not found in the hash map.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_init","title":"function opae_hash_map_init","text":"Initialize a hash map.
fpga_result opae_hash_map_init (\nopae_hash_map * hm,\nuint32_t num_buckets,\nuint32_t hash_seed,\nint flags,\nuint32_t(*)(uint32_t num_buckets, uint32_t hash_seed, void *key) key_hash,\nint(*)(void *keya, void *keyb) key_compare,\nvoid(*)(void *key, void *context) key_cleanup,\nvoid(*)(void *value, void *context) value_cleanup\n)
Populates the hash map data structure and allocates the buckets array.
Parameters:
hm A pointer to the storage for the hash map object. num_buckets The desired size of the buckets array. Each array entry may be empty (NULL), or may contain a list of opae_hash_map_item structures for which the given key_hash function returned the same key hash value. hash_seed A seed value used during key hash computation. This value will be the hash_seed parameter to the key hash function. flags Initialization flags. See opae_hash_map_flags. key_hash A pointer to a function that produces the hash value, given the number of buckets, the hash seed, and the key. Valid values are between 0 and num_buckets - 1, inclusively. key_compare A pointer to a function that compares two keys. The return value is similar to that of strcmp(), where a negative value means that keya < keyb, 0 means that keya == keyb, and a positive values means that keya > keyb. key_cleanup A pointer to a function that is called when a key is being removed from the map. This function is optional and may be NULL. When supplied, the function is responsible for freeing any resources allocated when the key was created. value_cleanup A pointer to a function that is called when a value is being removed from the map. This function is optional and may be NULL. When supplied, the function is responsible for freeing any resources allocated when the value was created.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the required parameters are NULL, or FPGA_NO_MEMORY if the bucket array could not be allocated.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_is_empty","title":"function opae_hash_map_is_empty","text":"Determine whether a hash map is empty.
bool opae_hash_map_is_empty (\nopae_hash_map * hm\n)
Parameters:
hm A pointer to the storage for the hash map object.
Returns:
true if there are no key/value mappings present, false otherwise.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_remove","title":"function opae_hash_map_remove","text":"Remove a key/value association.
fpga_result opae_hash_map_remove (\nopae_hash_map * hm,\nvoid * key\n)
Given a key that was previously passed to opae_hash_map_add(), remove the key and its associated value, calling the cleanup functions as needed.
Parameters:
hm A pointer to the storage for the hash map object. key The hash map key.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM when hm is NULL or when the key hash produced by key_hash is out of bounds, or FPGA_NOT_FOUND if the key is not found in the hash map.
"},{"location":"opae-code/hash__map_8h/#function-opae_u64_key_compare","title":"function opae_u64_key_compare","text":"Convenience key comparison function for 64-bit values.
int opae_u64_key_compare (\nvoid * keya,\nvoid * keyb\n)
Simply converts the key pointers to uint64_t's and performs unsigned integer comparison.
"},{"location":"opae-code/hash__map_8h/#function-opae_u64_key_hash","title":"function opae_u64_key_hash","text":"Convenience hash function for arbitrary pointers/64-bit values.
uint32_t opae_u64_key_hash (\nuint32_t num_buckets,\nuint32_t hash_seed,\nvoid * key\n)
Simply converts the key to a uint64_t and then performs the modulus operation with the configured num_buckets. hash_seed is unused.
The documentation for this class was generated from the following file docs/sw/include/opae/hash_map.h
"},{"location":"opae-code/hash__map_8h_source/","title":"File hash_map.h","text":"File List > docs > sw > include > opae > hash_map.h
Go to the documentation of this file.
// Copyright(c) 2022-2023, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_HASH_MAP_H__\n#define __OPAE_HASH_MAP_H__\n#include <stdint.h>\n#include <stdbool.h>\n#include <opae/types_enum.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif // __cplusplus\ntypedef enum _opae_hash_map_flags {\nOPAE_HASH_MAP_UNIQUE_KEYSPACE = (1u << 0)\n} opae_hash_map_flags;\ntypedef struct _opae_hash_map_item {\nvoid *key;\nvoid *value;\nstruct _opae_hash_map_item *next;\n} opae_hash_map_item;\ntypedef struct _opae_hash_map {\nuint32_t num_buckets;\nuint32_t hash_seed;\nopae_hash_map_item **buckets;\nint flags;\nvoid *cleanup_context; uint32_t (*key_hash)(uint32_t num_buckets, uint32_t hash_seed,\nvoid *key);\nint (*key_compare)(void *keya, void *keyb); void (*key_cleanup)(void *key, void *context); void (*value_cleanup)(void *value, void *context); } opae_hash_map;\nfpga_result opae_hash_map_init(opae_hash_map *hm,\nuint32_t num_buckets,\nuint32_t hash_seed,\nint flags,\nuint32_t (*key_hash)(uint32_t num_buckets,\nuint32_t hash_seed,\nvoid *key),\nint (*key_compare)(void *keya, void *keyb),\nvoid (*key_cleanup)(void *key, void *context),\nvoid (*value_cleanup)(void *value, void *context));\nfpga_result opae_hash_map_add(opae_hash_map *hm,\nvoid *key,\nvoid *value);\nfpga_result opae_hash_map_find(opae_hash_map *hm,\nvoid *key,\nvoid **value);\nfpga_result opae_hash_map_remove(opae_hash_map *hm,\nvoid *key);\nfpga_result opae_hash_map_destroy(opae_hash_map *hm);\nbool opae_hash_map_is_empty(opae_hash_map *hm);\nuint32_t opae_u64_key_hash(uint32_t num_buckets,\nuint32_t hash_seed,\nvoid *key);\nint opae_u64_key_compare(void *keya, void *keyb);\n#ifdef __cplusplus\n}\n#endif // __cplusplus\n#endif // __OPAE_HASH_MAP_H__\n
"},{"location":"opae-code/init_8h/","title":"File init.h","text":"FileList > docs > sw > include > opae > init.h
Go to the source code of this file.
Initialization routine.
#include <opae/types_enum.h>
"},{"location":"opae-code/init_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaFinalize (void) Finalize the OPAE library. fpga_result fpgaInitialize (const char * config_file) Initialize the OPAE library."},{"location":"opae-code/init_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/init_8h/#function-fpgafinalize","title":"function fpgaFinalize","text":"Finalize the OPAE library.
fpga_result fpgaFinalize (\nvoid\n)
Returns:
Whether OPAE finalized successfully.
"},{"location":"opae-code/init_8h/#function-fpgainitialize","title":"function fpgaInitialize","text":"Initialize the OPAE library.
fpga_result fpgaInitialize (\nconst char * config_file\n)
Initialize OPAE using the given configuration file path, or perform default initialization if config_file is NULL.
Parameters:
config_file Path to OPAE configuration file.
Returns:
Whether OPAE initialized successfully.
The documentation for this class was generated from the following file docs/sw/include/opae/init.h
"},{"location":"opae-code/init_8h_source/","title":"File init.h","text":"File List > docs > sw > include > opae > init.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_INIT_H__\n#define __FPGA_INIT_H__\n#include <opae/types_enum.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaInitialize(const char *config_file);\nfpga_result fpgaFinalize(void);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_INIT_H__\n
"},{"location":"opae-code/log_8h/","title":"File log.h","text":"FileList > docs > sw > include > opae > log.h
Go to the source code of this file.
#include <stdint.h>#include <stdlib.h>#include <string.h>#include <errno.h>#include <opae/types.h>
"},{"location":"opae-code/log_8h/#public-types","title":"Public Types","text":"Type Name enum opae_loglevel"},{"location":"opae-code/log_8h/#public-functions","title":"Public Functions","text":"Type Name void opae_print (int loglevel, const char * fmt, ...)"},{"location":"opae-code/log_8h/#macros","title":"Macros","text":"Type Name define OPAE_DBG (format, ...) { } define OPAE_DEFAULT_LOGLEVEL OPAE_LOG_ERROR define OPAE_ERR (format, ...) define OPAE_MSG (format, ...) define __SHORT_FILE__"},{"location":"opae-code/log_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/log_8h/#enum-opae_loglevel","title":"enum opae_loglevel","text":"enum opae_loglevel {\nOPAE_LOG_ERROR = 0,\nOPAE_LOG_MESSAGE,\nOPAE_LOG_DEBUG\n};\n
"},{"location":"opae-code/log_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/log_8h/#function-opae_print","title":"function opae_print","text":"void opae_print (\nint loglevel,\nconst char * fmt,\n...\n)
"},{"location":"opae-code/log_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/log_8h/#define-opae_dbg","title":"define OPAE_DBG","text":"#define OPAE_DBG (\nformat,\n...\n) { }\n
"},{"location":"opae-code/log_8h/#define-opae_default_loglevel","title":"define OPAE_DEFAULT_LOGLEVEL","text":"#define OPAE_DEFAULT_LOGLEVEL OPAE_LOG_ERROR\n
"},{"location":"opae-code/log_8h/#define-opae_err","title":"define OPAE_ERR","text":"#define OPAE_ERR (\nformat,\n...\n) opae_print ( OPAE_LOG_ERROR , \\\n \"%s:%u:%s() **ERROR** : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n
"},{"location":"opae-code/log_8h/#define-opae_msg","title":"define OPAE_MSG","text":"#define OPAE_MSG (\nformat,\n...\n) opae_print ( OPAE_LOG_MESSAGE , \"%s:%u:%s() : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n
"},{"location":"opae-code/log_8h/#define-__short_file__","title":"define __SHORT_FILE__","text":"#define __SHORT_FILE__ ({ \\\n const char *file = __FILE__; \\\n const char *p = file; \\\n while (*p) \\\n ++p; \\\n while ((p > file) && ('/' != *p) && ('\\\\' != *p)) \\\n --p; \\\n if (p > file) \\\n ++p; \\\n p; \\\n })\n
The documentation for this class was generated from the following file docs/sw/include/opae/log.h
"},{"location":"opae-code/log_8h_source/","title":"File log.h","text":"File List > docs > sw > include > opae > log.h
Go to the documentation of this file.
// Copyright(c) 2018-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_LOG_H__\n#define __OPAE_LOG_H__\n#include <stdint.h>\n#include <stdlib.h>\n#include <string.h>\n#include <errno.h>\n#include <opae/types.h>\n/*\n* Convenience macros for printing messages and errors.\n*/\n#ifdef __SHORT_FILE__\n#undef __SHORT_FILE__\n#endif // __SHORT_FILE__\n#define __SHORT_FILE__ \\\n ({ \\\n const char *file = __FILE__; \\\n const char *p = file; \\\n while (*p) \\\n ++p; \\\n while ((p > file) && ('/' != *p) && ('\\\\' != *p)) \\\n --p; \\\n if (p > file) \\\n ++p; \\\n p; \\\n })\n#ifdef OPAE_MSG\n#undef OPAE_MSG\n#endif // OPAE_MSG\n#define OPAE_MSG(format, ...) \\\n opae_print(OPAE_LOG_MESSAGE, \"%s:%u:%s() : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n#ifdef OPAE_ERR\n#undef OPAE_ERR\n#endif // OPAE_ERR\n#define OPAE_ERR(format, ...) \\\n opae_print(OPAE_LOG_ERROR, \\\n \"%s:%u:%s() **ERROR** : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n#ifdef OPAE_DBG\n#undef OPAE_DBG\n#endif // OPAE_DBG\n#ifdef LIBOPAE_DEBUG\n#define OPAE_DBG(format, ...) \\\n opae_print(OPAE_LOG_DEBUG, \\\n \"%s:%u:%s() *DEBUG* : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n#else\n#define OPAE_DBG(format, ...) \\\n{ }\n#endif // LIBOPAE_DEBUG\n/*\n* Logging functions\n*/\nenum opae_loglevel {\nOPAE_LOG_ERROR = 0, /* critical errors (always print) */\nOPAE_LOG_MESSAGE, /* information (i.e. explain return code */\nOPAE_LOG_DEBUG /* debugging (also needs #define DEBUG 1) */\n};\n#define OPAE_DEFAULT_LOGLEVEL OPAE_LOG_ERROR\n#ifdef __cplusplus\nextern \"C\" {\n#endif // __cplusplus\nvoid opae_print(int loglevel, const char *fmt, ...);\n#ifdef __cplusplus\n}\n#endif // __cplusplus\n#endif // __OPAE_LOG_H__\n
"},{"location":"opae-code/manage_8h/","title":"File manage.h","text":"FileList > docs > sw > include > opae > manage.h
Go to the source code of this file.
Functions for managing FPGA configurations. More...
#include <opae/types.h>
"},{"location":"opae-code/manage_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaAssignPortToInterface (fpga_handle fpga, uint32_t interface_num, uint32_t slot_num, int flags) Assign Port to a host interface. fpga_result fpgaAssignToInterface (fpga_handle fpga, fpga_token accelerator, uint32_t host_interface, int flags) Assign an accelerator to a host interface. fpga_result fpgaReconfigureSlot (fpga_handle fpga, uint32_t slot, const uint8_t * bitstream, size_t bitstream_len, int flags) Reconfigure a slot. fpga_result fpgaReleaseFromInterface (fpga_handle fpga, fpga_token accelerator) Unassign a previously assigned accelerator."},{"location":"opae-code/manage_8h/#detailed-description","title":"Detailed Description","text":"FPGA accelerators can be reprogrammed at run time by providing new partial bitstreams (\"green bitstreams\"). This file defines API functions for programming green bitstreams as well as for assigning accelerators to host interfaces for more complex deployment setups, such as virtualized systems.
"},{"location":"opae-code/manage_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/manage_8h/#function-fpgaassignporttointerface","title":"function fpgaAssignPortToInterface","text":"Assign Port to a host interface.
fpga_result fpgaAssignPortToInterface (\nfpga_handle fpga,\nuint32_t interface_num,\nuint32_t slot_num,\nint flags\n)
This function assign Port to a host interface for subsequent use. Only Port that have been assigned to a host interface can be opened by fpgaOpen().
Parameters:
fpga Handle to an FPGA object previously opened that both the host interface and the slot belong to interface_num Host interface number slot_num Slot number flags Flags (to be defined)
Returns:
FPGA_OK on success FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if an exception occcurred accessing the fpga handle. FPGA_NOT_SUPPORTED if driver does not support assignment.
"},{"location":"opae-code/manage_8h/#function-fpgaassigntointerface","title":"function fpgaAssignToInterface","text":"Assign an accelerator to a host interface.
fpga_result fpgaAssignToInterface (\nfpga_handle fpga,\nfpga_token accelerator,\nuint32_t host_interface,\nint flags\n)
This function assigns an accelerator to a host interface for subsequent use. Only accelerators that have been assigned to a host interface can be opened by fpgaOpen().
Note:
This function is currently not supported.
Parameters:
fpga Handle to an FPGA object previously opened that both the host interface and the accelerator belong to accelerator accelerator to assign host_interface Host interface to assign accelerator to flags Flags (to be defined)
Returns:
FPGA_OK on success
"},{"location":"opae-code/manage_8h/#function-fpgareconfigureslot","title":"function fpgaReconfigureSlot","text":"Reconfigure a slot.
fpga_result fpgaReconfigureSlot (\nfpga_handle fpga,\nuint32_t slot,\nconst uint8_t * bitstream,\nsize_t bitstream_len,\nint flags\n)
Sends a green bitstream file to an FPGA to reconfigure a specific slot. This call, if successful, will overwrite the currently programmed AFU in that slot with the AFU in the provided bitstream.
As part of the reconfiguration flow, all accelerators associated with this slot will be unassigned and reset.
Parameters:
fpga Handle to an FPGA object previously opened slot Token identifying the slot to reconfigure bitstream Pointer to memory holding the bitstream bitstream_len Length of the bitstream in bytes flags Flags that control behavior of reconfiguration. Value of 0 indicates no flags. FPGA_RECONF_FORCE indicates that the bitstream is programmed into the slot without checking if the resource is currently in use.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if the provided parameters are not valid. FPGA_EXCEPTION if an internal error occurred accessing the handle or while sending the bitstream data to the driver. FPGA_BUSY if the accelerator for the given slot is in use. FPGA_RECONF_ERROR on errors reported by the driver (such as CRC or protocol errors).
Note:
By default, fpgaReconfigureSlot will not allow reconfiguring a slot with an accelerator in use. Add the flag FPGA_RECONF_FORCE to force reconfiguration without checking for accelerators in use.
"},{"location":"opae-code/manage_8h/#function-fpgareleasefrominterface","title":"function fpgaReleaseFromInterface","text":"Unassign a previously assigned accelerator.
fpga_result fpgaReleaseFromInterface (\nfpga_handle fpga,\nfpga_token accelerator\n)
This function removes the assignment of an accelerator to an host interface (e.g. to be later assigned to a different host interface). As a consequence, the accelerator referred to by token 'accelerator' will be reset during the course of this function.
Note:
This function is currently not supported.
Parameters:
fpga Handle to an FPGA object previously opened that both the host interface and the accelerator belong to accelerator accelerator to unassign/release
Returns:
FPGA_OK on success
The documentation for this class was generated from the following file docs/sw/include/opae/manage.h
"},{"location":"opae-code/manage_8h_source/","title":"File manage.h","text":"File List > docs > sw > include > opae > manage.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_MANAGE_H__\n#define __FPGA_MANAGE_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaAssignPortToInterface(fpga_handle fpga,\nuint32_t interface_num,\nuint32_t slot_num,\nint flags);\nfpga_result fpgaAssignToInterface(fpga_handle fpga,\nfpga_token accelerator,\nuint32_t host_interface,\nint flags);\nfpga_result fpgaReleaseFromInterface(fpga_handle fpga,\nfpga_token accelerator);\nfpga_result fpgaReconfigureSlot(fpga_handle fpga,\nuint32_t slot,\nconst uint8_t *bitstream,\nsize_t bitstream_len, int flags);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_MANAGE_H__\n
"},{"location":"opae-code/mem__alloc_8h/","title":"File mem_alloc.h","text":"FileList > docs > sw > include > opae > mem_alloc.h
Go to the source code of this file.
#include <stdint.h>
"},{"location":"opae-code/mem__alloc_8h/#classes","title":"Classes","text":"Type Name struct mem_alloc struct mem_link Provides an API for allocating/freeing a logical address space."},{"location":"opae-code/mem__alloc_8h/#public-functions","title":"Public Functions","text":"Type Name int mem_alloc_add_free (struct mem_alloc * m, uint64_t address, uint64_t size) Add a memory region to an allocator. void mem_alloc_destroy (struct mem_alloc * m) Destroy a memory allocator object. int mem_alloc_get (struct mem_alloc * m, uint64_t * address, uint64_t size) Allocate memory. void mem_alloc_init (struct mem_alloc * m) Initialize a memory allocator object. int mem_alloc_put (struct mem_alloc * m, uint64_t address) Free memory."},{"location":"opae-code/mem__alloc_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_add_free","title":"function mem_alloc_add_free","text":"Add a memory region to an allocator.
int mem_alloc_add_free (\nstruct mem_alloc * m,\nuint64_t address,\nuint64_t size\n)
The memory region is added to the allocatable space and is immediately ready for allocation.
Parameters:
m The memory allocator object. address The beginning address of the memory region. size The size of the memory region.
Returns:
Non-zero on error. Zero on success.
Example struct mem_alloc m;
mem_alloc_init(&m);
if (mem_alloc_add_free(&m, 0x4000, 4096)) { // handle error }
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_destroy","title":"function mem_alloc_destroy","text":"Destroy a memory allocator object.
void mem_alloc_destroy (\nstruct mem_alloc * m\n)
Frees all of the allocator's internal resources.
Parameters:
m The address of the memory allocator to destroy.
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_get","title":"function mem_alloc_get","text":"Allocate memory.
int mem_alloc_get (\nstruct mem_alloc * m,\nuint64_t * address,\nuint64_t size\n)
Retrieve an available memory address for a free block that is at least size bytes.
Parameters:
m The memory allocator object. address The retrieved address for the allocation. size The request size in bytes.
Returns:
Non-zero on error. Zero on success.
Example struct mem_alloc m; uint64_t addr = 0;
mem_alloc_init(&m);
if (mem_alloc_add_free(&m, 0x4000, 4096)) { // handle error }
...
if (mem_alloc_get(&m, &addr, 4096)) { // handle allocation error }
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_init","title":"function mem_alloc_init","text":"Initialize a memory allocator object.
void mem_alloc_init (\nstruct mem_alloc * m\n)
After the call, the allocator is initialized but \"empty\". To add allocatable memory regions, further initialize the allocator with mem_alloc_add_free().
Parameters:
m The address of the memory allocator to initialize.
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_put","title":"function mem_alloc_put","text":"Free memory.
int mem_alloc_put (\nstruct mem_alloc * m,\nuint64_t address\n)
Release a previously-allocated memory block.
Parameters:
m The memory allocator object. address The address to free.
Returns:
Non-zero on error. Zero on success.
Example struct mem_alloc m; uint64_t addr = 0;
mem_alloc_init(&m);
if (mem_alloc_add_free(&m, 0x4000, 4096)) { // handle error }
...
if (mem_alloc_get(&m, &addr, 4096)) { // handle allocation error }
...
if (mem_alloc_put(&m, addr)) { // handle free error }
The documentation for this class was generated from the following file docs/sw/include/opae/mem_alloc.h
"},{"location":"opae-code/mem__alloc_8h_source/","title":"File mem_alloc.h","text":"File List > docs > sw > include > opae > mem_alloc.h
Go to the documentation of this file.
// Copyright(c) 2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_MEM_ALLOC_H__\n#define __OPAE_MEM_ALLOC_H__\n#include <stdint.h>\nstruct mem_link {\nuint64_t address;\nuint64_t size;\nstruct mem_link *prev;\nstruct mem_link *next;\n};\nstruct mem_alloc {\nstruct mem_link free;\nstruct mem_link allocated;\n};\n#ifdef __cplusplus\nextern \"C\" {\n#endif // __cplusplus\nvoid mem_alloc_init(struct mem_alloc *m);\nvoid mem_alloc_destroy(struct mem_alloc *m);\nint mem_alloc_add_free(struct mem_alloc *m,\nuint64_t address,\nuint64_t size);\nint mem_alloc_get(struct mem_alloc *m,\nuint64_t *address,\nuint64_t size);\nint mem_alloc_put(struct mem_alloc *m,\nuint64_t address);\n#ifdef __cplusplus\n}\n#endif // __cplusplus\n#endif // __OPAE_MEM_ALLOC_H__\n
"},{"location":"opae-code/metrics_8h/","title":"File metrics.h","text":"FileList > docs > sw > include > opae > metrics.h
Go to the source code of this file.
Functions for Discover/ Enumerates metrics and retrieves values.
#include <opae/types.h>
"},{"location":"opae-code/metrics_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetMetricsByIndex (fpga_handle handle, uint64_t * metric_num, uint64_t num_metric_indexes, fpga_metric * metrics) Retrieve metrics values by index. fpga_result fpgaGetMetricsByName (fpga_handle handle, char ** metrics_names, uint64_t num_metric_names, fpga_metric * metrics) Retrieve metric values by names. fpga_result fpgaGetMetricsInfo (fpga_handle handle, fpga_metric_info * metric_info, uint64_t * num_metrics) Retrieve metrics information. fpga_result fpgaGetMetricsThresholdInfo (fpga_handle handle, struct metric_threshold * metric_thresholds, uint32_t * num_thresholds) Retrieve metrics / sendor threshold information and values. fpga_result fpgaGetNumMetrics (fpga_handle handle, uint64_t * num_metrics) Enumerates number of metrics."},{"location":"opae-code/metrics_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsbyindex","title":"function fpgaGetMetricsByIndex","text":"Retrieve metrics values by index.
fpga_result fpgaGetMetricsByIndex (\nfpga_handle handle,\nuint64_t * metric_num,\nuint64_t num_metric_indexes,\nfpga_metric * metrics\n)
Parameters:
handle Handle to previously opened fpga resource metric_num Pointer to array of metric index user allocates metric array num_metric_indexes Size of metric array metrics pointer to array of metric struct
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found. FPGA_NO_MEMORY if there was not enough memory to enumerates metrics.
"},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsbyname","title":"function fpgaGetMetricsByName","text":"Retrieve metric values by names.
fpga_result fpgaGetMetricsByName (\nfpga_handle handle,\nchar ** metrics_names,\nuint64_t num_metric_names,\nfpga_metric * metrics\n)
Parameters:
handle Handle to previously opened fpga resource metrics_names Pointer to array of metrics name user allocates metrics name array num_metric_names Size of metric name array metrics Pointer to array of metric struct
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found
"},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsinfo","title":"function fpgaGetMetricsInfo","text":"Retrieve metrics information.
fpga_result fpgaGetMetricsInfo (\nfpga_handle handle,\nfpga_metric_info * metric_info,\nuint64_t * num_metrics\n)
Parameters:
handle Handle to previously opened fpga resource metric_info Pointer to array of metric info struct user allocates metrics info arraynum_metrics Size of metric info array
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found. FPGA_NO_MEMORY if there was not enough memory to enumerates metrics.
"},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsthresholdinfo","title":"function fpgaGetMetricsThresholdInfo","text":"Retrieve metrics / sendor threshold information and values.
fpga_result fpgaGetMetricsThresholdInfo (\nfpga_handle handle,\nstruct metric_threshold * metric_thresholds,\nuint32_t * num_thresholds\n)
Parameters:
handle Handle to previously opened fpga resource metrics_threshold pointer to array of metric thresholds user allocates threshold array memory Number of thresholds returns enumerated thresholds if user pass NULL metrics_thresholds num_thresholds number of thresholds
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found. FPGA_NO_MEMORY if there was not enough memory to enumerates metrics.
"},{"location":"opae-code/metrics_8h/#function-fpgagetnummetrics","title":"function fpgaGetNumMetrics","text":"Enumerates number of metrics.
fpga_result fpgaGetNumMetrics (\nfpga_handle handle,\nuint64_t * num_metrics\n)
Parameters:
handle Handle to previously opened fpga resource num_metrics Number of metrics are discovered in fpga resource
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not discovered
The documentation for this class was generated from the following file docs/sw/include/opae/metrics.h
"},{"location":"opae-code/metrics_8h_source/","title":"File metrics.h","text":"File List > docs > sw > include > opae > metrics.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_METRICS_H__\n#define __FPGA_METRICS_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetNumMetrics(fpga_handle handle,\nuint64_t *num_metrics);\nfpga_result fpgaGetMetricsInfo(fpga_handle handle,\nfpga_metric_info *metric_info,\nuint64_t *num_metrics);\nfpga_result fpgaGetMetricsByIndex(fpga_handle handle,\nuint64_t *metric_num,\nuint64_t num_metric_indexes,\nfpga_metric *metrics);\nfpga_result fpgaGetMetricsByName(fpga_handle handle,\nchar **metrics_names,\nuint64_t num_metric_names,\nfpga_metric *metrics);\nfpga_result fpgaGetMetricsThresholdInfo(fpga_handle handle,\nstruct metric_threshold *metric_thresholds,\nuint32_t *num_thresholds);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_METRICS_H__\n
"},{"location":"opae-code/mmio_8h/","title":"File mmio.h","text":"FileList > docs > sw > include > opae > mmio.h
Go to the source code of this file.
Functions for mapping and accessing MMIO space. More...
#include <opae/types.h>
"},{"location":"opae-code/mmio_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaMapMMIO (fpga_handle handle, uint32_t mmio_num, uint64_t ** mmio_ptr) Map MMIO space. fpga_result fpgaReadMMIO32 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint32_t * value) Read 32 bit value from MMIO space. fpga_result fpgaReadMMIO64 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint64_t * value) Read 64 bit value from MMIO space. fpga_result fpgaUnmapMMIO (fpga_handle handle, uint32_t mmio_num) Unmap MMIO space. fpga_result fpgaWriteMMIO32 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint32_t value) Write 32 bit value to MMIO space. fpga_result fpgaWriteMMIO512 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, const void * value) Write 512 bit value to MMIO space. fpga_result fpgaWriteMMIO64 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint64_t value) Write 64 bit value to MMIO space."},{"location":"opae-code/mmio_8h/#detailed-description","title":"Detailed Description","text":"Most FPGA accelerators provide access to control registers through memory-mappable address spaces, commonly referred to as \"MMIO spaces\". This file provides functions to map, unmap, read, and write MMIO spaces.
Note that an accelerator may have multiple MMIO spaces, denoted by the mmio_num argument of the APIs below. The meaning and properties of each MMIO space are up to the accelerator designer.
"},{"location":"opae-code/mmio_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/mmio_8h/#function-fpgamapmmio","title":"function fpgaMapMMIO","text":"Map MMIO space.
fpga_result fpgaMapMMIO (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t ** mmio_ptr\n)
This function will return a pointer to the specified MMIO space of the target object in process virtual memory, if supported by the target. Some MMIO spaces may be restricted to privileged processes, depending on the used handle and type.
After mapping the respective MMIO space, you can access it through direct pointer operations (observing supported access sizes and alignments of the target platform and accelerator).
Note:
Some targets (such as the ASE simulator) do not support memory-mapping of IO register spaces and will not return a pointer to an actually mapped space. Instead, they will return FPGA_NOT_SUPPORTED. Usually, these platforms still allow the application to issue MMIO operations using fpgaReadMMIO32(), fpgaWriteMMIO32(), fpgeReadMMIO64(), and fpgaWriteMMIO64().
If the caller passes in NULL for mmio_ptr, no mapping will be performed, and no virtual address will be returned, though the call will return FPGA_OK. This implies that all accesses will be performed through fpgaReadMMIO32(), fpgaWriteMMIO32(), fpgeReadMMIO64(), and fpgaWriteMMIO64(). This is the only supported case for ASE.
The number of available MMIO spaces can be retrieved through the num_mmio property (fpgaPropertyGetNumMMIO()).
Parameters:
handle Handle to previously opened resource mmio_num Number of MMIO space to access mmio_ptr Pointer to memory where a pointer to the MMIO space will be returned. May be NULL, in which case no pointer is returned. Returned address may be NULL if underlying platform does not support memory mapping for register access.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle. FPGA_NO_ACCESS if the process' permissions are not sufficient to map the requested MMIO space. FPGA_NOT_SUPPORTED if platform does not support memory mapped IO.
"},{"location":"opae-code/mmio_8h/#function-fpgareadmmio32","title":"function fpgaReadMMIO32","text":"Read 32 bit value from MMIO space.
fpga_result fpgaReadMMIO32 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint32_t * value\n)
This function will read from MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Pointer to memory where read value is returned (32 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgareadmmio64","title":"function fpgaReadMMIO64","text":"Read 64 bit value from MMIO space.
fpga_result fpgaReadMMIO64 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint64_t * value\n)
This function will read from MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Pointer to memory where read value is returned (64 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgaunmapmmio","title":"function fpgaUnmapMMIO","text":"Unmap MMIO space.
fpga_result fpgaUnmapMMIO (\nfpga_handle handle,\nuint32_t mmio_num\n)
This function will unmap a previously mapped MMIO space of the target object, rendering any pointers to it invalid.
Note:
This call is only supported by hardware targets, not by ASE simulation.
Parameters:
handle Handle to previously opened resource mmio_num Number of MMIO space to access
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgawritemmio32","title":"function fpgaWriteMMIO32","text":"Write 32 bit value to MMIO space.
fpga_result fpgaWriteMMIO32 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint32_t value\n)
This function will write to MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Value to write (32 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgawritemmio512","title":"function fpgaWriteMMIO512","text":"Write 512 bit value to MMIO space.
fpga_result fpgaWriteMMIO512 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nconst void * value\n)
512 bit MMIO writes may not be supported on all platforms.
This function will write to MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Pointer to memory holding value to write (512 bits)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgawritemmio64","title":"function fpgaWriteMMIO64","text":"Write 64 bit value to MMIO space.
fpga_result fpgaWriteMMIO64 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint64_t value\n)
This function will write to MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Value to write (64 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
The documentation for this class was generated from the following file docs/sw/include/opae/mmio.h
"},{"location":"opae-code/mmio_8h_source/","title":"File mmio.h","text":"File List > docs > sw > include > opae > mmio.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_MMIO_H__\n#define __FPGA_MMIO_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaWriteMMIO64(fpga_handle handle,\nuint32_t mmio_num, uint64_t offset,\nuint64_t value);\nfpga_result fpgaReadMMIO64(fpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset, uint64_t *value);\nfpga_result fpgaWriteMMIO32(fpga_handle handle,\nuint32_t mmio_num, uint64_t offset,\nuint32_t value);\nfpga_result fpgaReadMMIO32(fpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset, uint32_t *value);\nfpga_result fpgaWriteMMIO512(fpga_handle handle,\nuint32_t mmio_num, uint64_t offset,\nconst void *value);\nfpga_result fpgaMapMMIO(fpga_handle handle,\nuint32_t mmio_num, uint64_t **mmio_ptr);\nfpga_result fpgaUnmapMMIO(fpga_handle handle,\nuint32_t mmio_num);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_MMIO_H__\n
"},{"location":"opae-code/properties_8h/","title":"File properties.h","text":"FileList > docs > sw > include > opae > properties.h
Go to the source code of this file.
Functions for examining and manipulating fpga_properties objects.More...
#include <opae/types.h>
"},{"location":"opae-code/properties_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaClearProperties (fpga_properties prop) Clear a fpga_properties object. fpga_result fpgaCloneProperties (fpga_properties src, fpga_properties * dst) Clone a fpga_properties object. fpga_result fpgaDestroyProperties (fpga_properties * prop) Destroy a fpga_properties object. fpga_result fpgaGetProperties (fpga_token token, fpga_properties * prop) Create a fpga_properties object. fpga_result fpgaGetPropertiesFromHandle (fpga_handle handle, fpga_properties * prop) Create a fpga_properties object. fpga_result fpgaPropertiesGetAcceleratorState (const fpga_properties prop, fpga_accelerator_state * state) Get the state of a accelerator resource property. fpga_result fpgaPropertiesGetBBSID (const fpga_properties prop, uint64_t * bbs_id) Get the BBS ID of an FPGA resource property. fpga_result fpgaPropertiesGetBBSVersion (const fpga_properties prop, fpga_version * bbs_version) Get the BBS Version of an FPGA resource property. fpga_result fpgaPropertiesGetBus (const fpga_properties prop, uint8_t * bus) Get the PCI bus number of a resource. fpga_result fpgaPropertiesGetCapabilities (const fpga_properties prop, uint64_t * capabilities) Get the capabilities FPGA resource property. fpga_result fpgaPropertiesGetDevice (const fpga_properties prop, uint8_t * device) Get the PCI device number of a resource. fpga_result fpgaPropertiesGetDeviceID (const fpga_properties prop, uint16_t * device_id) Get the device id of the resource. fpga_result fpgaPropertiesGetFunction (const fpga_properties prop, uint8_t * function) Get the PCI function number of a resource. fpga_result fpgaPropertiesGetGUID (const fpga_properties prop, fpga_guid * guid) Get the GUID of a resource. fpga_result fpgaPropertiesGetInterface (const fpga_properties prop, fpga_interface * interface) Get the OPAE plugin interface implemented by a resource. fpga_result fpgaPropertiesGetLocalMemorySize (const fpga_properties prop, uint64_t * lms) Get the local memory size of an FPGA resource property. fpga_result fpgaPropertiesGetModel (const fpga_properties prop, char * model) Get the model of an FPGA resource property. fpga_result fpgaPropertiesGetNumErrors (const fpga_properties prop, uint32_t * num_errors) Get the number of errors that can be reported by a resource. fpga_result fpgaPropertiesGetNumInterrupts (const fpga_properties prop, uint32_t * num_interrupts) Get the number of interrupts. fpga_result fpgaPropertiesGetNumMMIO (const fpga_properties prop, uint32_t * mmio_spaces) Get the number of mmio spaces. fpga_result fpgaPropertiesGetNumSlots (const fpga_properties prop, uint32_t * num_slots) Get the number of slots of an FPGA resource property. fpga_result fpgaPropertiesGetObjectID (const fpga_properties prop, uint64_t * object_id) Get the object ID of a resource. fpga_result fpgaPropertiesGetObjectType (const fpga_properties prop, fpga_objtype * objtype) Get the object type of a resource. fpga_result fpgaPropertiesGetParent (const fpga_properties prop, fpga_token * parent) Get the token of the parent object. fpga_result fpgaPropertiesGetSegment (const fpga_properties prop, uint16_t * segment) Get the PCI segment number of a resource. fpga_result fpgaPropertiesGetSocketID (const fpga_properties prop, uint8_t * socket_id) Get the socket id of a resource. fpga_result fpgaPropertiesGetSubsystemDeviceID (const fpga_properties prop, uint16_t * subsystem_device_id) Get the subsystem device id of an FPGA resource property. fpga_result fpgaPropertiesGetSubsystemVendorID (const fpga_properties prop, uint16_t * subsystem_vendor_id) Get the subsystem vendor id of an FPGA resource property. fpga_result fpgaPropertiesGetVendorID (const fpga_properties prop, uint16_t * vendor_id) Get the vendor id of an FPGA resource property. fpga_result fpgaPropertiesSetAcceleratorState (fpga_properties prop, fpga_accelerator_state state) Set the state of an accelerator resource property. fpga_result fpgaPropertiesSetBBSID (fpga_properties prop, uint64_t bbs_id) Set the BBS ID of an FPGA resource property. fpga_result fpgaPropertiesSetBBSVersion (fpga_properties prop, fpga_version version) Set the BBS Version of an FPGA resource property. fpga_result fpgaPropertiesSetBus (fpga_properties prop, uint8_t bus) Set the PCI bus number of a resource. fpga_result fpgaPropertiesSetCapabilities (fpga_properties prop, uint64_t capabilities) Set the capabilities of an FPGA resource property. fpga_result fpgaPropertiesSetDevice (fpga_properties prop, uint8_t device) Set the PCI device number of a resource. fpga_result fpgaPropertiesSetDeviceID (fpga_properties prop, uint16_t device_id) Set the device id of the resource. fpga_result fpgaPropertiesSetFunction (fpga_properties prop, uint8_t function) Set the PCI function number of a resource. fpga_result fpgaPropertiesSetGUID (fpga_properties prop, fpga_guid guid) Set the GUID of a resource. fpga_result fpgaPropertiesSetInterface (const fpga_properties prop, fpga_interface interface) Set the OPAE plugin interface implemented by a resource. fpga_result fpgaPropertiesSetLocalMemorySize (fpga_properties prop, uint64_t lms) Set the local memory size of an FPGA resource property. fpga_result fpgaPropertiesSetModel (fpga_properties prop, char * model) Set the model of an FPGA resource property. fpga_result fpgaPropertiesSetNumErrors (const fpga_properties prop, uint32_t num_errors) Set the number of error registers. fpga_result fpgaPropertiesSetNumInterrupts (fpga_properties prop, uint32_t num_interrupts) Set the number of interrupts. fpga_result fpgaPropertiesSetNumMMIO (fpga_properties prop, uint32_t mmio_spaces) Set the number of mmio spaces. fpga_result fpgaPropertiesSetNumSlots (fpga_properties prop, uint32_t num_slots) Set the number of slots of an FPGA resource property. fpga_result fpgaPropertiesSetObjectID (const fpga_properties prop, uint64_t object_id) Set the object ID of a resource. fpga_result fpgaPropertiesSetObjectType (fpga_properties prop, fpga_objtype objtype) Set the object type of a resource. fpga_result fpgaPropertiesSetParent (fpga_properties prop, fpga_token parent) Set the token of the parent object. fpga_result fpgaPropertiesSetSegment (fpga_properties prop, uint16_t segment) Set the PCI segment number of a resource. fpga_result fpgaPropertiesSetSocketID (fpga_properties prop, uint8_t socket_id) Set the socket id of the resource. fpga_result fpgaPropertiesSetSubsystemDeviceID (fpga_properties prop, uint16_t subsystem_device_id) Set the subsystem device id of an FPGA resource property. fpga_result fpgaPropertiesSetSubsystemVendorID (fpga_properties prop, uint16_t subsystem_vendor_id) Set the subsystem vendor id of an FPGA resource property. fpga_result fpgaPropertiesSetVendorID (fpga_properties prop, uint16_t vendor_id) Set the vendor id of an FPGA resource property. fpga_result fpgaUpdateProperties (fpga_token token, fpga_properties prop) Update a fpga_properties object."},{"location":"opae-code/properties_8h/#detailed-description","title":"Detailed Description","text":"In OPAE, fpga_properties objects are used both for obtaining information about resources and for selectively enumerating resources based on their properties. This file provides accessor functions (get/set) to allow reading and writing individual items of an fpga_properties object. Generally, not all object types supported by OPAE carry all properties. If you call a property accessor method on a fpga_properties object that does not support this particular property, it will return FPGA_INVALID_PARAM.
"},{"location":"opae-code/properties_8h/#accessor-return-values","title":"Accessor Return Values","text":"In addition to the return values specified in the documentation below, all accessor functions return FPGA_OK on success, FPGA_INVALID_PARAM if you pass NULL or invalid parameters (i.e. non-initialized properties objects), FPGA_EXCEPTION if an internal exception occurred trying to access the properties object, FPGA_NOT_FOUND if the requested property is not part of the supplied properties object.
"},{"location":"opae-code/properties_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/properties_8h/#function-fpgaclearproperties","title":"function fpgaClearProperties","text":"Clear a fpga_properties object.
fpga_result fpgaClearProperties (\nfpga_properties prop\n)
Sets all fields of the properties object pointed at by 'prop' to 'don't care', which implies that the fpga_properties object would match all FPGA resources if used for an fpgaEnumerate() query. The matching criteria can be further refined by using fpgaSet* functions on the properties object.
Instead of creating a new fpga_properties object every time, this function can be used to re-use fpga_properties objects from previous queries.
Parameters:
prop fpga_properties object to clear
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if prop is not a valid object. FPGA_EXCEPTION if an * internal exception occured when trying to access prop.
"},{"location":"opae-code/properties_8h/#function-fpgacloneproperties","title":"function fpgaCloneProperties","text":"Clone a fpga_properties object.
fpga_result fpgaCloneProperties (\nfpga_properties src,\nfpga_properties * dst\n)
Creates a copy of an fpga_properties object.
Note:
This call creates a new properties object and allocates memory for it. Both the 'src' and the newly created 'dst' objects will eventually need to be destroyed using fpgaDestroyProperties().
Parameters:
src fpga_properties object to copy dst New fpga_properties object cloned from 'src'
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if src is not a valid object, or if dst is NULL. FPGA_NO_MEMORY if there was not enough memory to allocate an fpga_properties object for dst. FPGA_EXCEPTION if an internal exception occurred either accessing src or updating dst.
"},{"location":"opae-code/properties_8h/#function-fpgadestroyproperties","title":"function fpgaDestroyProperties","text":"Destroy a fpga_properties object.
fpga_result fpgaDestroyProperties (\nfpga_properties * prop\n)
Destroys an existing fpga_properties object that the caller has previously created using fpgaGetProperties() or fpgaCloneProperties().
Note:
fpgaDestroyProperties() requires the address of an fpga_properties object, similar to fpgaGetPropertiesFromHandle(), fpgaGetProperties(), and fpgaCloneProperties(). Passing any other value results in undefined behavior.
Parameters:
prop Pointer to the fpga_properties object to destroy
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM is prop is not a valid object. FPGA_EXCEPTION if an internal exception occurrred while trying to access prop.
"},{"location":"opae-code/properties_8h/#function-fpgagetproperties","title":"function fpgaGetProperties","text":"Create a fpga_properties object.
fpga_result fpgaGetProperties (\nfpga_token token,\nfpga_properties * prop\n)
Initializes the memory pointed at by prop to represent a properties object, and populates it with the properties of the resource referred to by token. Individual properties can then be queried using fpgaPropertiesGet*() accessor functions.
If token is NULL, an \"empty\" properties object is created to be used as a filter for fpgaEnumerate(). All individual fields are set to dont care`, which implies that the fpga_properties object would match all FPGA resources if used for an fpgaEnumerate() query. The matching criteria can be further refined by using fpgaSet* functions on the properties object, or the object can be populated with the actual properties of a resource by using fpgaUpdateProperties().
Note:
fpgaGetProperties() will allocate memory for the created properties object returned in prop. It is the responsibility of the caller to free this memory after use by calling fpgaDestroyProperties().
Parameters:
token Token to get properties for. Can be NULL, which will create an empty properties object to be used as a filter for fpgaEnumerate(). prop Pointer to a variable of type fpga_properties
Returns:
FPGA_OK on success. FPGA_NO_MEMORY if no memory could be allocated to create the fpga_properties object. FPGA_EXCEPTION if an exception happend while initializing the fpga_properties object.
"},{"location":"opae-code/properties_8h/#function-fpgagetpropertiesfromhandle","title":"function fpgaGetPropertiesFromHandle","text":"Create a fpga_properties object.
fpga_result fpgaGetPropertiesFromHandle (\nfpga_handle handle,\nfpga_properties * prop\n)
Initializes the memory pointed at by prop to represent a properties object, and populates it with the properties of the resource referred to by handle. Individual properties can then be queried using fpgaPropertiesGet*() accessor functions.
Note:
fpgaGetPropertiesFromHandle() will allocate memory for the created properties object returned in prop. It is the responsibility of the caller to free this memory after use by calling fpgaDestroyProperties().
Parameters:
handle Open handle to get properties for. prop Pointer to a variable of type fpga_properties
Returns:
FPGA_OK on success. FPGA_NO_MEMORY if no memory could be allocated to create the fpga_properties object. FPGA_EXCEPTION if an exception happend while initializing the fpga_properties object.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetacceleratorstate","title":"function fpgaPropertiesGetAcceleratorState","text":"Get the state of a accelerator resource property.
fpga_result fpgaPropertiesGetAcceleratorState (\nconst fpga_properties prop,\nfpga_accelerator_state * state\n)
Returns the accelerator state of a accelerator.
Parameters:
prop Properties object to query - must be of type FPGA_ACCELERATOR state Pointer to a accelerator state variable of the accelerator
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetbbsid","title":"function fpgaPropertiesGetBBSID","text":"Get the BBS ID of an FPGA resource property.
fpga_result fpgaPropertiesGetBBSID (\nconst fpga_properties prop,\nuint64_t * bbs_id\n)
Returns the blue bitstream id of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE bbs_id Pointer to a bbs id variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetbbsversion","title":"function fpgaPropertiesGetBBSVersion","text":"Get the BBS Version of an FPGA resource property.
fpga_result fpgaPropertiesGetBBSVersion (\nconst fpga_properties prop,\nfpga_version * bbs_version\n)
Returns the blue bitstream version of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE bbs_version Pointer to a bbs version variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetbus","title":"function fpgaPropertiesGetBus","text":"Get the PCI bus number of a resource.
fpga_result fpgaPropertiesGetBus (\nconst fpga_properties prop,\nuint8_t * bus\n)
Returns the bus number the queried resource.
Parameters:
prop Properties object to query bus Pointer to a PCI bus variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetcapabilities","title":"function fpgaPropertiesGetCapabilities","text":"Get the capabilities FPGA resource property.
fpga_result fpgaPropertiesGetCapabilities (\nconst fpga_properties prop,\nuint64_t * capabilities\n)
Returns the capabilities of an FPGA. Capabilities is a bitfield value
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE capabilities Pointer to a capabilities variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetdevice","title":"function fpgaPropertiesGetDevice","text":"Get the PCI device number of a resource.
fpga_result fpgaPropertiesGetDevice (\nconst fpga_properties prop,\nuint8_t * device\n)
Returns the device number the queried resource.
Parameters:
prop Properties object to query device Pointer to a PCI device variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetdeviceid","title":"function fpgaPropertiesGetDeviceID","text":"Get the device id of the resource.
fpga_result fpgaPropertiesGetDeviceID (\nconst fpga_properties prop,\nuint16_t * device_id\n)
Parameters:
prop Properties object to query device_id Pointer to a device id variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetfunction","title":"function fpgaPropertiesGetFunction","text":"Get the PCI function number of a resource.
fpga_result fpgaPropertiesGetFunction (\nconst fpga_properties prop,\nuint8_t * function\n)
Returns the function number the queried resource.
Parameters:
prop Properties object to query function Pointer to PCI function variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetguid","title":"function fpgaPropertiesGetGUID","text":"Get the GUID of a resource.
fpga_result fpgaPropertiesGetGUID (\nconst fpga_properties prop,\nfpga_guid * guid\n)
Returns the GUID of an FPGA or accelerator object.
For an accelerator, the GUID uniquely identifies a specific accelerator context type, i.e. different accelerators will have different GUIDs. For an FPGA, the GUID is used to identify a certain instance of an FPGA, e.g. to determine whether a given bitstream would be compatible.
Parameters:
prop Properties object to query guid Pointer to a GUID of the slot variable
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetinterface","title":"function fpgaPropertiesGetInterface","text":"Get the OPAE plugin interface implemented by a resource.
fpga_result fpgaPropertiesGetInterface (\nconst fpga_properties prop,\nfpga_interface * interface\n)
Returns the plugin interface enumerator.
Parameters:
prop Properties object to query interface Pointer to an fpga_interface location to store the interface in
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetlocalmemorysize","title":"function fpgaPropertiesGetLocalMemorySize","text":"Get the local memory size of an FPGA resource property.
fpga_result fpgaPropertiesGetLocalMemorySize (\nconst fpga_properties prop,\nuint64_t * lms\n)
Returns the local memory size of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE lms Pointer to a memory size variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetmodel","title":"function fpgaPropertiesGetModel","text":"Get the model of an FPGA resource property.
fpga_result fpgaPropertiesGetModel (\nconst fpga_properties prop,\nchar * model\n)
Returns the model of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE model Model of the FPGA resource (string of minimum FPGA_MODEL_LENGTH length
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnumerrors","title":"function fpgaPropertiesGetNumErrors","text":"Get the number of errors that can be reported by a resource.
fpga_result fpgaPropertiesGetNumErrors (\nconst fpga_properties prop,\nuint32_t * num_errors\n)
Returns the number of error registers understood by a resource.
Parameters:
prop Properties object to query num_errors Pointer to a 32 bit memory location to store the number of supported errors in
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnuminterrupts","title":"function fpgaPropertiesGetNumInterrupts","text":"Get the number of interrupts.
fpga_result fpgaPropertiesGetNumInterrupts (\nconst fpga_properties prop,\nuint32_t * num_interrupts\n)
Returns the number of interrupts of an accelerator properties structure.
Parameters:
prop Properties object to query - must be of type FPGA_ACCELERATOR num_interrupts Pointer to a variable for number of interrupts
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnummmio","title":"function fpgaPropertiesGetNumMMIO","text":"Get the number of mmio spaces.
fpga_result fpgaPropertiesGetNumMMIO (\nconst fpga_properties prop,\nuint32_t * mmio_spaces\n)
Returns the number of mmio spaces of an AFU properties structure.
Parameters:
prop Properties object to query - must be of type FPGA_ACCELERATOR mmio_spaces Pointer to a variable for number of mmio spaces
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnumslots","title":"function fpgaPropertiesGetNumSlots","text":"Get the number of slots of an FPGA resource property.
fpga_result fpgaPropertiesGetNumSlots (\nconst fpga_properties prop,\nuint32_t * num_slots\n)
Returns the number of slots present in an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE num_slots Pointer to number of slots variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetobjectid","title":"function fpgaPropertiesGetObjectID","text":"Get the object ID of a resource.
fpga_result fpgaPropertiesGetObjectID (\nconst fpga_properties prop,\nuint64_t * object_id\n)
Returns the object ID of a resource. The object ID is a 64 bit identifier that is unique within a single node or system. It represents a similar concept as the token, but can be used across processes (e.g. passed on the command line).
Parameters:
prop Properties object to query object_id Pointer to a 64bit memory location to store the object ID in
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetobjecttype","title":"function fpgaPropertiesGetObjectType","text":"Get the object type of a resource.
fpga_result fpgaPropertiesGetObjectType (\nconst fpga_properties prop,\nfpga_objtype * objtype\n)
Returns the object type of the queried resource.
Parameters:
prop Properties object to query objtype Pointer to an object type variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetparent","title":"function fpgaPropertiesGetParent","text":"Get the token of the parent object.
fpga_result fpgaPropertiesGetParent (\nconst fpga_properties prop,\nfpga_token * parent\n)
Returns the token of the parent of the queried resource in '*parent'.
Parameters:
prop Properties object to query parent Pointer to a token variable of the resource 'prop' is associated with
Returns:
FPGA_NOT_FOUND if resource does not have a parent (e.g. an FPGA_DEVICE resource does not have parents). Also see \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsegment","title":"function fpgaPropertiesGetSegment","text":"Get the PCI segment number of a resource.
fpga_result fpgaPropertiesGetSegment (\nconst fpga_properties prop,\nuint16_t * segment\n)
Returns the segment number of the queried resource.
Parameters:
prop Properties object to query segment Pointer to a PCI segment variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsocketid","title":"function fpgaPropertiesGetSocketID","text":"Get the socket id of a resource.
fpga_result fpgaPropertiesGetSocketID (\nconst fpga_properties prop,\nuint8_t * socket_id\n)
Returns the socket id of the queried resource.
Parameters:
prop Properties object to query socket_id Pointer to a socket id variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsubsystemdeviceid","title":"function fpgaPropertiesGetSubsystemDeviceID","text":"Get the subsystem device id of an FPGA resource property.
fpga_result fpgaPropertiesGetSubsystemDeviceID (\nconst fpga_properties prop,\nuint16_t * subsystem_device_id\n)
Returns the subsystem device id of an FPGA.
Parameters:
prop Properties object to query subsystem_device_id Pointer to a device id variable of the FPGA
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsubsystemvendorid","title":"function fpgaPropertiesGetSubsystemVendorID","text":"Get the subsystem vendor id of an FPGA resource property.
fpga_result fpgaPropertiesGetSubsystemVendorID (\nconst fpga_properties prop,\nuint16_t * subsystem_vendor_id\n)
Returns the subsystem vendor id of an FPGA.
Parameters:
prop Properties object to query subsystem_vendor_id Pointer to a vendor id variable of the FPGA
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetvendorid","title":"function fpgaPropertiesGetVendorID","text":"Get the vendor id of an FPGA resource property.
fpga_result fpgaPropertiesGetVendorID (\nconst fpga_properties prop,\nuint16_t * vendor_id\n)
Returns the vendor id of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE vendor_id Pointer to a vendor id variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetacceleratorstate","title":"function fpgaPropertiesSetAcceleratorState","text":"Set the state of an accelerator resource property.
fpga_result fpgaPropertiesSetAcceleratorState (\nfpga_properties prop,\nfpga_accelerator_state state\n)
Parameters:
prop Properties object to modify - must be of type FPGA_ACCELERATOR state accelerator state of the accelerator resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetbbsid","title":"function fpgaPropertiesSetBBSID","text":"Set the BBS ID of an FPGA resource property.
fpga_result fpgaPropertiesSetBBSID (\nfpga_properties prop,\nuint64_t bbs_id\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE bbs_id Blue bitstream id of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetbbsversion","title":"function fpgaPropertiesSetBBSVersion","text":"Set the BBS Version of an FPGA resource property.
fpga_result fpgaPropertiesSetBBSVersion (\nfpga_properties prop,\nfpga_version version\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE version Blue bitstream version of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetbus","title":"function fpgaPropertiesSetBus","text":"Set the PCI bus number of a resource.
fpga_result fpgaPropertiesSetBus (\nfpga_properties prop,\nuint8_t bus\n)
Parameters:
prop Properties object to modify bus PCI bus number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetcapabilities","title":"function fpgaPropertiesSetCapabilities","text":"Set the capabilities of an FPGA resource property.
fpga_result fpgaPropertiesSetCapabilities (\nfpga_properties prop,\nuint64_t capabilities\n)
Capabilities is a bitfield value
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE capabilities Capabilities of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetdevice","title":"function fpgaPropertiesSetDevice","text":"Set the PCI device number of a resource.
fpga_result fpgaPropertiesSetDevice (\nfpga_properties prop,\nuint8_t device\n)
Enforces the limitation on the number of devices as specified in the PCI spec.
Parameters:
prop Properties object to modify device PCI device number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetdeviceid","title":"function fpgaPropertiesSetDeviceID","text":"Set the device id of the resource.
fpga_result fpgaPropertiesSetDeviceID (\nfpga_properties prop,\nuint16_t device_id\n)
Parameters:
prop Properties object to modify device_id Device id of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetfunction","title":"function fpgaPropertiesSetFunction","text":"Set the PCI function number of a resource.
fpga_result fpgaPropertiesSetFunction (\nfpga_properties prop,\nuint8_t function\n)
Enforces the limitation on the number of functions as specified in the PCI spec.
Parameters:
prop Properties object to modify function PCI function number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetguid","title":"function fpgaPropertiesSetGUID","text":"Set the GUID of a resource.
fpga_result fpgaPropertiesSetGUID (\nfpga_properties prop,\nfpga_guid guid\n)
Sets the GUID of an FPGA or accelerator object.
For an accelerator, the GUID uniquely identifies a specific accelerator context type, i.e. different accelerators will have different GUIDs. For an FPGA, the GUID is used to identify a certain instance of an FPGA, e.g. to determine whether a given bitstream would be compatible.
Parameters:
prop Properties object to modify guid Pointer to a GUID of the slot variable
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetinterface","title":"function fpgaPropertiesSetInterface","text":"Set the OPAE plugin interface implemented by a resource.
fpga_result fpgaPropertiesSetInterface (\nconst fpga_properties prop,\nfpga_interface interface\n)
Set the plugin interface enumerator.
Parameters:
prop Properties object to query interface The interface enumerator to set
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetlocalmemorysize","title":"function fpgaPropertiesSetLocalMemorySize","text":"Set the local memory size of an FPGA resource property.
fpga_result fpgaPropertiesSetLocalMemorySize (\nfpga_properties prop,\nuint64_t lms\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE lms Local memory size of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetmodel","title":"function fpgaPropertiesSetModel","text":"Set the model of an FPGA resource property.
fpga_result fpgaPropertiesSetModel (\nfpga_properties prop,\nchar * model\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE model Model of the FPGA resource (string of maximum FPGA_MODEL_LENGTH length
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnumerrors","title":"function fpgaPropertiesSetNumErrors","text":"Set the number of error registers.
fpga_result fpgaPropertiesSetNumErrors (\nconst fpga_properties prop,\nuint32_t num_errors\n)
Set the number of error registers understood by a resource to enumerate.
Parameters:
prop Properties object to query num_errors Number of errors
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnuminterrupts","title":"function fpgaPropertiesSetNumInterrupts","text":"Set the number of interrupts.
fpga_result fpgaPropertiesSetNumInterrupts (\nfpga_properties prop,\nuint32_t num_interrupts\n)
Sets the number of interrupts of an accelerator properties structure.
Parameters:
prop Properties object to modify - must be of type FPGA_ACCELERATOR num_interrupts Number of interrupts of the accelerator
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnummmio","title":"function fpgaPropertiesSetNumMMIO","text":"Set the number of mmio spaces.
fpga_result fpgaPropertiesSetNumMMIO (\nfpga_properties prop,\nuint32_t mmio_spaces\n)
Sets the number of mmio spaces of an AFU properties structure.
Parameters:
prop Properties object to modify - must be of type FPGA_ACCELERATOR mmio_spaces Number of MMIO spaces of the accelerator
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnumslots","title":"function fpgaPropertiesSetNumSlots","text":"Set the number of slots of an FPGA resource property.
fpga_result fpgaPropertiesSetNumSlots (\nfpga_properties prop,\nuint32_t num_slots\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE num_slots Number of slots of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetobjectid","title":"function fpgaPropertiesSetObjectID","text":"Set the object ID of a resource.
fpga_result fpgaPropertiesSetObjectID (\nconst fpga_properties prop,\nuint64_t object_id\n)
Sets the object ID of a resource. The object ID is a 64 bit identifier that is unique within a single node or system. It represents a similar concept as the token, but can be used across processes (e.g. passed on the command line).
Parameters:
prop Properties object to query object_id A 64bit value to use as the object ID
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetobjecttype","title":"function fpgaPropertiesSetObjectType","text":"Set the object type of a resource.
fpga_result fpgaPropertiesSetObjectType (\nfpga_properties prop,\nfpga_objtype objtype\n)
Sets the object type of the resource. * Currently supported object types are FPGA_DEVICE and FPGA_ACCELERATOR.
Parameters:
prop Properties object to modify objtype Object type of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetparent","title":"function fpgaPropertiesSetParent","text":"Set the token of the parent object.
fpga_result fpgaPropertiesSetParent (\nfpga_properties prop,\nfpga_token parent\n)
Parameters:
prop Properties object to modify parent Pointer to a token variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsegment","title":"function fpgaPropertiesSetSegment","text":"Set the PCI segment number of a resource.
fpga_result fpgaPropertiesSetSegment (\nfpga_properties prop,\nuint16_t segment\n)
Parameters:
prop Properties object to modify segment PCI segment number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsocketid","title":"function fpgaPropertiesSetSocketID","text":"Set the socket id of the resource.
fpga_result fpgaPropertiesSetSocketID (\nfpga_properties prop,\nuint8_t socket_id\n)
Parameters:
prop Properties object to modify socket_id Socket id of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsubsystemdeviceid","title":"function fpgaPropertiesSetSubsystemDeviceID","text":"Set the subsystem device id of an FPGA resource property.
fpga_result fpgaPropertiesSetSubsystemDeviceID (\nfpga_properties prop,\nuint16_t subsystem_device_id\n)
Parameters:
prop Properties object to modify subsystem_device_id Subsystem Device id of the FPGA resource
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsubsystemvendorid","title":"function fpgaPropertiesSetSubsystemVendorID","text":"Set the subsystem vendor id of an FPGA resource property.
fpga_result fpgaPropertiesSetSubsystemVendorID (\nfpga_properties prop,\nuint16_t subsystem_vendor_id\n)
Parameters:
prop Properties object to modify subsystem_vendor_id Subsystem Vendor id of the FPGA resource
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetvendorid","title":"function fpgaPropertiesSetVendorID","text":"Set the vendor id of an FPGA resource property.
fpga_result fpgaPropertiesSetVendorID (\nfpga_properties prop,\nuint16_t vendor_id\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE vendor_id Vendor id of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgaupdateproperties","title":"function fpgaUpdateProperties","text":"Update a fpga_properties object.
fpga_result fpgaUpdateProperties (\nfpga_token token,\nfpga_properties prop\n)
Populates the properties object 'prop' with properties of the resource referred to by 'token'. Unlike fpgaGetProperties(), this call will not create a new properties object or allocate memory for it, but use a previously created properties object.
Parameters:
token Token to retrieve properties for prop fpga_properties object to update
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if token or prop are not valid objects. FPGA_NOT_FOUND if the resource referred to by token was not found. FPGA_NO_DRIVER if not driver is loaded. FPGA_EXCEPTION if an internal exception occured when trying to update prop.
The documentation for this class was generated from the following file docs/sw/include/opae/properties.h
"},{"location":"opae-code/properties_8h_source/","title":"File properties.h","text":"File List > docs > sw > include > opae > properties.h
Go to the documentation of this file.
// Copyright(c) 2017-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_PROPERTIES_H__\n#define __FPGA_PROPERTIES_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetPropertiesFromHandle(fpga_handle handle, fpga_properties *prop);\nfpga_result fpgaGetProperties(fpga_token token, fpga_properties *prop);\nfpga_result fpgaUpdateProperties(fpga_token token, fpga_properties prop);\nfpga_result fpgaClearProperties(fpga_properties prop);\nfpga_result fpgaCloneProperties(fpga_properties src, fpga_properties *dst);\nfpga_result fpgaDestroyProperties(fpga_properties *prop);\nfpga_result fpgaPropertiesGetParent(const fpga_properties prop,\nfpga_token *parent);\nfpga_result fpgaPropertiesSetParent(fpga_properties prop,\nfpga_token parent);\nfpga_result fpgaPropertiesGetObjectType(const fpga_properties prop,\nfpga_objtype *objtype);\nfpga_result fpgaPropertiesSetObjectType(fpga_properties prop,\nfpga_objtype objtype);\nfpga_result fpgaPropertiesGetSegment(const fpga_properties prop, uint16_t *segment);\nfpga_result fpgaPropertiesSetSegment(fpga_properties prop, uint16_t segment);\nfpga_result fpgaPropertiesGetBus(const fpga_properties prop, uint8_t *bus);\nfpga_result fpgaPropertiesSetBus(fpga_properties prop, uint8_t bus);\nfpga_result fpgaPropertiesGetDevice(const fpga_properties prop,\nuint8_t *device);\nfpga_result fpgaPropertiesSetDevice(fpga_properties prop,\nuint8_t device);\nfpga_result fpgaPropertiesGetFunction(const fpga_properties prop,\nuint8_t *function);\nfpga_result fpgaPropertiesSetFunction(fpga_properties prop,\nuint8_t function);\nfpga_result fpgaPropertiesGetSocketID(const fpga_properties prop,\nuint8_t *socket_id);\nfpga_result fpgaPropertiesSetSocketID(fpga_properties prop,\nuint8_t socket_id);\nfpga_result fpgaPropertiesGetDeviceID(const fpga_properties prop,\nuint16_t *device_id);\nfpga_result fpgaPropertiesSetDeviceID(fpga_properties prop,\nuint16_t device_id);\nfpga_result fpgaPropertiesGetNumSlots(const fpga_properties prop,\nuint32_t *num_slots);\nfpga_result fpgaPropertiesSetNumSlots(fpga_properties prop,\nuint32_t num_slots);\nfpga_result fpgaPropertiesGetBBSID(const fpga_properties prop,\nuint64_t *bbs_id);\nfpga_result fpgaPropertiesSetBBSID(fpga_properties prop,\nuint64_t bbs_id);\nfpga_result fpgaPropertiesGetBBSVersion(const fpga_properties prop,\nfpga_version *bbs_version);\nfpga_result fpgaPropertiesSetBBSVersion(fpga_properties prop,\nfpga_version version);\nfpga_result fpgaPropertiesGetVendorID(const fpga_properties prop,\nuint16_t *vendor_id);\nfpga_result fpgaPropertiesSetVendorID(fpga_properties prop,\nuint16_t vendor_id);\nfpga_result fpgaPropertiesGetModel(const fpga_properties prop,\nchar *model);\nfpga_result fpgaPropertiesSetModel(fpga_properties prop,\nchar *model);\nfpga_result fpgaPropertiesGetLocalMemorySize(const fpga_properties prop,\nuint64_t *lms);\nfpga_result fpgaPropertiesSetLocalMemorySize(fpga_properties prop,\nuint64_t lms);\nfpga_result fpgaPropertiesGetCapabilities(const fpga_properties prop,\nuint64_t *capabilities);\nfpga_result fpgaPropertiesSetCapabilities(fpga_properties prop,\nuint64_t capabilities);\nfpga_result fpgaPropertiesGetGUID(const fpga_properties prop,\nfpga_guid *guid);\nfpga_result fpgaPropertiesSetGUID(fpga_properties prop, fpga_guid guid);\nfpga_result fpgaPropertiesGetNumMMIO(const fpga_properties prop,\nuint32_t *mmio_spaces);\nfpga_result fpgaPropertiesSetNumMMIO(fpga_properties prop,\nuint32_t mmio_spaces);\nfpga_result fpgaPropertiesGetNumInterrupts(const fpga_properties prop,\nuint32_t *num_interrupts);\nfpga_result fpgaPropertiesSetNumInterrupts(fpga_properties prop,\nuint32_t num_interrupts);\nfpga_result fpgaPropertiesGetAcceleratorState(const fpga_properties prop,\nfpga_accelerator_state *state);\nfpga_result fpgaPropertiesSetAcceleratorState(fpga_properties prop,\nfpga_accelerator_state state);\nfpga_result fpgaPropertiesGetObjectID(const fpga_properties prop,\nuint64_t *object_id);\nfpga_result fpgaPropertiesSetObjectID(const fpga_properties prop,\nuint64_t object_id);\nfpga_result fpgaPropertiesGetNumErrors(const fpga_properties prop,\nuint32_t *num_errors);\nfpga_result fpgaPropertiesSetNumErrors(const fpga_properties prop,\nuint32_t num_errors);\nfpga_result fpgaPropertiesGetInterface(const fpga_properties prop,\nfpga_interface *interface);\nfpga_result fpgaPropertiesSetInterface(const fpga_properties prop,\nfpga_interface interface);\nfpga_result fpgaPropertiesGetSubsystemVendorID(const fpga_properties prop,\nuint16_t *subsystem_vendor_id);\nfpga_result fpgaPropertiesSetSubsystemVendorID(fpga_properties prop,\nuint16_t subsystem_vendor_id);\nfpga_result fpgaPropertiesGetSubsystemDeviceID(const fpga_properties prop,\nuint16_t *subsystem_device_id);\nfpga_result fpgaPropertiesSetSubsystemDeviceID(fpga_properties prop,\nuint16_t subsystem_device_id);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_PROPERTIES_H__\n
"},{"location":"opae-code/sysobject_8h/","title":"File sysobject.h","text":"FileList > docs > sw > include > opae > sysobject.h
Go to the source code of this file.
Functions to read/write from system objects. More...
#include <opae/types.h>
"},{"location":"opae-code/sysobject_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaDestroyObject (fpga_object * obj) Free memory used for the fpga_object data structure. fpga_result fpgaHandleGetObject (fpga_handle handle, const char * name, fpga_object * object, int flags) Create an fpga_object data structure from a handle. fpga_result fpgaObjectGetObject (fpga_object parent, const char * name, fpga_object * object, int flags) Create an fpga_object data structure from a parent object. fpga_result fpgaObjectGetObjectAt (fpga_object parent, size_t idx, fpga_object * object) Create an fpga_object data structure from a parent object using a given index. fpga_result fpgaObjectGetSize (fpga_object obj, uint32_t * value, int flags) Retrieve the size of the object. fpga_result fpgaObjectGetType (fpga_object obj, enum fpga_sysobject_type * type) Get the sysobject type (container or attribute) fpga_result fpgaObjectRead (fpga_object obj, uint8_t * buffer, size_t offset, size_t len, int flags) Read bytes from an FPGA object. fpga_result fpgaObjectRead64 (fpga_object obj, uint64_t * value, int flags) Read a 64-bit value from an FPGA object. fpga_result fpgaObjectWrite64 (fpga_object obj, uint64_t value, int flags) Write 64-bit value to an FPGA object. fpga_result fpgaTokenGetObject (fpga_token token, const char * name, fpga_object * object, int flags) Create an fpga_object data structures."},{"location":"opae-code/sysobject_8h/#detailed-description","title":"Detailed Description","text":"On Linux systems with the OPAE kernel driver, this is used to access sysfs nodes created by the driver.
"},{"location":"opae-code/sysobject_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/sysobject_8h/#function-fpgadestroyobject","title":"function fpgaDestroyObject","text":"Free memory used for the fpga_object data structure.
fpga_result fpgaDestroyObject (\nfpga_object * obj\n)
Note:
fpgaDestroyObject() requires the address of an fpga_object as created by fpgaTokenGetObject(), fpgaHandleGetObject(), or fpgaObjectGetObject(). Passing any other value results in undefind behavior.
Parameters:
obj Pointer to the fpga_object instance to destroy
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if the object is NULL, FPGA_EXCEPTION if an internal error is encountered.
"},{"location":"opae-code/sysobject_8h/#function-fpgahandlegetobject","title":"function fpgaHandleGetObject","text":"Create an fpga_object data structure from a handle.
fpga_result fpgaHandleGetObject (\nfpga_handle handle,\nconst char * name,\nfpga_object * object,\nint flags\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register, or container. This object has read/write access..
Parameters:
handle Handle identifying a resource (accelerator or device) name A key identifying an object belonging to a resource. object Pointer to memory to store the object in flags Control behavior of object identification and creation FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
Note:
Names that begin with '.' or '/' or contain '..' are not allowed and result in FPGA_INVALID_PARAM being returned
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgetobject","title":"function fpgaObjectGetObject","text":"Create an fpga_object data structure from a parent object.
fpga_result fpgaObjectGetObject (\nfpga_object parent,\nconst char * name,\nfpga_object * object,\nint flags\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register, or container. If the parent object was created with a handle, then the new object will inherit the handle allowing it to have read-write access to the object data.
Parameters:
parent A parent container fpga_object. name A key identifying a sub-object of the parent container. object Pointer to memory to store the object in. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid - this includes a parent object that is not a container object. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
Note:
Names that begin with '.' or '/' or contain '..' are not allowed and result in FPGA_INVALID_PARAM being returned
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgetobjectat","title":"function fpgaObjectGetObjectAt","text":"Create an fpga_object data structure from a parent object using a given index.
fpga_result fpgaObjectGetObjectAt (\nfpga_object parent,\nsize_t idx,\nfpga_object * object\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register, or container. If the parent object was created with a handle, then the new object will inherit the handle allowing it to have read-write access to the object data.
Parameters:
parent A parent container 'fpga_object' idx A positive index less than the size reported by the parent. object Pointer to memory to store the object in.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid - this includes a parent object that is not a container object. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgetsize","title":"function fpgaObjectGetSize","text":"Retrieve the size of the object.
fpga_result fpgaObjectGetSize (\nfpga_object obj,\nuint32_t * value,\nint flags\n)
Parameters:
obj An fpga_object instance. value Pointer to variable to store size in. flags Flags that control how the object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the size.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of supplied parameters is invalid. FPGA_EXCEPTION if error occurred.
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgettype","title":"function fpgaObjectGetType","text":"Get the sysobject type (container or attribute)
fpga_result fpgaObjectGetType (\nfpga_object obj,\nenum fpga_sysobject_type * type\n)
Parameters:
obj An fpga_object instance type The type of object (FPGA_OBJECT_CONTAINER or FPGA_OBJECT_ATTRIBUTE)
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters are null or invalid
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectread","title":"function fpgaObjectRead","text":"Read bytes from an FPGA object.
fpga_result fpgaObjectRead (\nfpga_object obj,\nuint8_t * buffer,\nsize_t offset,\nsize_t len,\nint flags\n)
Parameters:
obj An fpga_object instance. buffer Pointer to a buffer to read bytes into. offset Byte offset relative to objects internal buffer where to begin reading bytes from. len The length, in bytes, to read from the object. flags Flags that control how object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters is invalid
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectread64","title":"function fpgaObjectRead64","text":"Read a 64-bit value from an FPGA object.
fpga_result fpgaObjectRead64 (\nfpga_object obj,\nuint64_t * value,\nint flags\n)
The value is assumed to be in string format and will be parsed. See flags below for changing that behavior.
Parameters:
obj An fpga_object instance value Pointer to a 64-bit variable to store the value in flags Flags that control how the object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data. If FPGA_OBJECT_RAW is used, then the data will be read as raw bytes into the uint64_t pointer variable.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters is invalid
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectwrite64","title":"function fpgaObjectWrite64","text":"Write 64-bit value to an FPGA object.
fpga_result fpgaObjectWrite64 (\nfpga_object obj,\nuint64_t value,\nint flags\n)
The value will be converted to string before writing. See flags below for changing that behavior.
Parameters:
obj An fpga_object instance. value The value to write to the object flags Flags that control how the object is written If FPGA_OBJECT_RAW is used, then the value will be written as raw bytes.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters is invalid
Note:
The object must have been created using a handle to a resource.
"},{"location":"opae-code/sysobject_8h/#function-fpgatokengetobject","title":"function fpgaTokenGetObject","text":"Create an fpga_object data structures.
fpga_result fpgaTokenGetObject (\nfpga_token token,\nconst char * name,\nfpga_object * object,\nint flags\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register or a container. This object is read-only.
Parameters:
token Token identifying a resource (accelerator or device) name A key identifying an object belonging to a resource. object Pointer to memory to store the object in flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
Note:
Names that begin with '.' or '/' or contain '..' are not allowed and result in FPGA_INVALID_PARAM being returned
The documentation for this class was generated from the following file docs/sw/include/opae/sysobject.h
"},{"location":"opae-code/sysobject_8h_source/","title":"File sysobject.h","text":"File List > docs > sw > include > opae > sysobject.h
Go to the documentation of this file.
// Copyright(c) 2017-2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_SYSOBJECT_H__\n#define __FPGA_SYSOBJECT_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaTokenGetObject(fpga_token token, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaHandleGetObject(fpga_handle handle, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaObjectGetObject(fpga_object parent, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaObjectGetObjectAt(fpga_object parent, size_t idx,\nfpga_object *object);\nfpga_result fpgaObjectGetType(fpga_object obj, enum fpga_sysobject_type *type);\nfpga_result fpgaDestroyObject(fpga_object *obj);\nfpga_result fpgaObjectGetSize(fpga_object obj, uint32_t *value, int flags);\nfpga_result fpgaObjectRead(fpga_object obj, uint8_t *buffer, size_t offset,\nsize_t len, int flags);\nfpga_result fpgaObjectRead64(fpga_object obj, uint64_t *value, int flags);\nfpga_result fpgaObjectWrite64(fpga_object obj, uint64_t value, int flags);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif /* !__FPGA_SYSOBJECT_H__ */\n
"},{"location":"opae-code/types_8h/","title":"File types.h","text":"FileList > docs > sw > include > opae > types.h
Go to the source code of this file.
Type definitions for FPGA API. More...
#include <stdint.h>#include <stddef.h>#include <stdbool.h>#include <opae/types_enum.h>
"},{"location":"opae-code/types_8h/#classes","title":"Classes","text":"Type Name struct _fpga_token_header Internal token type header. struct fpga_error_info struct fpga_metric Metric struct. struct fpga_metric_info Metric info struct. struct fpga_version Semantic version. struct metric_threshold struct threshold Threshold struct."},{"location":"opae-code/types_8h/#public-types","title":"Public Types","text":"Type Name typedef void * fpga_event_handle Handle to an event object. typedef uint8_t fpga_guid Globally unique identifier (GUID) typedef void * fpga_handle Handle to an FPGA resource. typedef struct fpga_metric fpga_metric Metric struct. typedef struct fpga_metric_info fpga_metric_info Metric info struct. typedef void * fpga_object Object pertaining to an FPGA resource as identified by a unique name. typedef void * fpga_properties Object for expressing FPGA resource properties. typedef void * fpga_token Token for referencing FPGA resources. typedef struct _fpga_token_header fpga_token_header Internal token type header. typedef struct metric_threshold metric_threshold union metric_value Metric value union. typedef struct threshold threshold Threshold struct."},{"location":"opae-code/types_8h/#macros","title":"Macros","text":"Type Name define FPGA_ERROR_NAME_MAX 64Information about an error register. define FPGA_METRIC_STR_SIZE 256FPGA Metric string size. define fpga_is_parent_child (__parent_hdr, __child_hdr) Determine token parent/child relationship."},{"location":"opae-code/types_8h/#detailed-description","title":"Detailed Description","text":"OPAE uses the three opaque types fpga_properties, fpga_token, and fpga_handle to create a hierarchy of objects that can be used to enumerate, reference, acquire, and query FPGA resources. This object model is designed to be extensible to account for different FPGA architectures and platforms.
"},{"location":"opae-code/types_8h/#initialization","title":"Initialization","text":"OPAEs management of the opaque types fpga_properties, fpga_token, and fpga_handle relies on the proper initialization of variables of these types. In other words, before doing anything with a variable of one of these opaque types, you need to first initialize them.
The respective functions that initialize opaque types are:
- fpgaGetProperties() and fpgaCloneProperties() for
fpga_properties - fpgaEnumerate() and fpgaCloneToken() for
fpga_token - fpgaOpen() for
fpga_handle
This should intuitively make sense - fpgaGetProperties() creates fpga_properties objects, fpgaEnumerate() creates fpga_token objects, fpgaOpen() creates fpga_handle objects, and fpgaCloneProperties() and fpgaCloneToken() clone (create) fpga_properties and fpga_token objects, respectively.
Since these opaque types are interpreted as pointers (they are typedef'd to a void *), passing an uninitialized opaque type into any function except the respective initailzation function will result in undefined behaviour, because OPAE will try to follow an invalid pointer. Undefined behaviour in this case may include an unexpected error code, or an application crash.
"},{"location":"opae-code/types_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/types_8h/#typedef-fpga_event_handle","title":"typedef fpga_event_handle","text":"Handle to an event object.
typedef void* fpga_event_handle;\n
OPAE provides an interface to asynchronous events that can be generated by different FPGA resources. The event API provides functions to register for these events; associated with every event a process has registered for is an fpga_event_handle, which encapsulates the OS-specific data structure for event objects.
After use, fpga_event_handle objects should be destroyed using fpgaDestroyEventHandle() to free backing memory used by the fpga_event_handle object.
"},{"location":"opae-code/types_8h/#typedef-fpga_guid","title":"typedef fpga_guid","text":"Globally unique identifier (GUID)
typedef uint8_t fpga_guid[16];\n
GUIDs are used widely within OPAE for helping identify FPGA resources. For example, every FPGA resource has a guid property, which can be (and in the case of FPGA_ACCELERATOR resource primarily is) used for enumerating a resource of a specific type.
fpga_guid is compatible with libuuid's uuid_t, so users can use libuuid functions like uuid_parse() to create and work with GUIDs.
"},{"location":"opae-code/types_8h/#typedef-fpga_handle","title":"typedef fpga_handle","text":"Handle to an FPGA resource.
typedef void* fpga_handle;\n
A valid fpga_handle object, as populated by fpgaOpen(), denotes ownership of an FPGA resource. Note that ownership can be exclusive or shared, depending on the flags used in fpgaOpen(). Ownership can be released by calling fpgaClose(), which will render the underlying handle invalid.
Many OPAE C API functions require a valid token (which is synonymous with ownership of the resource).
"},{"location":"opae-code/types_8h/#typedef-fpga_metric","title":"typedef fpga_metric","text":"typedef struct fpga_metric fpga_metric;\n
"},{"location":"opae-code/types_8h/#typedef-fpga_metric_info","title":"typedef fpga_metric_info","text":"typedef struct fpga_metric_info fpga_metric_info;\n
"},{"location":"opae-code/types_8h/#typedef-fpga_object","title":"typedef fpga_object","text":"Object pertaining to an FPGA resource as identified by a unique name.
typedef void* fpga_object;\n
An fpga_object represents either a device attribute or a container of attributes. Similar to filesystems, a '/' may be used to seperate objects in an object hierarchy. Once on object is acquired, it may be used to read or write data in a resource attribute or to query sub-objects if the object is a container object. The data in an object is buffered and will be kept around until the object is destroyed. Additionally, the data in an attribute can by synchronized from the owning resource using the FPGA_OBJECT_SYNC flag during read operations. The name identifying the object is unique with respect to the resource that owns it. A parent resource may be identified by an fpga_token object, by an fpga_handle object, or another fpga_object object. If a handle object is used when opening the object, then the object is opened with read-write access. Otherwise, the object is read-only.
"},{"location":"opae-code/types_8h/#typedef-fpga_properties","title":"typedef fpga_properties","text":"Object for expressing FPGA resource properties.
typedef void* fpga_properties;\n
fpga_properties objects encapsulate all enumerable information about an FPGA resources. They can be used for two purposes: selective enumeration (discovery) and querying information about existing resources.
For selective enumeration, usually an empty fpga_properties object is created (using fpgaGetProperties()) and then populated with the desired criteria for enumeration. An array of fpga_properties can then be passed to fpgaEnumerate(), which will return a list of fpga_token objects matching these criteria.
For querying properties of existing FPGA resources, fpgaGetProperties() can also take an fpga_token and will return an fpga_properties object populated with information about the resource referenced by that token.
After use, fpga_properties objects should be destroyed using fpga_destroyProperties() to free backing memory used by the fpga_properties object.
"},{"location":"opae-code/types_8h/#typedef-fpga_token","title":"typedef fpga_token","text":"Token for referencing FPGA resources.
typedef void* fpga_token;\n
An fpga_token serves as a reference to a specific FPGA resource present in the system. Holding an fpga_token does not constitute ownership of the FPGA resource - it merely allows the user to query further information about a resource, or to use fpgaOpen() to acquire ownership.
fpga_tokens are usually returned by fpgaEnumerate() or fpgaPropertiesGetParent(), and used by fpgaOpen() to acquire ownership and yield a handle to the resource. Some API calls also take fpga_tokens as arguments if they don't require ownership of the resource in question.
"},{"location":"opae-code/types_8h/#typedef-fpga_token_header","title":"typedef fpga_token_header","text":"Internal token type header.
typedef struct _fpga_token_header fpga_token_header;\n
Each plugin (dfl: libxfpga.so, vfio: libopae-v.so) implements its own proprietary token type. This header must appear at offset zero within that structure.
eg, see lib/plugins/xfpga/types_int.h:struct _fpga_token and lib/plugins/vfio/opae_vfio.h:struct _vfio_token.
"},{"location":"opae-code/types_8h/#typedef-metric_threshold","title":"typedef metric_threshold","text":"typedef struct metric_threshold metric_threshold;\n
"},{"location":"opae-code/types_8h/#union-metric_value","title":"union metric_value","text":""},{"location":"opae-code/types_8h/#typedef-threshold","title":"typedef threshold","text":"typedef struct threshold threshold;\n
"},{"location":"opae-code/types_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/types_8h/#define-fpga_error_name_max","title":"define FPGA_ERROR_NAME_MAX","text":"Information about an error register.
#define FPGA_ERROR_NAME_MAX 64\n
This data structure captures information about an error register exposed by an accelerator resource. The error API provides functions to retrieve these information structures from a particular resource.
"},{"location":"opae-code/types_8h/#define-fpga_metric_str_size","title":"define FPGA_METRIC_STR_SIZE","text":"#define FPGA_METRIC_STR_SIZE 256\n
"},{"location":"opae-code/types_8h/#define-fpga_is_parent_child","title":"define fpga_is_parent_child","text":"Determine token parent/child relationship.
#define fpga_is_parent_child (\n__parent_hdr,\n__child_hdr\n) (((__parent_hdr)->objtype == FPGA_DEVICE ) && \\\n ((__child_hdr)->objtype == FPGA_ACCELERATOR ) && \\\n ((__parent_hdr)->segment == (__child_hdr)->segment) && \\\n ((__parent_hdr)->bus == (__child_hdr)->bus) && \\\n ((__parent_hdr)->device == (__child_hdr)->device))\n
Given pointers to two fpga_token_header structs, determine whether the first is the parent of the second. A parent will have objtype == FPGA_DEVICE. A child will have objtype == FPGA_ACCELERATOR. The PCIe address of the two headers will match in all but the function fields.
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/types_8h_source/","title":"File types.h","text":"File List > docs > sw > include > opae > types.h
Go to the documentation of this file.
// Copyright(c) 2018-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_TYPES_H__\n#define __FPGA_TYPES_H__\n#include <stdint.h>\n#include <stddef.h>\n#include <stdbool.h>\n#include <opae/types_enum.h>\ntypedef void *fpga_properties;\ntypedef void *fpga_token;\ntypedef void *fpga_handle;\ntypedef uint8_t fpga_guid[16];\ntypedef struct {\nuint8_t major; uint8_t minor; uint16_t patch; } fpga_version;\ntypedef void *fpga_event_handle;\n#define FPGA_ERROR_NAME_MAX 64\nstruct fpga_error_info {\nchar name[FPGA_ERROR_NAME_MAX]; bool can_clear; };\ntypedef void *fpga_object;\n#define FPGA_METRIC_STR_SIZE 256\ntypedef union {\nuint64_t ivalue; // Metric integer value\ndouble dvalue; // Metric double value\nfloat fvalue; // Metric float value\nbool bvalue; // Metric bool value\n} metric_value;\ntypedef struct fpga_metric_info {\nuint64_t metric_num; // Metric index num\nfpga_guid metric_guid; // Metric guid\nchar qualifier_name[FPGA_METRIC_STR_SIZE]; // Metric full name\nchar group_name[FPGA_METRIC_STR_SIZE]; // Metric group name\nchar metric_name[FPGA_METRIC_STR_SIZE]; // Metric name\nchar metric_units[FPGA_METRIC_STR_SIZE]; // Metric units\nenum fpga_metric_datatype metric_datatype; // Metric data type\nenum fpga_metric_type metric_type; // Metric group type\n} fpga_metric_info;\ntypedef struct fpga_metric {\nuint64_t metric_num; // Metric index num\nmetric_value value; // Metric value\nbool isvalid; // Metric value is valid\n} fpga_metric;\ntypedef struct threshold {\nchar threshold_name[FPGA_METRIC_STR_SIZE]; // Threshold name\nuint32_t is_valid; // Threshold is valid\ndouble value; // Threshold value\n} threshold;\ntypedef struct metric_threshold {\nchar metric_name[FPGA_METRIC_STR_SIZE]; // Metric Threshold name\nthreshold upper_nr_threshold; // Upper Non-Recoverable Threshold\nthreshold upper_c_threshold; // Upper Critical Threshold\nthreshold upper_nc_threshold; // Upper Non-Critical Threshold\nthreshold lower_nr_threshold; // Lower Non-Recoverable Threshold\nthreshold lower_c_threshold; // Lower Critical Threshold\nthreshold lower_nc_threshold; // Lower Non-Critical Threshold\nthreshold hysteresis; // Hysteresis\n} metric_threshold;\ntypedef struct _fpga_token_header {\nuint64_t magic;\nuint16_t vendor_id;\nuint16_t device_id;\nuint16_t segment;\nuint8_t bus;\nuint8_t device;\nuint8_t function;\nfpga_interface interface;\nfpga_objtype objtype;\nuint64_t object_id;\nfpga_guid guid;\nuint16_t subsystem_vendor_id;\nuint16_t subsystem_device_id;\n} fpga_token_header;\n#define fpga_is_parent_child(__parent_hdr, __child_hdr) \\\n(((__parent_hdr)->objtype == FPGA_DEVICE) && \\\n ((__child_hdr)->objtype == FPGA_ACCELERATOR) && \\\n ((__parent_hdr)->segment == (__child_hdr)->segment) && \\\n ((__parent_hdr)->bus == (__child_hdr)->bus) && \\\n ((__parent_hdr)->device == (__child_hdr)->device))\n#endif // __FPGA_TYPES_H__\n
"},{"location":"opae-code/types__enum_8h/","title":"File types_enum.h","text":"FileList > docs > sw > include > opae > types_enum.h
Go to the source code of this file.
Definitions of enumerated types for the OPAE C API. More...
"},{"location":"opae-code/types__enum_8h/#public-types","title":"Public Types","text":"Type Name enum fpga_accelerator_state accelerator state enum fpga_buffer_flags Buffer flags. enum fpga_event_type FPGA events. enum fpga_interface OPAE plugin interface. enum fpga_metric_datatype Metrics data type. enum fpga_metric_type fpga metrics types opae defines power,thermal, performance counter and afu metric types enum fpga_objtype OPAE FPGA resources (objects) enum fpga_open_flags Open flags. enum fpga_reconf_flags Reconfiguration flags. enum fpga_result OPAE C API function return codes. enum fpga_sysobject_flags enum fpga_sysobject_type"},{"location":"opae-code/types__enum_8h/#detailed-description","title":"Detailed Description","text":"This file defines return and error codes, event and object types, states, and flags as used or reported by OPAE C API functions.
"},{"location":"opae-code/types__enum_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/types__enum_8h/#enum-fpga_accelerator_state","title":"enum fpga_accelerator_state","text":"enum fpga_accelerator_state {\nFPGA_ACCELERATOR_ASSIGNED = 0,\nFPGA_ACCELERATOR_UNASSIGNED\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_buffer_flags","title":"enum fpga_buffer_flags","text":"Buffer flags.
enum fpga_buffer_flags {\nFPGA_BUF_PREALLOCATED = (1u << 0),\nFPGA_BUF_QUIET = (1u << 1),\nFPGA_BUF_READ_ONLY = (1u << 2)\n};\n
These flags can be passed to the fpgaPrepareBuffer() function.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_event_type","title":"enum fpga_event_type","text":"FPGA events.
enum fpga_event_type {\nFPGA_EVENT_INTERRUPT = 0,\nFPGA_EVENT_ERROR,\nFPGA_EVENT_POWER_THERMAL\n};\n
OPAE currently defines the following event types that applications can register for. Note that not all FPGA resources and target platforms may support all event types.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_interface","title":"enum fpga_interface","text":"OPAE plugin interface.
enum fpga_interface {\nFPGA_IFC_DFL = 0,\nFPGA_IFC_VFIO,\nFPGA_IFC_SIM_DFL,\nFPGA_IFC_SIM_VFIO,\nFPGA_IFC_UIO\n};\n
These are the supported plugin interfaces.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_metric_datatype","title":"enum fpga_metric_datatype","text":"enum fpga_metric_datatype {\nFPGA_METRIC_DATATYPE_INT,\nFPGA_METRIC_DATATYPE_FLOAT,\nFPGA_METRIC_DATATYPE_DOUBLE,\nFPGA_METRIC_DATATYPE_BOOL,\nFPGA_METRIC_DATATYPE_UNKNOWN\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_metric_type","title":"enum fpga_metric_type","text":"enum fpga_metric_type {\nFPGA_METRIC_TYPE_POWER,\nFPGA_METRIC_TYPE_THERMAL,\nFPGA_METRIC_TYPE_PERFORMANCE_CTR,\nFPGA_METRIC_TYPE_AFU,\nFPGA_METRIC_TYPE_UNKNOWN\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_objtype","title":"enum fpga_objtype","text":"OPAE FPGA resources (objects)
enum fpga_objtype {\nFPGA_DEVICE = 0,\nFPGA_ACCELERATOR\n};\n
These are the FPGA resources currently supported by the OPAE object model.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_open_flags","title":"enum fpga_open_flags","text":"Open flags.
enum fpga_open_flags {\nFPGA_OPEN_SHARED = (1u << 0)\n};\n
These flags can be passed to the fpgaOpen() function.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_reconf_flags","title":"enum fpga_reconf_flags","text":"Reconfiguration flags.
enum fpga_reconf_flags {\nFPGA_RECONF_FORCE = (1u << 0),\nFPGA_RECONF_SKIP_USRCLK = (1u << 1)\n};\n
These flags can be passed to the fpgaReconfigureSlot() function.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_result","title":"enum fpga_result","text":"OPAE C API function return codes.
enum fpga_result {\nFPGA_OK = 0,\nFPGA_INVALID_PARAM,\nFPGA_BUSY,\nFPGA_EXCEPTION,\nFPGA_NOT_FOUND,\nFPGA_NO_MEMORY,\nFPGA_NOT_SUPPORTED,\nFPGA_NO_DRIVER,\nFPGA_NO_DAEMON,\nFPGA_NO_ACCESS,\nFPGA_RECONF_ERROR\n};\n
Every public API function exported by the OPAE C library will return one of these codes. Usually, FPGA_OK denotes successful completion of the requested operation, while any return code other than FPGA_OK indicates an error or other deviation from the expected behavior. Users of the OPAE C API should always check the return codes of the APIs they call, and not use output parameters of functions that did not execute successfully.
The fpgaErrStr() function converts error codes into printable messages.
OPAE also has a logging mechanism that allows a developer to get more information about why a particular call failed with a specific message. If enabled, any function that returns an error code different from FPGA_OK will also print out a message with further details. This mechanism can be enabled by setting the environment variable LIBOPAE_LOG to 1 before running the respective application.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_sysobject_flags","title":"enum fpga_sysobject_flags","text":"enum fpga_sysobject_flags {\nFPGA_OBJECT_SYNC = (1u << 0),\nFPGA_OBJECT_GLOB = (1u << 1),\nFPGA_OBJECT_RAW =\n(1u << 2),\nFPGA_OBJECT_RECURSE_ONE =\n(1u\n<< 3),\nFPGA_OBJECT_RECURSE_ALL =\n(1u\n<< 4)\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_sysobject_type","title":"enum fpga_sysobject_type","text":"enum fpga_sysobject_type {\nFPGA_OBJECT_CONTAINER =\n(1u << 0),\nFPGA_OBJECT_ATTRIBUTE =\n(1u << 1)\n};\n
The documentation for this class was generated from the following file docs/sw/include/opae/types_enum.h
"},{"location":"opae-code/types__enum_8h_source/","title":"File types_enum.h","text":"File List > docs > sw > include > opae > types_enum.h
Go to the documentation of this file.
// Copyright(c) 2017-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_TYPES_ENUM_H__\n#define __FPGA_TYPES_ENUM_H__\ntypedef enum {\nFPGA_OK = 0, FPGA_INVALID_PARAM, FPGA_BUSY, FPGA_EXCEPTION, FPGA_NOT_FOUND, FPGA_NO_MEMORY, FPGA_NOT_SUPPORTED, FPGA_NO_DRIVER, FPGA_NO_DAEMON, FPGA_NO_ACCESS, FPGA_RECONF_ERROR } fpga_result;\ntypedef enum {\nFPGA_EVENT_INTERRUPT = 0, FPGA_EVENT_ERROR, FPGA_EVENT_POWER_THERMAL } fpga_event_type;\n/* TODO: consider adding lifecycle events in the future\n * to help with orchestration. Need a complete specification\n * before including them in the API. Proposed events:\n * FPGA_EVENT_APPEAR\n * FPGA_EVENT_DISAPPEAR\n * FPGA_EVENT_CHANGE\n */\ntypedef enum {\nFPGA_ACCELERATOR_ASSIGNED = 0,\nFPGA_ACCELERATOR_UNASSIGNED\n} fpga_accelerator_state;\ntypedef enum {\nFPGA_DEVICE = 0,\nFPGA_ACCELERATOR\n} fpga_objtype;\ntypedef enum {\nFPGA_IFC_DFL = 0,\nFPGA_IFC_VFIO,\nFPGA_IFC_SIM_DFL,\nFPGA_IFC_SIM_VFIO,\nFPGA_IFC_UIO,\n} fpga_interface;\nenum fpga_buffer_flags {\nFPGA_BUF_PREALLOCATED = (1u << 0), FPGA_BUF_QUIET = (1u << 1), FPGA_BUF_READ_ONLY = (1u << 2) };\nenum fpga_open_flags {\nFPGA_OPEN_SHARED = (1u << 0)\n};\nenum fpga_reconf_flags {\nFPGA_RECONF_FORCE = (1u << 0),\nFPGA_RECONF_SKIP_USRCLK = (1u << 1)\n};\nenum fpga_sysobject_flags {\nFPGA_OBJECT_SYNC = (1u << 0), FPGA_OBJECT_GLOB = (1u << 1), FPGA_OBJECT_RAW =\n(1u << 2), FPGA_OBJECT_RECURSE_ONE =\n(1u\n<< 3), FPGA_OBJECT_RECURSE_ALL =\n(1u\n<< 4) };\nenum fpga_sysobject_type {\nFPGA_OBJECT_CONTAINER =\n(1u << 0), FPGA_OBJECT_ATTRIBUTE =\n(1u << 1) };\nenum fpga_metric_type {\nFPGA_METRIC_TYPE_POWER, // Metric power\nFPGA_METRIC_TYPE_THERMAL, // Metric Thermal\nFPGA_METRIC_TYPE_PERFORMANCE_CTR, // Metric Performance counter\nFPGA_METRIC_TYPE_AFU, // Metric AFU\nFPGA_METRIC_TYPE_UNKNOWN // Unknown\n};\nenum fpga_metric_datatype {\nFPGA_METRIC_DATATYPE_INT, // Metric datatype integer\nFPGA_METRIC_DATATYPE_FLOAT, // Metric datatype float\nFPGA_METRIC_DATATYPE_DOUBLE, // Metric datatype double\nFPGA_METRIC_DATATYPE_BOOL, // Metric datatype bool\nFPGA_METRIC_DATATYPE_UNKNOWN // Metric datatype unknown\n};\n#endif // __FPGA_TYPES_ENUM_H__\n
"},{"location":"opae-code/uio_8h/","title":"File uio.h","text":"FileList > docs > sw > include > opae > uio.h
Go to the source code of this file.
APIs to manage a PCIe device via UIO. More...
#include <stdio.h>#include <stdint.h>
"},{"location":"opae-code/uio_8h/#classes","title":"Classes","text":"Type Name struct opae_uio OPAE UIO device abstraction. struct opae_uio_device_region MMIO region info."},{"location":"opae-code/uio_8h/#public-functions","title":"Public Functions","text":"Type Name void opae_uio_close (struct opae_uio * u) Release and close a UIO device. int opae_uio_open (struct opae_uio * u, const char * dfl_device) Open and populate a UIO device. int opae_uio_region_get (struct opae_uio * u, uint32_t index, uint8_t ** ptr, size_t * size) Query device MMIO region."},{"location":"opae-code/uio_8h/#macros","title":"Macros","text":"Type Name define OPAE_UIO_PATH_MAX 256"},{"location":"opae-code/uio_8h/#detailed-description","title":"Detailed Description","text":"Presents a simple interface for interacting with a DFL device that is bound to its UIO driver. See https://kernel.org/doc/html/v4.14/driver-api/uio-howto.html for a description of UIO.
Provides APIs for opening/closing the device and for querying info about the MMIO regions of the device.
"},{"location":"opae-code/uio_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/uio_8h/#function-opae_uio_close","title":"function opae_uio_close","text":"Release and close a UIO device.
void opae_uio_close (\nstruct opae_uio * u\n)
The given device pointer must have been previously initialized by opae_uio_open. Releases all data structures.
Parameters:
u Storage for the device info. May be stack-resident.
Example struct opae_uio u;
if (opae_uio_open(&u, \"dfl_dev.10\")) { // handle error } else { // interact with the device ... // free the device opae_uio_close(&u); }
Example $ sudo opaeuio -r dfl_dev.10
"},{"location":"opae-code/uio_8h/#function-opae_uio_open","title":"function opae_uio_open","text":"Open and populate a UIO device.
int opae_uio_open (\nstruct opae_uio * u,\nconst char * dfl_device\n)
Opens the Device Feature List device corresponding to the device name given in dfl_device, eg \"dfl_dev.10\". The device must be bound to the dfl-uio-pdev driver prior to opening it. The data structures corresponding to the MMIO regions are initialized.
Parameters:
u Storage for the device. May be stack-resident. dfl_device The name of the desired DFL device.
Returns:
Non-zero on error. Zero on success.
Example $ sudo opaeuio -i -u lab -g lab dfl_dev.10
Example struct opae_uio u;
if (opae_uio_open(&u, \"dfl_dev.10\")) { // handle error }
"},{"location":"opae-code/uio_8h/#function-opae_uio_region_get","title":"function opae_uio_region_get","text":"Query device MMIO region.
int opae_uio_region_get (\nstruct opae_uio * u,\nuint32_t index,\nuint8_t ** ptr,\nsize_t * size\n)
Retrieves info describing the MMIO region with the given region index. The device structure u must have been previously opened by a call to opae_uio_open.
Parameters:
u The open OPAE UIO device. index The zero-based index of the desired MMIO region. ptr Optional pointer to receive the virtual address for the region. Pass NULL to ignore. size Optional pointer to receive the size of the MMIO region. Pass NULL to ignore.
Returns:
Non-zero on error (including index out-of-range). Zero on success.
Example struct opae_uio u;
uint8_t *virt = NULL; size_t size = 0;
if (opae_uio_open(&u, \"dfl_dev.10\")) { // handle error } else { opae_uio_region_get(&u, 0, &virt, &size); }
"},{"location":"opae-code/uio_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/uio_8h/#define-opae_uio_path_max","title":"define OPAE_UIO_PATH_MAX","text":"#define OPAE_UIO_PATH_MAX 256\n
The documentation for this class was generated from the following file docs/sw/include/opae/uio.h
"},{"location":"opae-code/uio_8h_source/","title":"File uio.h","text":"File List > docs > sw > include > opae > uio.h
Go to the documentation of this file.
// Copyright(c) 2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_UIO_H__\n#define __OPAE_UIO_H__\n#include <stdio.h>\n#include <stdint.h>\n#define OPAE_UIO_PATH_MAX 256\nstruct opae_uio_device_region {\nuint32_t region_index;\nuint8_t *region_ptr;\nsize_t region_page_offset;\nsize_t region_size;\nstruct opae_uio_device_region *next;\n};\nstruct opae_uio {\nchar device_path[OPAE_UIO_PATH_MAX];\nint device_fd;\nstruct opae_uio_device_region *regions;\n};\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nint opae_uio_open(struct opae_uio *u,\nconst char *dfl_device);\nint opae_uio_region_get(struct opae_uio *u,\nuint32_t index,\nuint8_t **ptr,\nsize_t *size);\nvoid opae_uio_close(struct opae_uio *u);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __OPAE_UIO_H__\n
"},{"location":"opae-code/umsg_8h/","title":"File umsg.h","text":"FileList > docs > sw > include > opae > umsg.h
Go to the source code of this file.
FPGA UMsg API.
#include <opae/types.h>
"},{"location":"opae-code/umsg_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetNumUmsg (fpga_handle handle, uint64_t * value) Get number of Umsgs. fpga_result fpgaGetUmsgPtr (fpga_handle handle, uint64_t ** umsg_ptr) Access UMsg memory directly. fpga_result fpgaSetUmsgAttributes (fpga_handle handle, uint64_t value) Sets Umsg hint. fpga_result fpgaTriggerUmsg (fpga_handle handle, uint64_t value) Trigger Umsg."},{"location":"opae-code/umsg_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/umsg_8h/#function-fpgagetnumumsg","title":"function fpgaGetNumUmsg","text":"Get number of Umsgs.
fpga_result fpgaGetNumUmsg (\nfpga_handle handle,\nuint64_t * value\n)
Retuns number of umsg supported by AFU.
Parameters:
handle Handle to previously opened accelerator resource value Returns number of UMsgs
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid.
"},{"location":"opae-code/umsg_8h/#function-fpgagetumsgptr","title":"function fpgaGetUmsgPtr","text":"Access UMsg memory directly.
fpga_result fpgaGetUmsgPtr (\nfpga_handle handle,\nuint64_t ** umsg_ptr\n)
This function will return a pointer to the memory allocated for low latency accelerator notifications (UMsgs).
Parameters:
handle Handle to previously opened accelerator resource umsg_ptr Pointer to memory where a pointer to the virtual address space will be returned
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid. FPGA_NO_MEMORY if memory allocation fails or system doesn't configure huge pages.
"},{"location":"opae-code/umsg_8h/#function-fpgasetumsgattributes","title":"function fpgaSetUmsgAttributes","text":"Sets Umsg hint.
fpga_result fpgaSetUmsgAttributes (\nfpga_handle handle,\nuint64_t value\n)
Writes usmg hint bit.
Parameters:
handle Handle to previously opened accelerator resource value Value to use for UMsg hint, Umsg hit is N wide bitvector where N = number of Umsgs.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid.
"},{"location":"opae-code/umsg_8h/#function-fpgatriggerumsg","title":"function fpgaTriggerUmsg","text":"Trigger Umsg.
fpga_result fpgaTriggerUmsg (\nfpga_handle handle,\nuint64_t value\n)
Writes a 64-bit value to trigger low-latency accelerator notification mechanism (UMsgs).
Parameters:
handle Handle to previously opened accelerator resource value Value to use for UMsg
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid.
The documentation for this class was generated from the following file docs/sw/include/opae/umsg.h
"},{"location":"opae-code/umsg_8h_source/","title":"File umsg.h","text":"File List > docs > sw > include > opae > umsg.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_UMSG_H__\n#define __FPGA_UMSG_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetNumUmsg(fpga_handle handle, uint64_t *value);\nfpga_result fpgaSetUmsgAttributes(fpga_handle handle,\nuint64_t value);\nfpga_result fpgaTriggerUmsg(fpga_handle handle, uint64_t value);\nfpga_result fpgaGetUmsgPtr(fpga_handle handle, uint64_t **umsg_ptr);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_UMSG_H__\n
"},{"location":"opae-code/userclk_8h/","title":"File userclk.h","text":"FileList > docs > sw > include > opae > userclk.h
Go to the source code of this file.
Functions for setting and get afu user clock.
#include <opae/types.h>
"},{"location":"opae-code/userclk_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetUserClock (fpga_handle handle, uint64_t * high_clk, uint64_t * low_clk, int flags) Get afu user clock high and low. fpga_result fpgaSetUserClock (fpga_handle handle, uint64_t high_clk, uint64_t low_clk, int flags) set afu user clock high and low"},{"location":"opae-code/userclk_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/userclk_8h/#function-fpgagetuserclock","title":"function fpgaGetUserClock","text":"Get afu user clock high and low.
fpga_result fpgaGetUserClock (\nfpga_handle handle,\nuint64_t * high_clk,\nuint64_t * low_clk,\nint flags\n)
Parameters:
handle Handle to previously opened accelerator resource. high_clk AFU High user clock frequency in MHz. low_clk AFU Low user clock frequency in MHz. flags Flags Reserved.
.*
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/userclk_8h/#function-fpgasetuserclock","title":"function fpgaSetUserClock","text":"set afu user clock high and low
fpga_result fpgaSetUserClock (\nfpga_handle handle,\nuint64_t high_clk,\nuint64_t low_clk,\nint flags\n)
Parameters:
handle Handle to previously opened accelerator resource. high_clk AFU High user clock frequency in MHz. low_clk AFU Low user clock frequency in MHz. flags Flags Reserved.
.*
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
The documentation for this class was generated from the following file docs/sw/include/opae/userclk.h
"},{"location":"opae-code/userclk_8h_source/","title":"File userclk.h","text":"File List > docs > sw > include > opae > userclk.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_AFU_USER_CLOCK_H__\n#define __FPGA_AFU_USER_CLOCK_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaSetUserClock(fpga_handle handle,\nuint64_t high_clk, uint64_t low_clk, int flags);\nfpga_result fpgaGetUserClock(fpga_handle handle,\nuint64_t *high_clk, uint64_t *low_clk, int flags);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_AFU_USER_CLOCK_H__\n
"},{"location":"opae-code/utils_8h/","title":"File utils.h","text":"FileList > docs > sw > include > opae > utils.h
Go to the source code of this file.
Utility functions and macros for the FPGA API.
#include <opae/types.h>#include <stdio.h>
"},{"location":"opae-code/utils_8h/#public-functions","title":"Public Functions","text":"Type Name const char * fpgaErrStr (fpga_result e) Return human-readable error message."},{"location":"opae-code/utils_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/utils_8h/#function-fpgaerrstr","title":"function fpgaErrStr","text":"Return human-readable error message.
const char * fpgaErrStr (\nfpga_result e\n)
Returns a pointer to a human-readable error message corresponding to the provided fpga_error error code.
Parameters:
e Error code (as returned by another FPGA API function
Returns:
Pointer to a descriptive error message string
The documentation for this class was generated from the following file docs/sw/include/opae/utils.h
"},{"location":"opae-code/utils_8h_source/","title":"File utils.h","text":"File List > docs > sw > include > opae > utils.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_UTILS_H__\n#define __FPGA_UTILS_H__\n#include <opae/types.h>\n#include <stdio.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nconst char *fpgaErrStr(fpga_result e);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_UTILS_H__\n
"},{"location":"opae-code/version_8h/","title":"File version.h","text":"FileList > docs > sw > include > opae > version.h
Go to the source code of this file.
#include <opae/types.h>
"},{"location":"opae-code/version_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetOPAECBuildString (char * build_str, size_t len) Get build information about the OPAE library as a string. fpga_result fpgaGetOPAECVersion (fpga_version * version) Get version information about the OPAE library. fpga_result fpgaGetOPAECVersionString (char * version_str, size_t len) Get version information about the OPAE library as a string."},{"location":"opae-code/version_8h/#macros","title":"Macros","text":"Type Name define FPGA_BUILD_STR_MAX 41 define FPGA_VERSION_STR_MAX 10"},{"location":"opae-code/version_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/version_8h/#function-fpgagetopaecbuildstring","title":"function fpgaGetOPAECBuildString","text":"Get build information about the OPAE library as a string.
fpga_result fpgaGetOPAECBuildString (\nchar * build_str,\nsize_t len\n)
Retrieve the build identifier of the OPAE library.
Parameters:
build_str String to copy build information into len Length of build_str
Returns:
FPGA_INVALID_PARAM if build_str is NULL, FPGA_EXCEPTION if the version string cannot be copied into build_str, FPGA_OK otherwise.
"},{"location":"opae-code/version_8h/#function-fpgagetopaecversion","title":"function fpgaGetOPAECVersion","text":"Get version information about the OPAE library.
fpga_result fpgaGetOPAECVersion (\nfpga_version * version\n)
Retrieve major version, minor version, and revision information about the OPAE library.
Parameters:
version FPGA version
Returns:
FPGA_INVALID_PARAM if any of the output parameters is NULL, FPGA_OK otherwise.
"},{"location":"opae-code/version_8h/#function-fpgagetopaecversionstring","title":"function fpgaGetOPAECVersionString","text":"Get version information about the OPAE library as a string.
fpga_result fpgaGetOPAECVersionString (\nchar * version_str,\nsize_t len\n)
Retrieve major version, minor version, and revision information about the OPAE library, encoded in a human-readable string (e.g. \"1.0.0\").
Parameters:
version_str String to copy version information into len Length of version_str
Returns:
FPGA_INVALID_PARAM if version_str is NULL, FPGA_EXCEPTION if the version string cannot be copied into version_str, FPGA_OK otherwise.
"},{"location":"opae-code/version_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/version_8h/#define-fpga_build_str_max","title":"define FPGA_BUILD_STR_MAX","text":"#define FPGA_BUILD_STR_MAX 41\n
"},{"location":"opae-code/version_8h/#define-fpga_version_str_max","title":"define FPGA_VERSION_STR_MAX","text":"#define FPGA_VERSION_STR_MAX 10\n
The documentation for this class was generated from the following file docs/sw/include/opae/version.h
"},{"location":"opae-code/version_8h_source/","title":"File version.h","text":"File List > docs > sw > include > opae > version.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_VERSION_H__\n#define __FPGA_VERSION_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetOPAECVersion(fpga_version *version);\nfpga_result fpgaGetOPAECVersionString(char *version_str, size_t len);\n#define FPGA_VERSION_STR_MAX 10\nfpga_result fpgaGetOPAECBuildString(char *build_str, size_t len);\n#define FPGA_BUILD_STR_MAX 41\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_VERSION_H__\n
"},{"location":"opae-code/vfio_8h/","title":"File vfio.h","text":"FileList > docs > sw > include > opae > vfio.h
Go to the source code of this file.
APIs to manage a PCIe device via vfio-pci. More...
#include <stdio.h>#include <stdint.h>#include <pthread.h>#include <linux/vfio.h>#include <opae/mem_alloc.h>#include <opae/hash_map.h>
"},{"location":"opae-code/vfio_8h/#classes","title":"Classes","text":"Type Name struct opae_vfio OPAE VFIO device abstraction. struct opae_vfio_buffer System DMA buffer. struct opae_vfio_device VFIO device. struct opae_vfio_device_irq Interrupt info. struct opae_vfio_device_region MMIO region info. struct opae_vfio_group VFIO group. struct opae_vfio_iova_range IO Virtual Address Range. struct opae_vfio_sparse_info MMIO sparse-mappable region info."},{"location":"opae-code/vfio_8h/#public-types","title":"Public Types","text":"Type Name enum opae_vfio_buffer_flags Flags for opae_vfio_buffer_allocate_ex() ."},{"location":"opae-code/vfio_8h/#public-functions","title":"Public Functions","text":"Type Name int opae_vfio_buffer_allocate (struct opae_vfio * v, size_t * size, uint8_t ** buf, uint64_t * iova) Allocate and map system buffer. int opae_vfio_buffer_allocate_ex (struct opae_vfio * v, size_t * size, uint8_t ** buf, uint64_t * iova, int flags) Allocate and map system buffer (extended w/ flags) int opae_vfio_buffer_free (struct opae_vfio * v, uint8_t * buf) Unmap and free a system buffer. struct opae_vfio_buffer * opae_vfio_buffer_info (struct opae_vfio * v, uint8_t * vaddr) Extract the internal data structure pointer for the given vaddr. void opae_vfio_close (struct opae_vfio * v) Release and close a VFIO device. int opae_vfio_irq_disable (struct opae_vfio * v, uint32_t index, uint32_t subindex) Disable an IRQ. int opae_vfio_irq_enable (struct opae_vfio * v, uint32_t index, uint32_t subindex, int event_fd) Enable an IRQ. int opae_vfio_irq_mask (struct opae_vfio * v, uint32_t index, uint32_t subindex) Mask an IRQ. int opae_vfio_irq_unmask (struct opae_vfio * v, uint32_t index, uint32_t subindex) Unmask an IRQ. int opae_vfio_open (struct opae_vfio * v, const char * pciaddr) Open and populate a VFIO device. int opae_vfio_region_get (struct opae_vfio * v, uint32_t index, uint8_t ** ptr, size_t * size) Query device MMIO region. int opae_vfio_secure_open (struct opae_vfio * v, const char * pciaddr, const char * token) Open and populate a VFIO device."},{"location":"opae-code/vfio_8h/#detailed-description","title":"Detailed Description","text":"Presents a simple interface for interacting with a PCIe device that is bound to the vfio-pci driver. See https://kernel.org/doc/Documentation/vfio.txt for a description of vfio-pci.
Provides APIs for opening/closing the device, querying info about the MMIO regions of the device, and allocating/mapping and freeing/unmapping DMA buffers.
"},{"location":"opae-code/vfio_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/vfio_8h/#enum-opae_vfio_buffer_flags","title":"enum opae_vfio_buffer_flags","text":"enum opae_vfio_buffer_flags {\nOPAE_VFIO_BUF_PREALLOCATED = 1\n};\n
"},{"location":"opae-code/vfio_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_allocate","title":"function opae_vfio_buffer_allocate","text":"Allocate and map system buffer.
int opae_vfio_buffer_allocate (\nstruct opae_vfio * v,\nsize_t * size,\nuint8_t ** buf,\nuint64_t * iova\n)
Allocate, map, and retrieve info for a system buffer capable of DMA. Saves an entry in the v->cont_buffers list. If the buffer is not explicitly freed by opae_vfio_buffer_free, it will be freed during opae_vfio_close.
mmap is used for the allocation. If the size is greater than 2MB, then the allocation request is fulfilled by a 1GB huge page. Else, if the size is greater than 4096, then the request is fulfilled by a 2MB huge page. Else, the request is fulfilled by the non-huge page pool.
Note:
Allocations from the huge page pool require that huge pages be configured on the system. Huge pages may be configured on the kernel boot command prompt. Example default_hugepagesz=1G hugepagesz=1G hugepages=2 hugepagesz=2M hugepages=8
Note:
Huge pages may also be configured at runtime. Example sudo sh -c 'echo 8 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages' sudo sh -c 'echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'
Note:
Be sure that the IOMMU is also enabled using the follow kernel boot command: intel_iommu=on
Parameters:
v The open OPAE VFIO device. size A pointer to the requested size. The size may be rounded to the next page size prior to return from the function. buf Optional pointer to receive the virtual address for the buffer. Pass NULL to ignore. iova Optional pointer to receive the IOVA address for the buffer. Pass NULL to ignore.
Returns:
Non-zero on error. Zero on success.
Example opae_vfio v;
size_t sz; uint8_t *buf_2m_virt = NULL; uint8_t *buf_1g_virt = NULL; uint64_t buf_2m_iova = 0; uint64_t buf_1g_iova = 0;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { sz = 2 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_2m_virt, &buf_2m_iova)) { // handle allocation error }
sz = 1024 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_1g_virt, &buf_1g_iova)) { // handle allocation error } }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_allocate_ex","title":"function opae_vfio_buffer_allocate_ex","text":"Allocate and map system buffer (extended w/ flags)
int opae_vfio_buffer_allocate_ex (\nstruct opae_vfio * v,\nsize_t * size,\nuint8_t ** buf,\nuint64_t * iova,\nint flags\n)
Allocate, map, and retrieve info for a system buffer capable of DMA. Saves an entry in the v->cont_buffers list. If the buffer is not explicitly freed by opae_vfio_buffer_free, it will be freed during opae_vfio_close, unless OPAE_VFIO_BUF_PREALLOCATED is used in which case the buffer is not freed by this library.
When not using OPAE_VFIO_BUF_PREALLOCATED, mmap is used for the allocation. If the size is greater than 2MB, then the allocation request is fulfilled by a 1GB huge page. Else, if the size is greater than 4096, then the request is fulfilled by a 2MB huge page. Else, the request is fulfilled by the non-huge page pool.
Parameters:
v The open OPAE VFIO device. size A pointer to the requested size. The size may be rounded to the next page size prior to return from the function. buf Optional pointer to receive the virtual address for the buffer/input buffer pointer when using OPAE_VFIO_BUF_PREALLOCATED. Pass NULL to ignore. iova Optional pointer to receive the IOVA address for the buffer. Pass NULL to ignore.
Returns:
Non-zero on error. Zero on success.
Example opae_vfio v;
size_t sz = MY_BUF_SIZE; uint8_t *prealloc_virt = NULL; uint64_t iova = 0;
prealloc_virt = allocate_my_buffer(sz);
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { if (opae_vfio_buffer_allocate_ex(&v, &sz, &prealloc_virt, &iova, OPAE_VFIO_BUF_PREALLOCATED)) { // handle allocation error } }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_free","title":"function opae_vfio_buffer_free","text":"Unmap and free a system buffer.
int opae_vfio_buffer_free (\nstruct opae_vfio * v,\nuint8_t * buf\n)
The buffer corresponding to buf must have been created by a previous call to opae_vfio_buffer_allocate.
Parameters:
v The open OPAE VFIO device. buf The virtual address corresponding to the buffer to be freed.
Returns:
Non-zero on error. Zero on success.
Example size_t sz; uint8_t *buf_2m_virt = NULL; uint64_t buf_2m_iova = 0;
sz = 2 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_2m_virt, &buf_2m_iova)) { // handle allocation error } else { // use the buffer
if (opae_vfio_buffer_free(&v, buf_2m_virt)) { // handle free error } }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_info","title":"function opae_vfio_buffer_info","text":"Extract the internal data structure pointer for the given vaddr.
struct opae_vfio_buffer * opae_vfio_buffer_info (\nstruct opae_vfio * v,\nuint8_t * vaddr\n)
The virtual address vaddr must correspond to a buffer previously allocated by opae_vfio_buffer_allocate() or opae_vfio_buffer_allocate_ex().
Parameters:
v The open OPAE VFIO device. vaddr The user virtual address of the desired buffer info struct.
Returns:
NULL on lookup error.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_close","title":"function opae_vfio_close","text":"Release and close a VFIO device.
void opae_vfio_close (\nstruct opae_vfio * v\n)
The given device pointer must have been previously initialized by opae_vfio_open. Releases all data structures, including any DMA buffer allocations that have not be explicitly freed by opae_vfio_buffer_free.
Parameters:
v Storage for the device info. May be stack-resident.
Example opae_vfio v;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { // interact with the device ... // free the device opae_vfio_close(&v); }
Example $ sudo opaevfio -r 0000:00:00.0
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_disable","title":"function opae_vfio_irq_disable","text":"Disable an IRQ.
int opae_vfio_irq_disable (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to disable.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_enable","title":"function opae_vfio_irq_enable","text":"Enable an IRQ.
int opae_vfio_irq_enable (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex,\nint event_fd\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to enable. event_fd The file descriptor, created by eventfd(). Interrupts will result in this event_fd being signaled.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_mask","title":"function opae_vfio_irq_mask","text":"Mask an IRQ.
int opae_vfio_irq_mask (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to mask.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_unmask","title":"function opae_vfio_irq_unmask","text":"Unmask an IRQ.
int opae_vfio_irq_unmask (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to unmask.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_open","title":"function opae_vfio_open","text":"Open and populate a VFIO device.
int opae_vfio_open (\nstruct opae_vfio * v,\nconst char * pciaddr\n)
Opens the PCIe device corresponding to the address given in pciaddr. The device must be bound to the vfio-pci driver prior to opening it. The data structures corresponding to IOVA space, MMIO regions, and DMA buffers are initialized.
Parameters:
v Storage for the device info. May be stack-resident. pciaddr The PCIe address of the requested device.
Returns:
Non-zero on error. Zero on success.
Example $ sudo opaevfio -i 0000:00:00.0 -u user -g group
Example opae_vfio v;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_region_get","title":"function opae_vfio_region_get","text":"Query device MMIO region.
int opae_vfio_region_get (\nstruct opae_vfio * v,\nuint32_t index,\nuint8_t ** ptr,\nsize_t * size\n)
Retrieves info describing the MMIO region with the given region index. The device structure v must have been previously opened by a call to opae_vfio_open.
Parameters:
v The open OPAE VFIO device. index The zero-based index of the desired MMIO region. ptr Optional pointer to receive the virtual address for the region. Pass NULL to ignore. size Optional pointer to receive the size of the MMIO region. Pass NULL to ignore.
Returns:
Non-zero on error (including index out-of-range). Zero on success.
Example opae_vfio v;
uint8_t *fme_virt = NULL; uint8_t *port_virt = NULL; size_t fme_size = 0; size_t port_size = 0;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { opae_vfio_region_get(&v, 0, &fme_virt, &fme_size); opae_vfio_region_get(&v, 2, &port_virt, &port_size); }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_secure_open","title":"function opae_vfio_secure_open","text":"Open and populate a VFIO device.
int opae_vfio_secure_open (\nstruct opae_vfio * v,\nconst char * pciaddr,\nconst char * token\n)
Opens the PCIe device corresponding to the address given in pciaddr, using the VF token (GUID) given in token. The device must be bound to the vfio-pci driver prior to opening it. The data structures corresponding to IOVA space, MMIO regions, and DMA buffers are initialized.
Parameters:
v Storage for the device info. May be stack-resident. pciaddr The PCIe address of the requested device. token The GUID representing the VF token.
Returns:
Non-zero on error. Zero on success.
Example $ sudo opaevfio -i 0000:00:00.0 -u user -g group
Example opae_vfio v;
if (opae_vfio_secure_open(&v, \"0000:00:00.0\", \"00f5ad6b-2edd-422e-9d1e-34124c686fec\")) { // handle error }
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/vfio_8h_source/","title":"File vfio.h","text":"File List > docs > sw > include > opae > vfio.h
Go to the documentation of this file.
// Copyright(c) 2020-2023, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_VFIO_H__\n#define __OPAE_VFIO_H__\n#include <stdio.h>\n#include <stdint.h>\n#include <pthread.h>\n#include <linux/vfio.h>\n#include <opae/mem_alloc.h>\n#include <opae/hash_map.h>\nstruct opae_vfio_iova_range {\nuint64_t start; uint64_t end; struct opae_vfio_iova_range *next; };\nstruct opae_vfio_group {\nchar *group_device; int group_fd; };\nstruct opae_vfio_sparse_info {\nuint32_t index; uint32_t offset; uint32_t size; uint8_t *ptr; struct opae_vfio_sparse_info *next; };\nstruct opae_vfio_device_region {\nuint32_t region_index; uint8_t *region_ptr; size_t region_size; struct opae_vfio_sparse_info *region_sparse; struct opae_vfio_device_region *next; };\nstruct opae_vfio_device_irq {\nuint32_t flags; uint32_t index; uint32_t count; int32_t *event_fds; int32_t *masks; struct opae_vfio_device_irq *next; };\nstruct opae_vfio_device {\nint device_fd; uint64_t device_config_offset; uint32_t device_num_regions; struct opae_vfio_device_region *regions; uint32_t device_num_irqs; struct opae_vfio_device_irq *irqs; };\nstruct opae_vfio_buffer {\nuint8_t *buffer_ptr; size_t buffer_size; uint64_t buffer_iova; int flags; };\nstruct opae_vfio {\npthread_mutex_t lock; char *cont_device; char *cont_pciaddr; int cont_fd; struct opae_vfio_iova_range *cont_ranges; struct mem_alloc iova_alloc; struct opae_vfio_group group; struct opae_vfio_device device; opae_hash_map cont_buffers; };\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nint opae_vfio_open(struct opae_vfio *v,\nconst char *pciaddr);\nint opae_vfio_secure_open(struct opae_vfio *v,\nconst char *pciaddr,\nconst char *token);\nint opae_vfio_region_get(struct opae_vfio *v,\nuint32_t index,\nuint8_t **ptr,\nsize_t *size);\nint opae_vfio_buffer_allocate(struct opae_vfio *v,\nsize_t *size,\nuint8_t **buf,\nuint64_t *iova);\nenum opae_vfio_buffer_flags {\nOPAE_VFIO_BUF_PREALLOCATED = 1, };\nint opae_vfio_buffer_allocate_ex(struct opae_vfio *v,\nsize_t *size,\nuint8_t **buf,\nuint64_t *iova,\nint flags);\nstruct opae_vfio_buffer *opae_vfio_buffer_info(struct opae_vfio *v,\nuint8_t *vaddr);\nint opae_vfio_buffer_free(struct opae_vfio *v,\nuint8_t *buf);\nint opae_vfio_irq_enable(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex,\nint event_fd);\nint opae_vfio_irq_unmask(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex);\nint opae_vfio_irq_mask(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex);\nint opae_vfio_irq_disable(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex);\nvoid opae_vfio_close(struct opae_vfio *v);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __OPAE_VFIO_H__\n
"},{"location":"opae-code/dir_9a6968a8846ef48cff617fcd6355d7b4/","title":"Dir docs/sw/samples","text":"FileList > docs > sw > samples
"},{"location":"opae-code/dir_9a6968a8846ef48cff617fcd6355d7b4/#directories","title":"Directories","text":"Type Name dir hello_events dir hello_fpga The documentation for this class was generated from the following file docs/sw/samples/
"},{"location":"opae-code/dir_d66a8e4b979fa79493bebe26e2602d2b/","title":"Dir docs/sw/samples/hello_events","text":"FileList > docs > sw > samples > hello_events
"},{"location":"opae-code/dir_d66a8e4b979fa79493bebe26e2602d2b/#files","title":"Files","text":"Type Name file hello_events.c A code sample of using OPAE event API. The documentation for this class was generated from the following file docs/sw/samples/hello_events/
"},{"location":"opae-code/hello__events_8c/","title":"File hello_events.c","text":"FileList > docs > sw > samples > hello_events > hello_events.c
Go to the source code of this file.
A code sample of using OPAE event API. More...
#include <stdio.h>#include <stdlib.h>#include <string.h>#include <unistd.h>#include <getopt.h>#include <poll.h>#include <errno.h>#include <sys/stat.h>#include <pthread.h>#include <opae/fpga.h>#include <argsfilter.h>#include \"mock/opae_std.h\"
"},{"location":"opae-code/hello__events_8c/#classes","title":"Classes","text":"Type Name struct ras_inject_error"},{"location":"opae-code/hello__events_8c/#public-functions","title":"Public Functions","text":"Type Name void * error_thread (void * arg) fpga_result find_fpga (fpga_properties device_filter, fpga_token * fpga, uint32_t * num_matches) void help (void) fpga_result inject_ras_fatal_error (fpga_token fme_token, uint8_t err) int main (int argc, char * argv) fpga_result parse_args (int argc, char * argv) void print_err (const char * s, fpga_result res) int usleep (unsigned)"},{"location":"opae-code/hello__events_8c/#macros","title":"Macros","text":"Type Name define FME_SYSFS_INJECT_ERROR \"errors/inject_errors\" define GETOPT_STRING \"hv\" define ON_ERR_GOTO (res, label, desc)"},{"location":"opae-code/hello__events_8c/#detailed-description","title":"Detailed Description","text":"This sample starts two processes. One process injects an artificial fatal error to sysfs; while the other tries to asynchronously capture and handle the event. This sample code exercises all major functions of the event API, including creating and destroying event handles, register and unregister events, polling on event file descriptor, and getting the OS object associated with an event. For a full discussion of OPAE event API, refer to event.h.
"},{"location":"opae-code/hello__events_8c/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/hello__events_8c/#function-error_thread","title":"function error_thread","text":"void * error_thread (\nvoid * arg\n)
"},{"location":"opae-code/hello__events_8c/#function-find_fpga","title":"function find_fpga","text":"fpga_result find_fpga (\nfpga_properties device_filter,\nfpga_token * fpga,\nuint32_t * num_matches\n)
"},{"location":"opae-code/hello__events_8c/#function-help","title":"function help","text":"void help (\nvoid\n)
"},{"location":"opae-code/hello__events_8c/#function-inject_ras_fatal_error","title":"function inject_ras_fatal_error","text":"fpga_result inject_ras_fatal_error (\nfpga_token fme_token,\nuint8_t err\n)
"},{"location":"opae-code/hello__events_8c/#function-main","title":"function main","text":"int main (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__events_8c/#function-parse_args","title":"function parse_args","text":"fpga_result parse_args (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__events_8c/#function-print_err","title":"function print_err","text":"void print_err (\nconst char * s,\nfpga_result res\n)
"},{"location":"opae-code/hello__events_8c/#function-usleep","title":"function usleep","text":"int usleep (\nunsigned\n)
"},{"location":"opae-code/hello__events_8c/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/hello__events_8c/#define-fme_sysfs_inject_error","title":"define FME_SYSFS_INJECT_ERROR","text":"#define FME_SYSFS_INJECT_ERROR \"errors/inject_errors\"\n
"},{"location":"opae-code/hello__events_8c/#define-getopt_string","title":"define GETOPT_STRING","text":"#define GETOPT_STRING \"hv\"\n
"},{"location":"opae-code/hello__events_8c/#define-on_err_goto","title":"define ON_ERR_GOTO","text":"#define ON_ERR_GOTO (\nres,\nlabel,\ndesc\n) do { \\\n if ((res) != FPGA_OK ) { \\ print_err ((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\n
The documentation for this class was generated from the following file docs/sw/samples/hello_events/hello_events.c
"},{"location":"opae-code/hello__events_8c_source/","title":"File hello_events.c","text":"File List > docs > sw > samples > hello_events > hello_events.c
Go to the documentation of this file.
// Copyright(c) 2017-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifdef HAVE_CONFIG_H\n#include <config.h>\n#endif // HAVE_CONFIG_H\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n#include <unistd.h>\n#include <getopt.h>\n#include <poll.h>\n#include <errno.h>\n#include <sys/stat.h>\n#include <pthread.h>\n#include <opae/fpga.h>\n#include <argsfilter.h>\n#include \"mock/opae_std.h\"\nint usleep(unsigned);\n#define FME_SYSFS_INJECT_ERROR \"errors/inject_errors\"\n#define ON_ERR_GOTO(res, label, desc) \\\n do { \\\n if ((res) != FPGA_OK) { \\\n print_err((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\nvoid print_err(const char *s, fpga_result res)\n{\nfprintf(stderr, \"Error %s: %s\\n\", s, fpgaErrStr(res));\n}\n// RAS Error Inject CSR\nstruct ras_inject_error {\nunion {\nuint64_t csr;\nstruct {\n/* Catastrophic error */\nuint64_t catastrophicr_error : 1;\n/* Fatal error */\nuint64_t fatal_error : 1;\n/* Non-fatal error */\nuint64_t nonfatal_error : 1;\n/* Reserved */\nuint64_t rsvd : 61;\n};\n};\n};\nfpga_result inject_ras_fatal_error(fpga_token fme_token, uint8_t err)\n{\nfpga_result res1 = FPGA_OK;\nfpga_result res2 = FPGA_OK;\nfpga_handle fme_handle = NULL;\nstruct ras_inject_error inj_error = { {0} };\nfpga_object inj_err_object;\nres1 = fpgaOpen(fme_token, &fme_handle, FPGA_OPEN_SHARED);\nif (res1 != FPGA_OK) {\nOPAE_ERR(\"Failed to open FPGA\");\nreturn res1;\n}\nres1 = fpgaHandleGetObject(fme_handle, FME_SYSFS_INJECT_ERROR, &inj_err_object, 0);\nON_ERR_GOTO(res1, out_close, \"Failed to get Handle Object\");\n// Inject fatal error\ninj_error.fatal_error = err;\nres1 = fpgaObjectWrite64(inj_err_object, inj_error.csr, 0);\nON_ERR_GOTO(res1, out_destroy_obj, \"Failed to Read Object\");\nout_destroy_obj:\nres2 = fpgaDestroyObject(&inj_err_object);\nON_ERR_GOTO(res2, out_close, \"Failed to Destroy Object\");\nout_close:\nres2 = fpgaClose(fme_handle);\nif (res2 != FPGA_OK) {\nOPAE_ERR(\"Failed to close FPGA\");\n}\nreturn res1 != FPGA_OK ? res1 : res2;\n}\nvoid *error_thread(void *arg)\n{\nfpga_token token = (fpga_token) arg;\nfpga_result res;\nusleep(5000000);\nprintf(\"injecting error\\n\");\nres = inject_ras_fatal_error(token, 1);\nif (res != FPGA_OK)\nprint_err(\"setting inject error register\", res);\nusleep(5000000);\nprintf(\"clearing error\\n\");\nres = inject_ras_fatal_error(token, 0);\nif (res != FPGA_OK)\nprint_err(\"clearing inject error register\", res);\nreturn NULL;\n}\nvoid help(void)\n{\nprintf(\"\\n\"\n\"hello_events\\n\"\n\"OPAE Events API sample\\n\"\n\"\\n\"\n\"Usage:\\n\"\n\" hello_events [-hv] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\\n\"\n\"\\n\"\n\" -h,--help Print this help\\n\"\n\" -v,--version Print version info and exit\\n\"\n\"\\n\");\n}\n/*\n * Parse command line arguments\n */\n#define GETOPT_STRING \"hv\"\nfpga_result parse_args(int argc, char *argv[])\n{\nstruct option longopts[] = {\n{ \"help\", no_argument, NULL, 'h' },\n{ \"version\", no_argument, NULL, 'v' },\n{ NULL, 0, NULL, 0 }\n};\nint getopt_ret;\nint option_index;\nwhile (-1 != (getopt_ret = getopt_long(argc, argv, GETOPT_STRING, longopts, &option_index))) {\nconst char *tmp_optarg = optarg;\n/* checks to see if optarg is null and if not goes to value of optarg */\nif ((optarg) && ('=' == *tmp_optarg)) {\n++tmp_optarg;\n}\nswitch (getopt_ret) {\ncase 'h': /* help */\nhelp();\nreturn -1;\ncase 'v': /* version */\nprintf(\"hello_events %s %s%s\\n\",\nOPAE_VERSION,\nOPAE_GIT_COMMIT_HASH,\nOPAE_GIT_SRC_TREE_DIRTY ? \"*\":\"\");\nreturn -1;\ndefault: /* invalid option */\nfprintf(stderr, \"Invalid cmdline options\\n\");\nreturn FPGA_EXCEPTION;\n}\n}\nreturn FPGA_OK;\n}\nfpga_result find_fpga(fpga_properties device_filter,\nfpga_token *fpga,\nuint32_t *num_matches)\n{\nfpga_properties filter = NULL;\nfpga_result res = FPGA_OK;\nfpga_result dres = FPGA_OK;\nres = fpgaCloneProperties(device_filter, &filter);\nON_ERR_GOTO(res, out, \"cloning properties object\");\nres = fpgaPropertiesSetObjectType(filter, FPGA_DEVICE);\nON_ERR_GOTO(res, out_destroy, \"setting interface ID\");\nres = fpgaEnumerate(&filter, 1, fpga, 1, num_matches);\nON_ERR_GOTO(res, out_destroy, \"enumerating FPGAs\");\nout_destroy:\ndres = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(dres, out, \"destroying properties object\");\nout:\nreturn (res == FPGA_OK) ? dres : res;\n}\nint main(int argc, char *argv[])\n{\nfpga_token fpga_device_token = NULL;\nfpga_handle fpga_device_handle = NULL;\nuint32_t num_matches = 1;\nfpga_result res1 = FPGA_OK;\nfpga_result res2 = FPGA_OK;\nfpga_event_handle eh;\nuint64_t count = 0;\nint res;\nstruct pollfd pfd;\nint timeout = 10000;\nint poll_ret = 0;\nssize_t bytes_read = 0;\npthread_t errthr;\nfpga_properties device_filter = NULL;\nres1 = fpgaGetProperties(NULL, &device_filter);\nif (res1) {\nprint_err(\"failed to alloc properties\", res1);\nreturn res1;\n}\nif (opae_set_properties_from_args(device_filter,\n&res1,\n&argc,\nargv)) {\nprint_err(\"failed arg parse\", res1);\nres1 = FPGA_EXCEPTION;\ngoto out_exit;\n} else if (res1) {\nprint_err(\"failed to set properties\", res1);\ngoto out_exit;\n}\nres1 = parse_args(argc, argv);\nif ((int)res1 < 0)\ngoto out_exit;\nON_ERR_GOTO(res1, out_exit, \"parsing arguments\");\nres1 = find_fpga(device_filter, &fpga_device_token, &num_matches);\nON_ERR_GOTO(res1, out_exit, \"finding FPGA accelerator\");\nif (num_matches < 1) {\nfprintf(stderr, \"No matches for address provided.\\n\");\nres1 = FPGA_NOT_FOUND;\ngoto out_exit;\n}\nif (num_matches > 1) {\nfprintf(stderr, \"Found more than one suitable slot.\\n\");\n}\nres1 = fpgaOpen(fpga_device_token, &fpga_device_handle, FPGA_OPEN_SHARED);\nON_ERR_GOTO(res1, out_destroy_tok, \"opening accelerator\");\nres1 = fpgaCreateEventHandle(&eh);\nON_ERR_GOTO(res1, out_close, \"creating event handle\");\nres1 = fpgaRegisterEvent(fpga_device_handle, FPGA_EVENT_ERROR, eh, 0);\nON_ERR_GOTO(res1, out_destroy_eh, \"registering an FME event\");\nprintf(\"Waiting for interrupts now...\\n\");\nres = pthread_create(&errthr, NULL, error_thread, fpga_device_token);\nif (res) {\nprintf(\"Failed to create error_thread.\\n\");\nres1 = FPGA_EXCEPTION;\ngoto out_destroy_eh;\n}\nres1 = fpgaGetOSObjectFromEventHandle(eh, &pfd.fd);\nON_ERR_GOTO(res1, out_join, \"getting file descriptor\");\npfd.events = POLLIN;\npoll_ret = poll(&pfd, 1, timeout);\nif (poll_ret < 0) {\nprintf(\"Poll error errno = %s\\n\", strerror(errno));\nres1 = FPGA_EXCEPTION;\ngoto out_join;\n} else if (poll_ret == 0) {\nprintf(\"Poll timeout occurred\\n\");\nres1 = FPGA_EXCEPTION;\ngoto out_join;\n} else {\nprintf(\"FME Interrupt occurred\\n\");\nbytes_read = opae_read(pfd.fd, &count, sizeof(count));\nif (bytes_read <= 0)\nprintf(\"WARNING: error reading from poll fd: %s\\n\",\nbytes_read < 0 ? strerror(errno) : \"zero bytes read\");\n}\nres1 = fpgaUnregisterEvent(fpga_device_handle, FPGA_EVENT_ERROR, eh);\nON_ERR_GOTO(res1, out_join, \"unregistering an FME event\");\nprintf(\"Successfully tested Register/Unregister for FME events!\\n\");\nout_join:\npthread_join(errthr, NULL);\nout_destroy_eh:\nres2 = fpgaDestroyEventHandle(&eh);\nON_ERR_GOTO(res2, out_close, \"deleting event handle\");\nout_close:\nres2 = fpgaClose(fpga_device_handle);\nON_ERR_GOTO(res2, out_destroy_tok, \"closing accelerator\");\nout_destroy_tok:\nres2 = fpgaDestroyToken(&fpga_device_token);\nON_ERR_GOTO(res2, out_exit, \"destroying token\");\nout_exit:\nfpgaDestroyProperties(&device_filter);\nreturn res1 != FPGA_OK ? res1 : res2;\n}\n
"},{"location":"opae-code/dir_a3c160366dc832de1042e5d4d49ef034/","title":"Dir docs/sw/samples/hello_fpga","text":"FileList > docs > sw > samples > hello_fpga
"},{"location":"opae-code/dir_a3c160366dc832de1042e5d4d49ef034/#files","title":"Files","text":"Type Name file hello_fpga.c A code sample illustrates the basic usage of the OPAE C API. The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/
"},{"location":"opae-code/hello__fpga_8c/","title":"File hello_fpga.c","text":"FileList > docs > sw > samples > hello_fpga > hello_fpga.c
Go to the source code of this file.
A code sample illustrates the basic usage of the OPAE C API. More...
#include <stdio.h>#include <stdlib.h>#include <string.h>#include <stdint.h>#include <getopt.h>#include <unistd.h>#include <uuid/uuid.h>#include <opae/fpga.h>#include <argsfilter.h>#include \"mock/opae_std.h\"
"},{"location":"opae-code/hello__fpga_8c/#classes","title":"Classes","text":"Type Name struct cache_line struct config"},{"location":"opae-code/hello__fpga_8c/#public-attributes","title":"Public Attributes","text":"Type Name struct config config = = { .open_flags = 0, .run_n3000 = 0 }"},{"location":"opae-code/hello__fpga_8c/#public-functions","title":"Public Functions","text":"Type Name fpga_result find_fpga (fpga_properties device_filter, fpga_guid afu_guid, fpga_token * accelerator_token, uint32_t * num_matches_accelerators) fpga_result find_nlb_n3000 (fpga_handle accelerator_handle, uint64_t * afu_baddr) void help (void) int main (int argc, char * argv) fpga_result parse_args (int argc, char * argv) void print_err (const char * s, fpga_result res) bool probe_for_ase (void) int usleep (unsigned)"},{"location":"opae-code/hello__fpga_8c/#macros","title":"Macros","text":"Type Name define CACHELINE_ALIGNED_ADDR (p) ((p) >> LOG2_CL) define CL (x) ((x) * 64) define CSR_AFU_DSM_BASEL 0x0110 define CSR_CFG 0x0140 define CSR_CTL 0x0138 define CSR_DST_ADDR 0x0128 define CSR_NUM_LINES 0x0130 define CSR_SRC_ADDR 0x0120 define CSR_STATUS1 0x0168 define DSM_STATUS_TEST_COMPLETE 0x40 define FPGA_NLB0_UUID_H 0xd8424dc4a4a3c413 define FPGA_NLB0_UUID_L 0xf89e433683f9040b define GETOPT_STRING \"hscv\" define LOG2_CL 6 define LPBK1_BUFFER_ALLOCATION_SIZE MB(2) define LPBK1_BUFFER_SIZE MB(1) define LPBK1_DSM_SIZE MB(2) define MB (x) ((x) * 1024 * 1024) define N3000_AFUID \"9AEFFE5F-8457-0612-C000-C9660D824272\" define NLB0_AFUID \"D8424DC4-A4A3-C413-F89E-433683F9040B\" define ON_ERR_GOTO (res, label, desc) define TEST_TIMEOUT 30000"},{"location":"opae-code/hello__fpga_8c/#detailed-description","title":"Detailed Description","text":"The sample is a host application that demonstrates the basic steps of interacting with FPGA using the OPAE library. These steps include:
- FPGA enumeration
- Resource acquiring and releasing
- Managing shared memory buffer
- MMIO read and write
The sample also demonstrates OPAE's object model, such as tokens, handles, and properties.
The sample requires a native loopback mode (NLB) test image to be loaded on the FPGA. Refer to Quick Start Guide for full instructions on building, configuring, and running this code sample.
"},{"location":"opae-code/hello__fpga_8c/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/hello__fpga_8c/#variable-config","title":"variable config","text":"struct config config;\n
"},{"location":"opae-code/hello__fpga_8c/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/hello__fpga_8c/#function-find_fpga","title":"function find_fpga","text":"fpga_result find_fpga (\nfpga_properties device_filter,\nfpga_guid afu_guid,\nfpga_token * accelerator_token,\nuint32_t * num_matches_accelerators\n)
"},{"location":"opae-code/hello__fpga_8c/#function-find_nlb_n3000","title":"function find_nlb_n3000","text":"fpga_result find_nlb_n3000 (\nfpga_handle accelerator_handle,\nuint64_t * afu_baddr\n)
"},{"location":"opae-code/hello__fpga_8c/#function-help","title":"function help","text":"void help (\nvoid\n)
"},{"location":"opae-code/hello__fpga_8c/#function-main","title":"function main","text":"int main (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__fpga_8c/#function-parse_args","title":"function parse_args","text":"fpga_result parse_args (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__fpga_8c/#function-print_err","title":"function print_err","text":"void print_err (\nconst char * s,\nfpga_result res\n)
"},{"location":"opae-code/hello__fpga_8c/#function-probe_for_ase","title":"function probe_for_ase","text":"bool probe_for_ase (\nvoid\n)
"},{"location":"opae-code/hello__fpga_8c/#function-usleep","title":"function usleep","text":"int usleep (\nunsigned\n)
"},{"location":"opae-code/hello__fpga_8c/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/hello__fpga_8c/#define-cacheline_aligned_addr","title":"define CACHELINE_ALIGNED_ADDR","text":"#define CACHELINE_ALIGNED_ADDR (\np\n) ((p) >> LOG2_CL )\n
"},{"location":"opae-code/hello__fpga_8c/#define-cl","title":"define CL","text":"#define CL (\nx\n) ((x) * 64)\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_afu_dsm_basel","title":"define CSR_AFU_DSM_BASEL","text":"#define CSR_AFU_DSM_BASEL 0x0110\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_cfg","title":"define CSR_CFG","text":"#define CSR_CFG 0x0140\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_ctl","title":"define CSR_CTL","text":"#define CSR_CTL 0x0138\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_dst_addr","title":"define CSR_DST_ADDR","text":"#define CSR_DST_ADDR 0x0128\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_num_lines","title":"define CSR_NUM_LINES","text":"#define CSR_NUM_LINES 0x0130\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_src_addr","title":"define CSR_SRC_ADDR","text":"#define CSR_SRC_ADDR 0x0120\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_status1","title":"define CSR_STATUS1","text":"#define CSR_STATUS1 0x0168\n
"},{"location":"opae-code/hello__fpga_8c/#define-dsm_status_test_complete","title":"define DSM_STATUS_TEST_COMPLETE","text":"#define DSM_STATUS_TEST_COMPLETE 0x40\n
"},{"location":"opae-code/hello__fpga_8c/#define-fpga_nlb0_uuid_h","title":"define FPGA_NLB0_UUID_H","text":"#define FPGA_NLB0_UUID_H 0xd8424dc4a4a3c413\n
"},{"location":"opae-code/hello__fpga_8c/#define-fpga_nlb0_uuid_l","title":"define FPGA_NLB0_UUID_L","text":"#define FPGA_NLB0_UUID_L 0xf89e433683f9040b\n
"},{"location":"opae-code/hello__fpga_8c/#define-getopt_string","title":"define GETOPT_STRING","text":"#define GETOPT_STRING \"hscv\"\n
"},{"location":"opae-code/hello__fpga_8c/#define-log2_cl","title":"define LOG2_CL","text":"#define LOG2_CL 6\n
"},{"location":"opae-code/hello__fpga_8c/#define-lpbk1_buffer_allocation_size","title":"define LPBK1_BUFFER_ALLOCATION_SIZE","text":"#define LPBK1_BUFFER_ALLOCATION_SIZE MB (2)\n
"},{"location":"opae-code/hello__fpga_8c/#define-lpbk1_buffer_size","title":"define LPBK1_BUFFER_SIZE","text":"#define LPBK1_BUFFER_SIZE MB (1)\n
"},{"location":"opae-code/hello__fpga_8c/#define-lpbk1_dsm_size","title":"define LPBK1_DSM_SIZE","text":"#define LPBK1_DSM_SIZE MB (2)\n
"},{"location":"opae-code/hello__fpga_8c/#define-mb","title":"define MB","text":"#define MB (\nx\n) ((x) * 1024 * 1024)\n
"},{"location":"opae-code/hello__fpga_8c/#define-n3000_afuid","title":"define N3000_AFUID","text":"#define N3000_AFUID \"9AEFFE5F-8457-0612-C000-C9660D824272\"\n
"},{"location":"opae-code/hello__fpga_8c/#define-nlb0_afuid","title":"define NLB0_AFUID","text":"#define NLB0_AFUID \"D8424DC4-A4A3-C413-F89E-433683F9040B\"\n
"},{"location":"opae-code/hello__fpga_8c/#define-on_err_goto","title":"define ON_ERR_GOTO","text":"#define ON_ERR_GOTO (\nres,\nlabel,\ndesc\n) do { \\\n if ((res) != FPGA_OK ) { \\ print_err ((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\n
"},{"location":"opae-code/hello__fpga_8c/#define-test_timeout","title":"define TEST_TIMEOUT","text":"#define TEST_TIMEOUT 30000\n
The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/hello_fpga.c
"},{"location":"opae-code/hello__fpga_8c_source/","title":"File hello_fpga.c","text":"File List > docs > sw > samples > hello_fpga > hello_fpga.c
Go to the documentation of this file.
// Copyright(c) 2017-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifdef HAVE_CONFIG_H\n#include <config.h>\n#endif // HAVE_CONFIG_H\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n#include <stdint.h>\n#include <getopt.h>\n#include <unistd.h>\n#include <uuid/uuid.h>\n#include <opae/fpga.h>\n#include <argsfilter.h>\n#include \"mock/opae_std.h\"\nint usleep(unsigned);\n#ifndef TEST_TIMEOUT\n#define TEST_TIMEOUT 30000\n#endif // TEST_TIMEOUT\n#ifndef CL\n# define CL(x) ((x) * 64)\n#endif // CL\n#ifndef LOG2_CL\n# define LOG2_CL 6\n#endif // LOG2_CL\n#ifndef MB\n# define MB(x) ((x) * 1024 * 1024)\n#endif // MB\n#define CACHELINE_ALIGNED_ADDR(p) ((p) >> LOG2_CL)\n#define LPBK1_BUFFER_SIZE MB(1)\n#define LPBK1_BUFFER_ALLOCATION_SIZE MB(2)\n#define LPBK1_DSM_SIZE MB(2)\n#define CSR_SRC_ADDR 0x0120\n#define CSR_DST_ADDR 0x0128\n#define CSR_CTL 0x0138\n#define CSR_STATUS1 0x0168\n#define CSR_CFG 0x0140\n#define CSR_NUM_LINES 0x0130\n#define DSM_STATUS_TEST_COMPLETE 0x40\n#define CSR_AFU_DSM_BASEL 0x0110\n/* NLB0 AFU_ID */\n#define NLB0_AFUID \"D8424DC4-A4A3-C413-F89E-433683F9040B\"\n/* NLB0 AFU_ID for N3000 */\n#define N3000_AFUID \"9AEFFE5F-8457-0612-C000-C9660D824272\"\n#define FPGA_NLB0_UUID_H 0xd8424dc4a4a3c413\n#define FPGA_NLB0_UUID_L 0xf89e433683f9040b\n/*\n * macro to check return codes, print error message, and goto cleanup label\n * NOTE: this changes the program flow (uses goto)!\n */\n#define ON_ERR_GOTO(res, label, desc) \\\n do { \\\n if ((res) != FPGA_OK) { \\\n print_err((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\n/* Type definitions */\ntypedef struct {\nuint32_t uint[16];\n} cache_line;\nvoid print_err(const char *s, fpga_result res)\n{\nfprintf(stderr, \"Error %s: %s\\n\", s, fpgaErrStr(res));\n}\n/*\n * Global configuration of bus, set during parse_args()\n * */\nstruct config {\nint open_flags;\nint run_n3000;\n}\nconfig = {\n.open_flags = 0,\n.run_n3000 = 0\n};\nvoid help(void)\n{\nprintf(\"\\n\"\n\"hello_fpga\\n\"\n\"OPAE Native Loopback 0 (NLB0) sample\\n\"\n\"\\n\"\n\"Usage:\\n\"\n\" hello_fpga [-schv] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\\n\"\n\"\\n\"\n\" -s,--shared Open accelerator in shared mode\\n\"\n\" -c,--n3000 Assume N3000 MMIO layout\\n\"\n\" -h,--help Print this help\\n\"\n\" -v,--version Print version info and exit\\n\"\n\"\\n\");\n}\n#define GETOPT_STRING \"hscv\"\nfpga_result parse_args(int argc, char *argv[])\n{\nstruct option longopts[] = {\n{ \"help\", no_argument, NULL, 'h' },\n{ \"shared\", no_argument, NULL, 's' },\n{ \"n3000\", no_argument, NULL, 'c' },\n{ \"version\", no_argument, NULL, 'v' },\n{ NULL, 0, NULL, 0 }\n};\nint getopt_ret;\nint option_index;\nchar version[32];\nchar build[32];\nwhile (-1 != (getopt_ret = getopt_long(argc, argv, GETOPT_STRING,\nlongopts, &option_index))) {\nconst char *tmp_optarg = optarg;\n/* Checks to see if optarg is null and if not it goes to value of optarg */\nif ((optarg) && ('=' == *tmp_optarg)) {\n++tmp_optarg;\n}\nswitch (getopt_ret) {\ncase 'h':\nhelp();\nreturn -1;\ncase 's':\nconfig.open_flags |= FPGA_OPEN_SHARED;\nbreak;\ncase 'c':\nconfig.run_n3000 = 1;\nbreak;\ncase 'v':\nfpgaGetOPAECVersionString(version, sizeof(version));\nfpgaGetOPAECBuildString(build, sizeof(build));\nprintf(\"hello_fpga %s %s\\n\",\nversion, build);\nreturn -1;\ndefault: /* invalid option */\nfprintf(stderr, \"Invalid cmdline option \\n\");\nreturn FPGA_EXCEPTION;\n}\n}\nreturn FPGA_OK;\n}\nfpga_result find_fpga(fpga_properties device_filter,\nfpga_guid afu_guid,\nfpga_token *accelerator_token,\nuint32_t *num_matches_accelerators)\n{\nfpga_properties filter = NULL;\nfpga_result res1;\nfpga_result res2 = FPGA_OK;\nres1 = fpgaCloneProperties(device_filter, &filter);\nON_ERR_GOTO(res1, out, \"cloning properties object\");\nres1 = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nON_ERR_GOTO(res1, out_destroy, \"setting object type\");\nres1 = fpgaPropertiesSetGUID(filter, afu_guid);\nON_ERR_GOTO(res1, out_destroy, \"setting GUID\");\nres1 = fpgaEnumerate(&filter, 1, accelerator_token, 1, num_matches_accelerators);\nON_ERR_GOTO(res1, out_destroy, \"enumerating accelerators\");\nout_destroy:\nres2 = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(res2, out, \"destroying properties object\");\nout:\nreturn res1 != FPGA_OK ? res1 : res2;\n}\n/* Is the FPGA simulated with ASE? */\nbool probe_for_ase(void)\n{\nfpga_result r = FPGA_OK;\nuint16_t device_id = 0;\nfpga_properties filter = NULL;\nuint32_t num_matches = 1;\nfpga_token fme_token;\n/* Connect to the FPGA management engine */\nfpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_DEVICE);\n/* Connecting to one is sufficient to find ASE */\nfpgaEnumerate(&filter, 1, &fme_token, 1, &num_matches);\nif (0 != num_matches) {\n/* Retrieve the device ID of the FME */\nfpgaDestroyProperties(&filter);\nfpgaGetProperties(fme_token, &filter);\nr = fpgaPropertiesGetDeviceID(filter, &device_id);\nfpgaDestroyToken(&fme_token);\n}\nfpgaDestroyProperties(&filter);\n/* ASE's device ID is 0xa5e */\nreturn ((FPGA_OK == r) && (0xa5e == device_id));\n}\nfpga_result find_nlb_n3000(fpga_handle accelerator_handle,\nuint64_t *afu_baddr)\n{\nfpga_result res1 = FPGA_OK;\nint end_of_list = 0;\nint nlb0_found = 0;\nuint64_t header = 0;\nuint64_t uuid_hi = 0;\nuint64_t uuid_lo = 0;\nuint64_t next_offset = 0;\nuint64_t nlb0_offset = 0;\n/* find NLB0 in AFU */\ndo {\n// Read the next feature header\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb0_offset, &header);\nON_ERR_GOTO(res1, out_exit, \"fpgaReadMMIO64\");\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb0_offset+8, &uuid_lo);\nON_ERR_GOTO(res1, out_exit, \"fpgaReadMMIO64\");\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb0_offset+16, &uuid_hi);\nON_ERR_GOTO(res1, out_exit, \"fpgaReadMMIO64\");\n// printf(\"%zx: %zx %zx %zx\\n\", nlb0_offset, header, uuid_lo, uuid_hi);\nif ((((header >> 60) & 0xf) == 0x1) &&\n(uuid_lo == FPGA_NLB0_UUID_L) && (uuid_hi == FPGA_NLB0_UUID_H)) {\nnlb0_found = 1;\nbreak;\n}\n// End of the list flag\nend_of_list = (header >> 40) & 1;\n// Move to the next feature header\nnext_offset = (header >> 16) & 0xffffff;\nif ((next_offset == 0xffff) || (next_offset == 0)) {\nnlb0_found = 0;\nbreak;\n}\nnlb0_offset = nlb0_offset + next_offset;\n} while (!end_of_list);\nif (!nlb0_found) {\nprintf(\"AFU NLB0 not found\\n\");\nreturn FPGA_EXCEPTION;\n}\nprintf(\"AFU NLB0 found @ %zx\\n\", nlb0_offset);\n*afu_baddr = nlb0_offset;\nreturn FPGA_OK;\nout_exit:\nreturn FPGA_EXCEPTION;\n}\nint main(int argc, char *argv[])\n{\nfpga_token accelerator_token;\nfpga_handle accelerator_handle;\nfpga_guid guid;\nuint32_t num_matches_accelerators = 0;\nuint32_t use_ase;\nvolatile uint64_t *dsm_ptr = NULL;\nvolatile uint64_t *status_ptr = NULL;\nvolatile uint64_t *input_ptr = NULL;\nvolatile uint64_t *output_ptr = NULL;\nuint64_t dsm_wsid;\nuint64_t input_wsid;\nuint64_t output_wsid;\nuint32_t i;\nuint32_t timeout;\nfpga_result res1 = FPGA_OK;\nfpga_result res2 = FPGA_OK;\nfpga_properties device_filter = NULL;\nres1 = fpgaGetProperties(NULL, &device_filter);\nif (res1 != FPGA_OK) {\nprint_err(\"failed to allocate properties.\\n\", res1);\nreturn 1;\n}\nif (opae_set_properties_from_args(device_filter,\n&res1,\n&argc,\nargv)) {\nprint_err(\"failed arg parse.\\n\", res1);\nres1 = FPGA_EXCEPTION;\ngoto out_exit;\n} else if (res1) {\nprint_err(\"failed to set properties.\\n\", res1);\ngoto out_exit;\n}\nres1 = parse_args(argc, argv);\nif ((int)res1 < 0)\ngoto out_exit;\nON_ERR_GOTO(res1, out_exit, \"parsing arguments\");\nif (config.run_n3000) {\nif (uuid_parse(N3000_AFUID, guid) < 0)\nres1 = FPGA_EXCEPTION;\n} else {\nif (uuid_parse(NLB0_AFUID, guid) < 0)\nres1 = FPGA_EXCEPTION;\n}\nON_ERR_GOTO(res1, out_exit, \"parsing guid\");\nuse_ase = probe_for_ase();\nif (use_ase) {\nprintf(\"Running in ASE mode\\n\");\n}\n/* Look for accelerator with NLB0_AFUID */\nres1 = find_fpga(device_filter,\nguid,\n&accelerator_token,\n&num_matches_accelerators);\nON_ERR_GOTO(res1, out_exit, \"finding FPGA accelerator\");\nif (num_matches_accelerators == 0) {\nres1 = FPGA_NOT_FOUND;\n}\nON_ERR_GOTO(res1, out_exit, \"no matching accelerator\");\nif (num_matches_accelerators > 1) {\nprintf(\"Found more than one suitable accelerator. \");\n}\n/* Open accelerator and map MMIO */\nres1 = fpgaOpen(accelerator_token, &accelerator_handle, config.open_flags);\nON_ERR_GOTO(res1, out_destroy_tok, \"opening accelerator\");\nres1 = fpgaMapMMIO(accelerator_handle, 0, NULL);\nON_ERR_GOTO(res1, out_close, \"mapping MMIO space\");\n/* Allocate buffers */\nres1 = fpgaPrepareBuffer(accelerator_handle, LPBK1_DSM_SIZE,\n(void **)&dsm_ptr, &dsm_wsid, 0);\nON_ERR_GOTO(res1, out_close, \"allocating DSM buffer\");\nres1 = fpgaPrepareBuffer(accelerator_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&input_ptr, &input_wsid, 0);\nON_ERR_GOTO(res1, out_free_dsm, \"allocating input buffer\");\nres1 = fpgaPrepareBuffer(accelerator_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&output_ptr, &output_wsid, 0);\nON_ERR_GOTO(res1, out_free_input, \"allocating output buffer\");\nprintf(\"Running Test\\n\");\n/* Initialize buffers */\nmemset((void *)dsm_ptr, 0, LPBK1_DSM_SIZE);\nmemset((void *)input_ptr, 0xAF, LPBK1_BUFFER_SIZE);\nmemset((void *)output_ptr, 0xBE, LPBK1_BUFFER_SIZE);\ncache_line *cl_ptr = (cache_line *)input_ptr;\nfor (i = 0; i < LPBK1_BUFFER_SIZE / CL(1); ++i) {\ncl_ptr[i].uint[15] = i+1; /* set the last uint in every cacheline */\n}\n/* Reset accelerator */\nres1 = fpgaReset(accelerator_handle);\nON_ERR_GOTO(res1, out_free_output, \"resetting accelerator\");\nuint64_t nlb_base_addr = 0;\nif (config.run_n3000) {\nres1 = find_nlb_n3000(accelerator_handle, &nlb_base_addr);\nON_ERR_GOTO(res1, out_free_output, \"finding nlb in AFU\");\n}\n/* Program DMA addresses */\nuint64_t iova = 0;\nres1 = fpgaGetIOAddress(accelerator_handle, dsm_wsid, &iova);\nON_ERR_GOTO(res1, out_free_output, \"getting DSM IOVA\");\nres1 = fpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_AFU_DSM_BASEL, iova);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_AFU_DSM_BASEL\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 0);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 1);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\nres1 = fpgaGetIOAddress(accelerator_handle, input_wsid, &iova);\nON_ERR_GOTO(res1, out_free_output, \"getting input IOVA\");\nres1 = fpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_SRC_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_SRC_ADDR\");\nres1 = fpgaGetIOAddress(accelerator_handle, output_wsid, &iova);\nON_ERR_GOTO(res1, out_free_output, \"getting output IOVA\");\nres1 = fpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_DST_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_DST_ADDR\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_NUM_LINES, LPBK1_BUFFER_SIZE / CL(1));\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_NUM_LINES\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CFG, 0x42000);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\nstatus_ptr = dsm_ptr + DSM_STATUS_TEST_COMPLETE/sizeof(uint64_t);\n/* Start the test */\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 3);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\n/* Wait for test completion */\ntimeout = TEST_TIMEOUT;\nwhile (0 == ((*status_ptr) & 0x1)) {\nusleep(100);\nif (!use_ase && (--timeout == 0)) {\nres1 = FPGA_EXCEPTION;\nON_ERR_GOTO(res1, out_free_output, \"test timed out\");\n}\n}\n/* Stop the device */\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 7);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\n/* Wait for the AFU's read/write traffic to complete */\nuint32_t afu_traffic_trips = 0;\nwhile (afu_traffic_trips < 100) {\n/*\n * CSR_STATUS1 holds two 32 bit values: num pending reads and writes.\n * Wait for it to be 0.\n */\nuint64_t s1;\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_STATUS1, &s1);\nON_ERR_GOTO(res1, out_free_output, \"reading CSR_STATUS1\");\nif (s1 == 0) {\nbreak;\n}\nafu_traffic_trips += 1;\nusleep(1000);\n}\n/* Check output buffer contents */\nfor (i = 0; i < LPBK1_BUFFER_SIZE; i++) {\nif (((uint8_t *)output_ptr)[i] != ((uint8_t *)input_ptr)[i]) {\nfprintf(stderr, \"Output does NOT match input \"\n\"at offset %i!\\n\", i);\nbreak;\n}\n}\nprintf(\"Done Running Test\\n\");\n/* Release buffers */\nout_free_output:\nres2 = fpgaReleaseBuffer(accelerator_handle, output_wsid);\nON_ERR_GOTO(res2, out_free_input, \"releasing output buffer\");\nout_free_input:\nres2 = fpgaReleaseBuffer(accelerator_handle, input_wsid);\nON_ERR_GOTO(res2, out_free_dsm, \"releasing input buffer\");\nout_free_dsm:\nres2 = fpgaReleaseBuffer(accelerator_handle, dsm_wsid);\nON_ERR_GOTO(res2, out_unmap, \"releasing DSM buffer\");\n/* Unmap MMIO space */\nout_unmap:\nres2 = fpgaUnmapMMIO(accelerator_handle, 0);\nON_ERR_GOTO(res2, out_close, \"unmapping MMIO space\");\n/* Release accelerator */\nout_close:\nres2 = fpgaClose(accelerator_handle);\nON_ERR_GOTO(res2, out_destroy_tok, \"closing accelerator\");\n/* Destroy token */\nout_destroy_tok:\nres2 = fpgaDestroyToken(&accelerator_token);\nON_ERR_GOTO(res2, out_exit, \"destroying token\");\nout_exit:\nfpgaDestroyProperties(&device_filter);\nreturn res1 != FPGA_OK ? res1 : res2;\n}\n
"},{"location":"opae-code/namespaces/","title":"Namespace List","text":"Here is a list of all namespaces with brief descriptions:
- namespace opae
- namespace fpga
- namespace types
- namespace detail
- namespace std
"},{"location":"opae-code/classes/","title":"Class Index","text":""},{"location":"opae-code/classes/#b","title":"b","text":" - busy (opae::fpga::types)
"},{"location":"opae-code/classes/#c","title":"c","text":" - cache_line
- config
"},{"location":"opae-code/classes/#e","title":"e","text":" - error (opae::fpga::types)
- event (opae::fpga::types)
- except (opae::fpga::types)
- exception (opae::fpga::types)
"},{"location":"opae-code/classes/#f","title":"f","text":" - fpga_error_info
- fpga_metric
- fpga_metric_info
- fpga_version
"},{"location":"opae-code/classes/#g","title":"g","text":" - guid_t (opae::fpga::types)
"},{"location":"opae-code/classes/#h","title":"h","text":" - handle (opae::fpga::types)
"},{"location":"opae-code/classes/#i","title":"i","text":" - invalid_param (opae::fpga::types)
"},{"location":"opae-code/classes/#m","title":"m","text":" - mem_alloc
- mem_link
- metric_threshold
"},{"location":"opae-code/classes/#n","title":"n","text":" - no_access (opae::fpga::types)
- no_daemon (opae::fpga::types)
- no_driver (opae::fpga::types)
- no_memory (opae::fpga::types)
- not_found (opae::fpga::types)
- not_supported (opae::fpga::types)
"},{"location":"opae-code/classes/#o","title":"o","text":" - opae_uio
- opae_uio_device_region
- opae_vfio
- opae_vfio_buffer
- opae_vfio_device
- opae_vfio_device_irq
- opae_vfio_device_region
- opae_vfio_group
- opae_vfio_iova_range
- opae_vfio_sparse_info
"},{"location":"opae-code/classes/#p","title":"p","text":" - properties (opae::fpga::types)
- pvalue (opae::fpga::types)
"},{"location":"opae-code/classes/#r","title":"r","text":" - reconf_error (opae::fpga::types)
- ras_inject_error
"},{"location":"opae-code/classes/#s","title":"s","text":" - shared_buffer (opae::fpga::types)
- src_location (opae::fpga::types)
- sysobject (opae::fpga::types)
"},{"location":"opae-code/classes/#t","title":"t","text":" - token (opae::fpga::types)
- type_t (opae::fpga::types::event)
- threshold
"},{"location":"opae-code/classes/#v","title":"v","text":" - version (opae::fpga::types)
## \\
- _fpga_token_header
- _opae_hash_map
- _opae_hash_map_item
"},{"location":"opae-code/hierarchy/","title":"Class Hierarchy","text":"This inheritance list is sorted roughly, but not completely, alphabetically:
- class opae::fpga::types::error An error object represents an error register for a resource.
- class opae::fpga::types::event Wraps fpga event routines in OPAE C.
- class opae::fpga::types::handle An allocated accelerator resource.
- class opae::fpga::types::properties Wraps an OPAE fpga_properties object.
- class opae::fpga::types::shared_buffer Host/AFU shared memory blocks.
- class opae::fpga::types::src_location Identify a particular line in a source file.
- class opae::fpga::types::sysobject Wraps the OPAE fpga_object primitive.
- class opae::fpga::types::token Wraps the OPAE fpga_token primitive.
- class opae::fpga::types::version
- struct _fpga_token_header Internal token type header.
- struct _opae_hash_map Hash map object.
- struct _opae_hash_map_item List link item.
- struct cache_line
- struct config
- struct fpga_error_info
- struct fpga_metric Metric struct.
- struct fpga_metric_info Metric info struct.
- struct fpga_version Semantic version.
- struct mem_alloc
- struct mem_link Provides an API for allocating/freeing a logical address space.
- struct metric_threshold
- struct opae::fpga::types::event::type_t C++ struct that is interchangeable with fpga_event_type enum.
- struct opae::fpga::types::guid_t Representation of the guid member of a properties object.
- struct opae::fpga::types::pvalue Wraps OPAE properties defined in the OPAE C API by associating an
fpga_properties reference with the getters and setters defined for a property. - struct opae_uio OPAE UIO device abstraction.
- struct opae_uio_device_region MMIO region info.
- struct opae_vfio OPAE VFIO device abstraction.
- struct opae_vfio_buffer System DMA buffer.
- struct opae_vfio_device VFIO device.
- struct opae_vfio_device_irq Interrupt info.
- struct opae_vfio_device_region MMIO region info.
- struct opae_vfio_group VFIO group.
- struct opae_vfio_iova_range IO Virtual Address Range.
- struct opae_vfio_sparse_info MMIO sparse-mappable region info.
- struct ras_inject_error
- struct threshold Threshold struct.
- class std::exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
"},{"location":"opae-code/modules/","title":"Modules","text":"Here is a list of all modules:
"},{"location":"opae-code/todo/","title":"Todo List","text":""},{"location":"opae-code/todo/#global-fpgaregisterevent-fpga_handle-handle-fpga_event_type-event_type-fpga_event_handle-event_handle-uint32_t-flags","title":"Global fpgaRegisterEvent (fpga_handle handle, fpga_event_type event_type, fpga_event_handle event_handle, uint32_t flags)","text":"define if calling fpgaRegisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored. define if calling fpgaUnregisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored.
"},{"location":"opae-code/pages/","title":"Class List","text":"Here are the classes, structs, unions and interfaces with brief descriptions:
"},{"location":"opae-code/class_members/","title":"Class Members","text":""},{"location":"opae-code/class_members/#a","title":"a","text":" - allocated (mem_alloc)
- address (mem_link)
- accelerator_state (opae::fpga::types::properties)
- allocate (opae::fpga::types::shared_buffer)
- attach (opae::fpga::types::shared_buffer)
- as_string (opae::fpga::types::version)
- as_struct (opae::fpga::types::version)
"},{"location":"opae-code/class_members/#b","title":"b","text":" - bus (_fpga_token_header, opae::fpga::types::properties)
- buckets (_opae_hash_map)
- busy (opae::fpga::types::busy)
- buf_ (opae::fpga::types::except)
- bbs_id (opae::fpga::types::properties)
- bbs_version (opae::fpga::types::properties)
- bytes (opae::fpga::types::sysobject)
- build (opae::fpga::types::version)
- buffer_iova (opae_vfio_buffer)
- buffer_ptr (opae_vfio_buffer)
- buffer_size (opae_vfio_buffer)
"},{"location":"opae-code/class_members/#c","title":"c","text":" - cleanup_context (_opae_hash_map)
- can_clear (fpga_error_info, opae::fpga::types::error)
- c_type (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
- close (opae::fpga::types::handle)
- capabilities (opae::fpga::types::properties)
- copy_ (opae::fpga::types::pvalue)
- copy_t (opae::fpga::types::pvalue)
- compare (opae::fpga::types::shared_buffer)
- cont_buffers (opae_vfio)
- cont_device (opae_vfio)
- cont_fd (opae_vfio)
- cont_pciaddr (opae_vfio)
- cont_ranges (opae_vfio)
- count (opae_vfio_device_irq)
- catastrophicr_error (ras_inject_error)
- csr (ras_inject_error)
"},{"location":"opae-code/class_members/#d","title":"d","text":" - device (_fpga_token_header, opae::fpga::types::properties, opae_vfio)
- device_id (_fpga_token_header, opae::fpga::types::properties)
- data_ (opae::fpga::types::guid_t)
- device_fd (opae_uio, opae_vfio_device)
- device_path (opae_uio)
- device_config_offset (opae_vfio_device)
- device_num_irqs (opae_vfio_device)
- device_num_regions (opae_vfio_device)
"},{"location":"opae-code/class_members/#e","title":"e","text":" - error (opae::fpga::types::error, opae::fpga::types::event::type_t)
- error_info_ (opae::fpga::types::error)
- error_num_ (opae::fpga::types::error)
- event (opae::fpga::types::event)
- event_handle_ (opae::fpga::types::event)
- except (opae::fpga::types::except)
- exception (opae::fpga::types::exception)
- enumerate (opae::fpga::types::token)
- event_fds (opae_vfio_device_irq)
- end (opae_vfio_iova_range)
"},{"location":"opae-code/class_members/#f","title":"f","text":" - function (_fpga_token_header, opae::fpga::types::properties)
- flags (_opae_hash_map, opae_vfio_buffer, opae_vfio_device_irq)
- free (mem_alloc)
- fill (opae::fpga::types::shared_buffer)
- file (opae::fpga::types::src_location)
- file_ (opae::fpga::types::src_location)
- fn (opae::fpga::types::src_location)
- fn_ (opae::fpga::types::src_location)
- fatal_error (ras_inject_error)
"},{"location":"opae-code/class_members/#g","title":"g","text":" - guid (_fpga_token_header, opae::fpga::types::properties)
- group_name (fpga_metric_info)
- get (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::properties, opae::fpga::types::sysobject)
- guid_t (opae::fpga::types::guid_t)
- get_token (opae::fpga::types::handle)
- get_ (opae::fpga::types::pvalue)
- get_value (opae::fpga::types::pvalue)
- getter_t (opae::fpga::types::pvalue)
- get_parent (opae::fpga::types::token)
- group (opae_vfio)
- group_device (opae_vfio_group)
- group_fd (opae_vfio_group)
"},{"location":"opae-code/class_members/#h","title":"h","text":" - hash_seed (_opae_hash_map)
- hysteresis (metric_threshold)
- handle_ (opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject)
- handle (opae::fpga::types::handle)
"},{"location":"opae-code/class_members/#i","title":"i","text":" - interface (_fpga_token_header, opae::fpga::types::properties)
- isvalid (fpga_metric)
- interrupt (opae::fpga::types::event::type_t)
- invalidate (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- is_set (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- is_set_ (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- invalid_param (opae::fpga::types::invalid_param)
- io_address (opae::fpga::types::shared_buffer)
- io_address_ (opae::fpga::types::shared_buffer)
- iova_alloc (opae_vfio)
- irqs (opae_vfio_device)
- index (opae_vfio_device_irq, opae_vfio_sparse_info)
- is_valid (threshold)
"},{"location":"opae-code/class_members/#k","title":"k","text":" - key_cleanup (_opae_hash_map)
- key_compare (_opae_hash_map)
- key_hash (_opae_hash_map)
- key (_opae_hash_map_item)
"},{"location":"opae-code/class_members/#l","title":"l","text":" - lower_c_threshold (metric_threshold)
- lower_nc_threshold (metric_threshold)
- lower_nr_threshold (metric_threshold)
- loc_ (opae::fpga::types::except)
- local_memory_size (opae::fpga::types::properties)
- len_ (opae::fpga::types::shared_buffer)
- line (opae::fpga::types::src_location)
- line_ (opae::fpga::types::src_location)
- lock (opae_vfio)
"},{"location":"opae-code/class_members/#m","title":"m","text":" - magic (_fpga_token_header)
- metric_num (fpga_metric, fpga_metric_info)
- metric_datatype (fpga_metric_info)
- metric_guid (fpga_metric_info)
- metric_name (fpga_metric_info, metric_threshold)
- metric_type (fpga_metric_info)
- metric_units (fpga_metric_info)
- major (fpga_version)
- minor (fpga_version)
- MAX_EXCEPT (opae::fpga::types::except)
- msg_ (opae::fpga::types::except)
- mmio_ptr (opae::fpga::types::handle)
- model (opae::fpga::types::properties)
- masks (opae_vfio_device_irq)
"},{"location":"opae-code/class_members/#n","title":"n","text":" - num_buckets (_opae_hash_map)
- next (_opae_hash_map_item, mem_link, opae_uio_device_region, opae_vfio_device_irq, opae_vfio_device_region, opae_vfio_iova_range, opae_vfio_sparse_info)
- name (fpga_error_info, opae::fpga::types::error)
- no_access (opae::fpga::types::no_access)
- no_daemon (opae::fpga::types::no_daemon)
- no_driver (opae::fpga::types::no_driver)
- no_memory (opae::fpga::types::no_memory)
- not_found (opae::fpga::types::not_found)
- not_supported (opae::fpga::types::not_supported)
- none (opae::fpga::types::properties)
- num_errors (opae::fpga::types::properties)
- num_interrupts (opae::fpga::types::properties)
- num_mmio (opae::fpga::types::properties)
- num_slots (opae::fpga::types::properties)
- nonfatal_error (ras_inject_error)
"},{"location":"opae-code/class_members/#o","title":"o","text":" - object_id (_fpga_token_header, opae::fpga::types::properties)
- objtype (_fpga_token_header)
- open_flags (config)
- operator= (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::pvalue, opae::fpga::types::shared_buffer, opae::fpga::types::src_location, opae::fpga::types::sysobject)
- operator fpga_event_type (opae::fpga::types::event::type_t)
- operator fpga_event_handle (opae::fpga::types::event)
- os_object (opae::fpga::types::event)
- os_object_ (opae::fpga::types::event)
- operator fpga_result (opae::fpga::types::except)
- operator uint8_t * (opae::fpga::types::guid_t)
- operator== (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- open (opae::fpga::types::handle)
- operator fpga_handle (opae::fpga::types::handle)
- operator fpga_properties (opae::fpga::types::properties)
- operator copy_t (opae::fpga::types::pvalue)
- owner (opae::fpga::types::shared_buffer)
- operator fpga_object (opae::fpga::types::sysobject)
- operator fpga_token (opae::fpga::types::token)
- offset (opae_vfio_sparse_info)
"},{"location":"opae-code/class_members/#p","title":"p","text":" - patch (fpga_version)
- prev (mem_link)
- ptr_t (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
- power_thermal (opae::fpga::types::event::type_t)
- parse (opae::fpga::types::guid_t)
- props_ (opae::fpga::types::guid_t, opae::fpga::types::properties, opae::fpga::types::pvalue)
- parent (opae::fpga::types::properties)
- properties (opae::fpga::types::properties)
- pvalue (opae::fpga::types::pvalue)
- ptr (opae_vfio_sparse_info)
"},{"location":"opae-code/class_members/#q","title":"q","text":" - qualifier_name (fpga_metric_info)
"},{"location":"opae-code/class_members/#r","title":"r","text":" - run_n3000 (config)
- read_value (opae::fpga::types::error)
- register_event (opae::fpga::types::event)
- res_ (opae::fpga::types::except)
- read_csr32 (opae::fpga::types::handle)
- read_csr64 (opae::fpga::types::handle)
- reconfigure (opae::fpga::types::handle)
- reset (opae::fpga::types::handle)
- reconf_error (opae::fpga::types::reconf_error)
- read (opae::fpga::types::shared_buffer)
- release (opae::fpga::types::shared_buffer)
- read64 (opae::fpga::types::sysobject)
- regions (opae_uio, opae_vfio_device)
- region_index (opae_uio_device_region, opae_vfio_device_region)
- region_page_offset (opae_uio_device_region)
- region_ptr (opae_uio_device_region, opae_vfio_device_region)
- region_size (opae_uio_device_region, opae_vfio_device_region)
- region_sparse (opae_vfio_device_region)
- rsvd (ras_inject_error)
"},{"location":"opae-code/class_members/#s","title":"s","text":" - segment (_fpga_token_header, opae::fpga::types::properties)
- subsystem_device_id (_fpga_token_header, opae::fpga::types::properties)
- subsystem_vendor_id (_fpga_token_header, opae::fpga::types::properties)
- size (mem_link, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae_vfio_sparse_info)
- socket_id (opae::fpga::types::properties)
- set_ (opae::fpga::types::pvalue)
- setter_t (opae::fpga::types::pvalue)
- shared_buffer (opae::fpga::types::shared_buffer)
- size_t (opae::fpga::types::shared_buffer)
- src_location (opae::fpga::types::src_location)
- sysobject (opae::fpga::types::sysobject)
- sysobject_ (opae::fpga::types::sysobject)
- start (opae_vfio_iova_range)
"},{"location":"opae-code/class_members/#t","title":"t","text":" - token_ (opae::fpga::types::error, opae::fpga::types::handle, opae::fpga::types::sysobject, opae::fpga::types::token)
- type_ (opae::fpga::types::event::type_t, opae::fpga::types::event)
- type_t (opae::fpga::types::event::type_t)
- type (opae::fpga::types::properties, opae::fpga::types::sysobject)
- token (opae::fpga::types::token)
- threshold_name (threshold)
"},{"location":"opae-code/class_members/#u","title":"u","text":" - uint (cache_line)
- upper_c_threshold (metric_threshold)
- upper_nc_threshold (metric_threshold)
- upper_nr_threshold (metric_threshold)
- update (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
"},{"location":"opae-code/class_members/#v","title":"v","text":" - vendor_id (_fpga_token_header, opae::fpga::types::properties)
- value_cleanup (_opae_hash_map)
- value (_opae_hash_map_item, fpga_metric, threshold)
- virt_ (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_members/#w","title":"w","text":" - what (opae::fpga::types::except)
- write_csr32 (opae::fpga::types::handle)
- write_csr512 (opae::fpga::types::handle)
- write_csr64 (opae::fpga::types::handle)
- write (opae::fpga::types::shared_buffer)
- wsid (opae::fpga::types::shared_buffer)
- wsid_ (opae::fpga::types::shared_buffer)
- write64 (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_members/#_1","title":"~","text":" - ~error (opae::fpga::types::error)
- ~event (opae::fpga::types::event)
- ~handle (opae::fpga::types::handle)
- ~properties (opae::fpga::types::properties)
- ~shared_buffer (opae::fpga::types::shared_buffer)
- ~sysobject (opae::fpga::types::sysobject)
- ~token (opae::fpga::types::token)
"},{"location":"opae-code/class_members/#_2","title":"@","text":" - @1 (ras_inject_error)
"},{"location":"opae-code/class_member_functions/","title":"Class Member Functions","text":""},{"location":"opae-code/class_member_functions/#a","title":"a","text":" - allocate (opae::fpga::types::shared_buffer)
- attach (opae::fpga::types::shared_buffer)
- as_string (opae::fpga::types::version)
- as_struct (opae::fpga::types::version)
"},{"location":"opae-code/class_member_functions/#b","title":"b","text":" - busy (opae::fpga::types::busy)
- bytes (opae::fpga::types::sysobject)
- build (opae::fpga::types::version)
"},{"location":"opae-code/class_member_functions/#c","title":"c","text":" - c_type (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
- can_clear (opae::fpga::types::error)
- close (opae::fpga::types::handle)
- compare (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_functions/#e","title":"e","text":" - error (opae::fpga::types::error)
- event (opae::fpga::types::event)
- except (opae::fpga::types::except)
- exception (opae::fpga::types::exception)
- enumerate (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#f","title":"f","text":" - fill (opae::fpga::types::shared_buffer)
- file (opae::fpga::types::src_location)
- fn (opae::fpga::types::src_location)
"},{"location":"opae-code/class_member_functions/#g","title":"g","text":" - get (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::properties, opae::fpga::types::sysobject)
- guid_t (opae::fpga::types::guid_t)
- get_token (opae::fpga::types::handle)
- get_value (opae::fpga::types::pvalue)
- get_parent (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#h","title":"h","text":" - handle (opae::fpga::types::handle)
"},{"location":"opae-code/class_member_functions/#i","title":"i","text":" - invalidate (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- is_set (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- invalid_param (opae::fpga::types::invalid_param)
- io_address (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_functions/#l","title":"l","text":" - line (opae::fpga::types::src_location)
"},{"location":"opae-code/class_member_functions/#m","title":"m","text":" - mmio_ptr (opae::fpga::types::handle)
"},{"location":"opae-code/class_member_functions/#n","title":"n","text":" - name (opae::fpga::types::error)
- no_access (opae::fpga::types::no_access)
- no_daemon (opae::fpga::types::no_daemon)
- no_driver (opae::fpga::types::no_driver)
- no_memory (opae::fpga::types::no_memory)
- not_found (opae::fpga::types::not_found)
- not_supported (opae::fpga::types::not_supported)
"},{"location":"opae-code/class_member_functions/#o","title":"o","text":" - operator= (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::pvalue, opae::fpga::types::shared_buffer, opae::fpga::types::src_location, opae::fpga::types::sysobject)
- operator fpga_event_type (opae::fpga::types::event::type_t)
- operator fpga_event_handle (opae::fpga::types::event)
- os_object (opae::fpga::types::event)
- operator fpga_result (opae::fpga::types::except)
- operator uint8_t * (opae::fpga::types::guid_t)
- operator== (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- open (opae::fpga::types::handle)
- operator fpga_handle (opae::fpga::types::handle)
- operator fpga_properties (opae::fpga::types::properties)
- operator copy_t (opae::fpga::types::pvalue)
- owner (opae::fpga::types::shared_buffer)
- operator fpga_object (opae::fpga::types::sysobject)
- operator fpga_token (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#p","title":"p","text":" - parse (opae::fpga::types::guid_t)
- properties (opae::fpga::types::properties)
- pvalue (opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_functions/#r","title":"r","text":" - read_value (opae::fpga::types::error)
- register_event (opae::fpga::types::event)
- read_csr32 (opae::fpga::types::handle)
- read_csr64 (opae::fpga::types::handle)
- reconfigure (opae::fpga::types::handle)
- reset (opae::fpga::types::handle)
- reconf_error (opae::fpga::types::reconf_error)
- read (opae::fpga::types::shared_buffer)
- release (opae::fpga::types::shared_buffer)
- read64 (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_functions/#s","title":"s","text":" - shared_buffer (opae::fpga::types::shared_buffer)
- size (opae::fpga::types::shared_buffer, opae::fpga::types::sysobject)
- src_location (opae::fpga::types::src_location)
- sysobject (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_functions/#t","title":"t","text":" - type_t (opae::fpga::types::event::type_t)
- type (opae::fpga::types::sysobject)
- token (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#u","title":"u","text":" - update (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_functions/#w","title":"w","text":" - what (opae::fpga::types::except)
- write_csr32 (opae::fpga::types::handle)
- write_csr512 (opae::fpga::types::handle)
- write_csr64 (opae::fpga::types::handle)
- write (opae::fpga::types::shared_buffer)
- wsid (opae::fpga::types::shared_buffer)
- write64 (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_functions/#_1","title":"~","text":" - ~error (opae::fpga::types::error)
- ~event (opae::fpga::types::event)
- ~handle (opae::fpga::types::handle)
- ~properties (opae::fpga::types::properties)
- ~shared_buffer (opae::fpga::types::shared_buffer)
- ~sysobject (opae::fpga::types::sysobject)
- ~token (opae::fpga::types::token)
"},{"location":"opae-code/class_member_variables/","title":"Class Member Variables","text":""},{"location":"opae-code/class_member_variables/#a","title":"a","text":" - allocated (mem_alloc)
- address (mem_link)
- accelerator_state (opae::fpga::types::properties)
"},{"location":"opae-code/class_member_variables/#b","title":"b","text":" - bus (_fpga_token_header, opae::fpga::types::properties)
- buckets (_opae_hash_map)
- buf_ (opae::fpga::types::except)
- bbs_id (opae::fpga::types::properties)
- bbs_version (opae::fpga::types::properties)
- buffer_iova (opae_vfio_buffer)
- buffer_ptr (opae_vfio_buffer)
- buffer_size (opae_vfio_buffer)
"},{"location":"opae-code/class_member_variables/#c","title":"c","text":" - cleanup_context (_opae_hash_map)
- can_clear (fpga_error_info)
- capabilities (opae::fpga::types::properties)
- copy_ (opae::fpga::types::pvalue)
- cont_buffers (opae_vfio)
- cont_device (opae_vfio)
- cont_fd (opae_vfio)
- cont_pciaddr (opae_vfio)
- cont_ranges (opae_vfio)
- count (opae_vfio_device_irq)
- catastrophicr_error (ras_inject_error)
- csr (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#d","title":"d","text":" - device (_fpga_token_header, opae::fpga::types::properties, opae_vfio)
- device_id (_fpga_token_header, opae::fpga::types::properties)
- data_ (opae::fpga::types::guid_t)
- device_fd (opae_uio, opae_vfio_device)
- device_path (opae_uio)
- device_config_offset (opae_vfio_device)
- device_num_irqs (opae_vfio_device)
- device_num_regions (opae_vfio_device)
"},{"location":"opae-code/class_member_variables/#e","title":"e","text":" - error_info_ (opae::fpga::types::error)
- error_num_ (opae::fpga::types::error)
- event_handle_ (opae::fpga::types::event)
- error (opae::fpga::types::event::type_t)
- event_fds (opae_vfio_device_irq)
- end (opae_vfio_iova_range)
"},{"location":"opae-code/class_member_variables/#f","title":"f","text":" - function (_fpga_token_header, opae::fpga::types::properties)
- flags (_opae_hash_map, opae_vfio_buffer, opae_vfio_device_irq)
- free (mem_alloc)
- file_ (opae::fpga::types::src_location)
- fn_ (opae::fpga::types::src_location)
- fatal_error (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#g","title":"g","text":" - guid (_fpga_token_header, opae::fpga::types::properties)
- group_name (fpga_metric_info)
- get_ (opae::fpga::types::pvalue)
- group (opae_vfio)
- group_device (opae_vfio_group)
- group_fd (opae_vfio_group)
"},{"location":"opae-code/class_member_variables/#h","title":"h","text":" - hash_seed (_opae_hash_map)
- hysteresis (metric_threshold)
- handle_ (opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_variables/#i","title":"i","text":" - interface (_fpga_token_header, opae::fpga::types::properties)
- isvalid (fpga_metric)
- interrupt (opae::fpga::types::event::type_t)
- is_set_ (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- io_address_ (opae::fpga::types::shared_buffer)
- iova_alloc (opae_vfio)
- irqs (opae_vfio_device)
- index (opae_vfio_device_irq, opae_vfio_sparse_info)
- is_valid (threshold)
"},{"location":"opae-code/class_member_variables/#k","title":"k","text":" - key_cleanup (_opae_hash_map)
- key_compare (_opae_hash_map)
- key_hash (_opae_hash_map)
- key (_opae_hash_map_item)
"},{"location":"opae-code/class_member_variables/#l","title":"l","text":" - lower_c_threshold (metric_threshold)
- lower_nc_threshold (metric_threshold)
- lower_nr_threshold (metric_threshold)
- loc_ (opae::fpga::types::except)
- local_memory_size (opae::fpga::types::properties)
- len_ (opae::fpga::types::shared_buffer)
- line_ (opae::fpga::types::src_location)
- lock (opae_vfio)
"},{"location":"opae-code/class_member_variables/#m","title":"m","text":" - magic (_fpga_token_header)
- metric_num (fpga_metric, fpga_metric_info)
- metric_datatype (fpga_metric_info)
- metric_guid (fpga_metric_info)
- metric_name (fpga_metric_info, metric_threshold)
- metric_type (fpga_metric_info)
- metric_units (fpga_metric_info)
- major (fpga_version)
- minor (fpga_version)
- MAX_EXCEPT (opae::fpga::types::except)
- msg_ (opae::fpga::types::except)
- model (opae::fpga::types::properties)
- masks (opae_vfio_device_irq)
"},{"location":"opae-code/class_member_variables/#n","title":"n","text":" - num_buckets (_opae_hash_map)
- next (_opae_hash_map_item, mem_link, opae_uio_device_region, opae_vfio_device_irq, opae_vfio_device_region, opae_vfio_iova_range, opae_vfio_sparse_info)
- name (fpga_error_info)
- none (opae::fpga::types::properties)
- num_errors (opae::fpga::types::properties)
- num_interrupts (opae::fpga::types::properties)
- num_mmio (opae::fpga::types::properties)
- num_slots (opae::fpga::types::properties)
- nonfatal_error (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#o","title":"o","text":" - object_id (_fpga_token_header, opae::fpga::types::properties)
- objtype (_fpga_token_header)
- open_flags (config)
- os_object_ (opae::fpga::types::event)
- offset (opae_vfio_sparse_info)
"},{"location":"opae-code/class_member_variables/#p","title":"p","text":" - patch (fpga_version)
- prev (mem_link)
- power_thermal (opae::fpga::types::event::type_t)
- props_ (opae::fpga::types::guid_t, opae::fpga::types::properties, opae::fpga::types::pvalue)
- parent (opae::fpga::types::properties)
- ptr (opae_vfio_sparse_info)
"},{"location":"opae-code/class_member_variables/#q","title":"q","text":" - qualifier_name (fpga_metric_info)
"},{"location":"opae-code/class_member_variables/#r","title":"r","text":" - run_n3000 (config)
- res_ (opae::fpga::types::except)
- regions (opae_uio, opae_vfio_device)
- region_index (opae_uio_device_region, opae_vfio_device_region)
- region_page_offset (opae_uio_device_region)
- region_ptr (opae_uio_device_region, opae_vfio_device_region)
- region_size (opae_uio_device_region, opae_vfio_device_region)
- region_sparse (opae_vfio_device_region)
- rsvd (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#s","title":"s","text":" - segment (_fpga_token_header, opae::fpga::types::properties)
- subsystem_device_id (_fpga_token_header, opae::fpga::types::properties)
- subsystem_vendor_id (_fpga_token_header, opae::fpga::types::properties)
- size (mem_link, opae_vfio_sparse_info)
- socket_id (opae::fpga::types::properties)
- set_ (opae::fpga::types::pvalue)
- sysobject_ (opae::fpga::types::sysobject)
- start (opae_vfio_iova_range)
"},{"location":"opae-code/class_member_variables/#t","title":"t","text":" - token_ (opae::fpga::types::error, opae::fpga::types::handle, opae::fpga::types::sysobject, opae::fpga::types::token)
- type_ (opae::fpga::types::event::type_t, opae::fpga::types::event)
- type (opae::fpga::types::properties)
- threshold_name (threshold)
"},{"location":"opae-code/class_member_variables/#u","title":"u","text":" - uint (cache_line)
- upper_c_threshold (metric_threshold)
- upper_nc_threshold (metric_threshold)
- upper_nr_threshold (metric_threshold)
"},{"location":"opae-code/class_member_variables/#v","title":"v","text":" - vendor_id (_fpga_token_header, opae::fpga::types::properties)
- value_cleanup (_opae_hash_map)
- value (_opae_hash_map_item, fpga_metric, threshold)
- virt_ (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_variables/#w","title":"w","text":" - wsid_ (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_variables/#_1","title":"@","text":" - @1 (ras_inject_error)
"},{"location":"opae-code/class_member_typedefs/","title":"Class Member Typedefs","text":""},{"location":"opae-code/class_member_typedefs/#c","title":"c","text":" - copy_t (opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_typedefs/#g","title":"g","text":" - getter_t (opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_typedefs/#p","title":"p","text":" - ptr_t (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
"},{"location":"opae-code/class_member_typedefs/#s","title":"s","text":" - setter_t (opae::fpga::types::pvalue)
- size_t (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_enums/","title":"Class Member Enums","text":""},{"location":"opae-code/namespace_members/","title":"Namespace Members","text":""},{"location":"opae-code/namespace_members/#a","title":"a","text":" - assert_fpga_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_members/#e","title":"e","text":" - exception_fn (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_members/#i","title":"i","text":" - is_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_members/#o","title":"o","text":" - opae_exceptions (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_functions/","title":"Namespace Member Functions","text":""},{"location":"opae-code/namespace_member_functions/#a","title":"a","text":" - assert_fpga_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_functions/#i","title":"i","text":" - is_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_variables/","title":"Namespace Member Variables","text":""},{"location":"opae-code/namespace_member_variables/#o","title":"o","text":" - opae_exceptions (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_typedefs/","title":"Namespace Member Typedefs","text":""},{"location":"opae-code/namespace_member_typedefs/#e","title":"e","text":" - exception_fn (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_enums/","title":"Namespace Member Enums","text":""},{"location":"opae-code/functions/","title":"Functions","text":""},{"location":"opae-code/functions/#e","title":"e","text":" - error_thread (hello_events.c)
"},{"location":"opae-code/functions/#f","title":"f","text":" - fpgaClose (access.h)
- fpgaOpen (access.h)
- fpgaReset (access.h)
- fpgaGetIOAddress (buffer.h)
- fpgaPrepareBuffer (buffer.h)
- fpgaReleaseBuffer (buffer.h)
- fpgaCloneToken (enum.h)
- fpgaDestroyToken (enum.h)
- fpgaEnumerate (enum.h)
- fpgaClearAllErrors (error.h)
- fpgaClearError (error.h)
- fpgaGetErrorInfo (error.h)
- fpgaReadError (error.h)
- fpgaCreateEventHandle (event.h)
- fpgaDestroyEventHandle (event.h)
- fpgaGetOSObjectFromEventHandle (event.h)
- fpgaRegisterEvent (event.h)
- fpgaUnregisterEvent (event.h)
- fpgaFinalize (init.h)
- fpgaInitialize (init.h)
- fpgaAssignPortToInterface (manage.h)
- fpgaAssignToInterface (manage.h)
- fpgaReconfigureSlot (manage.h)
- fpgaReleaseFromInterface (manage.h)
- fpgaGetMetricsByIndex (metrics.h)
- fpgaGetMetricsByName (metrics.h)
- fpgaGetMetricsInfo (metrics.h)
- fpgaGetMetricsThresholdInfo (metrics.h)
- fpgaGetNumMetrics (metrics.h)
- fpgaMapMMIO (mmio.h)
- fpgaReadMMIO32 (mmio.h)
- fpgaReadMMIO64 (mmio.h)
- fpgaUnmapMMIO (mmio.h)
- fpgaWriteMMIO32 (mmio.h)
- fpgaWriteMMIO512 (mmio.h)
- fpgaWriteMMIO64 (mmio.h)
- fpgaClearProperties (properties.h)
- fpgaCloneProperties (properties.h)
- fpgaDestroyProperties (properties.h)
- fpgaGetProperties (properties.h)
- fpgaGetPropertiesFromHandle (properties.h)
- fpgaPropertiesGetAcceleratorState (properties.h)
- fpgaPropertiesGetBBSID (properties.h)
- fpgaPropertiesGetBBSVersion (properties.h)
- fpgaPropertiesGetBus (properties.h)
- fpgaPropertiesGetCapabilities (properties.h)
- fpgaPropertiesGetDevice (properties.h)
- fpgaPropertiesGetDeviceID (properties.h)
- fpgaPropertiesGetFunction (properties.h)
- fpgaPropertiesGetGUID (properties.h)
- fpgaPropertiesGetInterface (properties.h)
- fpgaPropertiesGetLocalMemorySize (properties.h)
- fpgaPropertiesGetModel (properties.h)
- fpgaPropertiesGetNumErrors (properties.h)
- fpgaPropertiesGetNumInterrupts (properties.h)
- fpgaPropertiesGetNumMMIO (properties.h)
- fpgaPropertiesGetNumSlots (properties.h)
- fpgaPropertiesGetObjectID (properties.h)
- fpgaPropertiesGetObjectType (properties.h)
- fpgaPropertiesGetParent (properties.h)
- fpgaPropertiesGetSegment (properties.h)
- fpgaPropertiesGetSocketID (properties.h)
- fpgaPropertiesGetSubsystemDeviceID (properties.h)
- fpgaPropertiesGetSubsystemVendorID (properties.h)
- fpgaPropertiesGetVendorID (properties.h)
- fpgaPropertiesSetAcceleratorState (properties.h)
- fpgaPropertiesSetBBSID (properties.h)
- fpgaPropertiesSetBBSVersion (properties.h)
- fpgaPropertiesSetBus (properties.h)
- fpgaPropertiesSetCapabilities (properties.h)
- fpgaPropertiesSetDevice (properties.h)
- fpgaPropertiesSetDeviceID (properties.h)
- fpgaPropertiesSetFunction (properties.h)
- fpgaPropertiesSetGUID (properties.h)
- fpgaPropertiesSetInterface (properties.h)
- fpgaPropertiesSetLocalMemorySize (properties.h)
- fpgaPropertiesSetModel (properties.h)
- fpgaPropertiesSetNumErrors (properties.h)
- fpgaPropertiesSetNumInterrupts (properties.h)
- fpgaPropertiesSetNumMMIO (properties.h)
- fpgaPropertiesSetNumSlots (properties.h)
- fpgaPropertiesSetObjectID (properties.h)
- fpgaPropertiesSetObjectType (properties.h)
- fpgaPropertiesSetParent (properties.h)
- fpgaPropertiesSetSegment (properties.h)
- fpgaPropertiesSetSocketID (properties.h)
- fpgaPropertiesSetSubsystemDeviceID (properties.h)
- fpgaPropertiesSetSubsystemVendorID (properties.h)
- fpgaPropertiesSetVendorID (properties.h)
- fpgaUpdateProperties (properties.h)
- fpgaDestroyObject (sysobject.h)
- fpgaHandleGetObject (sysobject.h)
- fpgaObjectGetObject (sysobject.h)
- fpgaObjectGetObjectAt (sysobject.h)
- fpgaObjectGetSize (sysobject.h)
- fpgaObjectGetType (sysobject.h)
- fpgaObjectRead (sysobject.h)
- fpgaObjectRead64 (sysobject.h)
- fpgaObjectWrite64 (sysobject.h)
- fpgaTokenGetObject (sysobject.h)
- fpgaGetNumUmsg (umsg.h)
- fpgaGetUmsgPtr (umsg.h)
- fpgaSetUmsgAttributes (umsg.h)
- fpgaTriggerUmsg (umsg.h)
- fpgaGetUserClock (userclk.h)
- fpgaSetUserClock (userclk.h)
- fpgaErrStr (utils.h)
- fpgaGetOPAECBuildString (version.h)
- fpgaGetOPAECVersion (version.h)
- fpgaGetOPAECVersionString (version.h)
- find_fpga (hello_events.c, hello_fpga.c)
- find_nlb_n3000 (hello_fpga.c)
"},{"location":"opae-code/functions/#h","title":"h","text":" - help (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/functions/#i","title":"i","text":" - inject_ras_fatal_error (hello_events.c)
"},{"location":"opae-code/functions/#m","title":"m","text":" - mem_alloc_add_free (mem_alloc.h)
- mem_alloc_destroy (mem_alloc.h)
- mem_alloc_get (mem_alloc.h)
- mem_alloc_init (mem_alloc.h)
- mem_alloc_put (mem_alloc.h)
- main (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/functions/#o","title":"o","text":" - opae_hash_map_add (hash_map.h)
- opae_hash_map_destroy (hash_map.h)
- opae_hash_map_find (hash_map.h)
- opae_hash_map_init (hash_map.h)
- opae_hash_map_is_empty (hash_map.h)
- opae_hash_map_remove (hash_map.h)
- opae_u64_key_compare (hash_map.h)
- opae_u64_key_hash (hash_map.h)
- opae_print (log.h)
- opae_uio_close (uio.h)
- opae_uio_open (uio.h)
- opae_uio_region_get (uio.h)
- opae_vfio_buffer_allocate (vfio.h)
- opae_vfio_buffer_allocate_ex (vfio.h)
- opae_vfio_buffer_free (vfio.h)
- opae_vfio_buffer_info (vfio.h)
- opae_vfio_close (vfio.h)
- opae_vfio_irq_disable (vfio.h)
- opae_vfio_irq_enable (vfio.h)
- opae_vfio_irq_mask (vfio.h)
- opae_vfio_irq_unmask (vfio.h)
- opae_vfio_open (vfio.h)
- opae_vfio_region_get (vfio.h)
- opae_vfio_secure_open (vfio.h)
"},{"location":"opae-code/functions/#p","title":"p","text":" - parse_args (hello_events.c, hello_fpga.c)
- print_err (hello_events.c, hello_fpga.c)
- probe_for_ase (hello_fpga.c)
"},{"location":"opae-code/functions/#u","title":"u","text":" - usleep (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/macros/","title":"Macros","text":""},{"location":"opae-code/macros/#a","title":"a","text":" - ASSERT_FPGA_OK (except.h)
"},{"location":"opae-code/macros/#c","title":"c","text":" - CACHELINE_ALIGNED_ADDR (hello_fpga.c)
- CL (hello_fpga.c)
- CSR_AFU_DSM_BASEL (hello_fpga.c)
- CSR_CFG (hello_fpga.c)
- CSR_CTL (hello_fpga.c)
- CSR_DST_ADDR (hello_fpga.c)
- CSR_NUM_LINES (hello_fpga.c)
- CSR_SRC_ADDR (hello_fpga.c)
- CSR_STATUS1 (hello_fpga.c)
"},{"location":"opae-code/macros/#d","title":"d","text":" - DSM_STATUS_TEST_COMPLETE (hello_fpga.c)
"},{"location":"opae-code/macros/#f","title":"f","text":" - FPGA_ERROR_NAME_MAX (types.h)
- FPGA_METRIC_STR_SIZE (types.h)
- fpga_is_parent_child (types.h)
- FPGA_BUILD_STR_MAX (version.h)
- FPGA_VERSION_STR_MAX (version.h)
- FME_SYSFS_INJECT_ERROR (hello_events.c)
- FPGA_NLB0_UUID_H (hello_fpga.c)
- FPGA_NLB0_UUID_L (hello_fpga.c)
"},{"location":"opae-code/macros/#g","title":"g","text":" - GETOPT_STRING (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/macros/#l","title":"l","text":" - LOG2_CL (hello_fpga.c)
- LPBK1_BUFFER_ALLOCATION_SIZE (hello_fpga.c)
- LPBK1_BUFFER_SIZE (hello_fpga.c)
- LPBK1_DSM_SIZE (hello_fpga.c)
"},{"location":"opae-code/macros/#m","title":"m","text":" - MB (hello_fpga.c)
"},{"location":"opae-code/macros/#n","title":"n","text":" - N3000_AFUID (hello_fpga.c)
- NLB0_AFUID (hello_fpga.c)
"},{"location":"opae-code/macros/#o","title":"o","text":" - OPAECXX_HERE (except.h)
- OPAE_DBG (log.h)
- OPAE_DEFAULT_LOGLEVEL (log.h)
- OPAE_ERR (log.h)
- OPAE_MSG (log.h)
- OPAE_UIO_PATH_MAX (uio.h)
- ON_ERR_GOTO (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/macros/#t","title":"t","text":" - TEST_TIMEOUT (hello_fpga.c)
"},{"location":"opae-code/macros/#_","title":"_","text":" - __SHORT_FILE__ (log.h)
"},{"location":"opae-code/variables/","title":"Variables","text":""},{"location":"opae-code/variables/#c","title":"c","text":" - config (hello_fpga.c)
"},{"location":"opae-code/variables/#f","title":"f","text":" - fpga_event_handle (types.h)
- fpga_guid (types.h)
- fpga_handle (types.h)
- fpga_metric (types.h)
- fpga_metric_info (types.h)
- fpga_object (types.h)
- fpga_properties (types.h)
- fpga_token (types.h)
- fpga_token_header (types.h)
- fpga_accelerator_state (types_enum.h)
- fpga_buffer_flags (types_enum.h)
- fpga_event_type (types_enum.h)
- fpga_interface (types_enum.h)
- fpga_metric_datatype (types_enum.h)
- fpga_metric_type (types_enum.h)
- fpga_objtype (types_enum.h)
- fpga_open_flags (types_enum.h)
- fpga_reconf_flags (types_enum.h)
- fpga_result (types_enum.h)
- fpga_sysobject_flags (types_enum.h)
- fpga_sysobject_type (types_enum.h)
"},{"location":"opae-code/variables/#m","title":"m","text":" - metric_threshold (types.h)
"},{"location":"opae-code/variables/#o","title":"o","text":" - opae_hash_map (hash_map.h)
- opae_hash_map_flags (hash_map.h)
- opae_hash_map_item (hash_map.h)
- opae_loglevel (log.h)
- opae_vfio_buffer_flags (vfio.h)
"},{"location":"opae-code/variables/#t","title":"t","text":" - threshold (types.h)
"},{"location":"opae-code/variables/#_","title":"_","text":" - _opae_hash_map_flags (hash_map.h)
"},{"location":"opae-code/links/","title":"Links","text":"* Related Pages * Todo List * Modules * Class List * struct _fpga_token_header * struct _opae_hash_map * struct _opae_hash_map_item * struct cache_line * struct config * struct fpga_error_info * struct fpga_metric * struct fpga_metric_info * struct fpga_version * struct mem_alloc * struct mem_link * struct metric_threshold * union metric_value * namespace opae * namespace opae::fpga * namespace opae::fpga::types * class opae::fpga::types::busy * namespace opae::fpga::types::detail * class opae::fpga::types::error * class opae::fpga::types::event * struct opae::fpga::types::event::type_t * class opae::fpga::types::except * class opae::fpga::types::exception * struct opae::fpga::types::guid_t * class opae::fpga::types::handle * class opae::fpga::types::invalid_param * class opae::fpga::types::no_access * class opae::fpga::types::no_daemon * class opae::fpga::types::no_driver * class opae::fpga::types::no_memory * class opae::fpga::types::not_found * class opae::fpga::types::not_supported * class opae::fpga::types::properties * struct opae::fpga::types::pvalue * class opae::fpga::types::reconf_error * class opae::fpga::types::shared_buffer * class opae::fpga::types::src_location * class opae::fpga::types::sysobject * class opae::fpga::types::token * class opae::fpga::types::version * struct opae_uio * struct opae_uio_device_region * struct opae_vfio * struct opae_vfio_buffer * struct opae_vfio_device * struct opae_vfio_device_irq * struct opae_vfio_device_region * struct opae_vfio_group * struct opae_vfio_iova_range * struct opae_vfio_sparse_info * struct ras_inject_error * namespace std * struct threshold * Namespace ListNamespace List * Namespace Members * Namespace Member Functions * Namespace Member Variables * Namespace Member Typedefs * Namespace Member Enumerations * Class Index * Class Hierarchy * Class Members * Class Member Functions * Class Member Variables * Class Member Typedefs * Class Member Enumerations * Files * docs * docs/sw * docs/sw/include * docs/sw/include/opae * access.h * access.h source * buffer.h * buffer.h source * docs/sw/include/opae/cxx * core.h * core.h source * docs/sw/include/opae/cxx/core * errors.h * errors.h source * events.h * events.h source * except.h * except.h source * handle.h * handle.h source * properties.h * properties.h source * pvalue.h * pvalue.h source * shared_buffer.h * shared_buffer.h source * sysobject.h * sysobject.h source * token.h * token.h source * version.h * version.h source * enum.h * enum.h source * error.h * error.h source * event.h * event.h source * fpga.h * fpga.h source * hash_map.h * hash_map.h source * init.h * init.h source * log.h * log.h source * manage.h * manage.h source * mem_alloc.h * mem_alloc.h source * metrics.h * metrics.h source * mmio.h * mmio.h source * properties.h * properties.h source * sysobject.h * sysobject.h source * types.h * types.h source * types_enum.h * types_enum.h source * uio.h * uio.h source * umsg.h * umsg.h source * userclk.h * userclk.h source * utils.h * utils.h source * version.h * version.h source * vfio.h * vfio.h source * docs/sw/samples * docs/sw/samples/hello_events * hello_events.c * hello_events.c source * docs/sw/samples/hello_fpga * hello_fpga.c * hello_fpga.c source * File Variables * File Functions * File Macros
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Open FPGA Stack Overview","text":"
Open FPGA Stack (OFS): OFS is an open-source hardware and software framework that reduces the development time for creating your custom platform. Provided within this framework are reference designs targeting different Altera FPGA devices with upstreamed drivers and management software tools.
The reference shells, called FPGA Interface Manager (FIMs), provide an integrated, timing closed design with the most common interfaces for host attach applications. After selecting your starting shell, you can add or subtract interfaces depending on your application requirements. Then leverage the build scripts, RTL, unit tests, Universal Verification Methodology (UVM) environment, software and drivers for this reference shell as a starting point for your own FPGA platform solution.
OFS currently targets Stratix\u00ae 10 and Agilex\u00ae 7 FPGA Device Families. To find out more about Altera FPGAs, visit the Stratix 10 and Agilex 7 pages at Intel.com.
"},{"location":"#how-can-i-start-using-ofs","title":"How Can I Start Using OFS?","text":" -
If you are interested in a production card that uses OFS for your workload application development or for full deployment, please refer to the OFS Board Catalog.
-
If you are an FPGA developer, refer to our FPGA Developer Journey Guide: Open FPGA Stack to understand the OFS development flow as well as the reference shells, tools and development boards you can use to gest started. FPGA Developers interested in oneAPI should reference this guide as well.
-
If you are a software developer, refer to our Software Tab for driver and user space tool development resources.
-
If you are an application developer, preview our overview video on how OFS can help you develop FPGA-based workloads and review one of the AFU Developer Guides to find the OFS resources available for creating your own application workload.
- Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs
- Workload Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs
Beyond the resources we have on this site, you can navigate to the OFS GitHub repos by clicking the GitHub repository icon at the top left corner of this site page.
"},{"location":"#what-fim-reference-designs-are-available","title":"What FIM Reference Designs Are Available?","text":"
Below summarizes the five current reference designs (aka FIMs) you can choose from:
OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xR-tile, F-tile)
Key Feature Description Target OPN AGIB027R29A1E2VR3 PCIe R-tile PCIe* Gen5x16, 2xGen5x8, Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory * Four Fabric DDR4 channels, x64 (no ECC), 2666 MHz, 8GB Ethernet 2x4x25GbE, 2x200GbE, 2x400GbE Hard Processor System Not enabled Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xF-tile)
Key Feature Description Target OPN AGFB027R24C2E2VR2 PCIe F-tile PCIe* Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 3 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 2400 MHz, 1GB each* Two Fabric DDR4 banks, x64 (no ECC), 2400 MHz, 8GB Ethernet 2x4x25GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
OFS FIM Targeting Agilex\u00ae7 PCIe Attach (P-tile, E-tile)
Key Feature Description Target OPN AGFB014R24A2E2V PCIe P-tile PCIe* Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 5 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each* Four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB Ethernet 2x4x25GbE, 2x4x10GbE or 2x100GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration)* Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Intel\u00ae FPGA SmartNIC N6001-PL
Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
OFS FIM Features Targeting Agilex\u00ae 7 SoC Attach
Key Feature Description Device OPN AGFC023R25A2E2VR0 PCIe P-tile PCIe* Gen4x16 to the HostP-tile PCIe* Gen4x16 to the SoC (IceLake-D) Virtualization Host: 2 physical functions SoC: 1 physical function and 3 Virtual functions Memory Four Fabric DDR4 banks, x40 (optional ECC, be configured as x32 and ECC x8 ), 1200 MHz, 4GB Ethernet 2x4x25GbE Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) * Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported Software Support * Linux DFL drivers targeting OFS FIMs * OPAE Software Development Kit * OPAE Tools Target Board Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL
Note: Source code for BMC RTL/Firmware that works with this reference FIM can be obtained by contacting your Altera Sales Representative.
Click here for the OFS Collateral for Agilex\u00ae SoC Attach Reference FIM documentation collection.
OFS FIM Targeting Stratix\u00ae 10 FPGA PCIe Attach
Key Feature Description Device OPN 1SX280HN2F43E2VG Ethernet Configuration 1x10GbE example with 2x100GbE capability PCIe Gen3x16 EMIF Up to four DDR channels PF/VF 1 PF/3 VFs Management FPGA Management Engine (FME) with FIM management registers Interface Arm\u00ae AMBA\u00ae4 AXI Interface HLD support oneAPI Software Kernel code upstreamed to Linux.org Target Board Intel\u00ae FPGA Programmable Acceleration Card D5005
Click here for the OFS Collateral for Stratix\u00ae 10 FPGA PCIe Attach Reference FIM) documentation.
To find information on the latest releases, go to the Discussions Tab in the OFS GitHub repository.
"},{"location":"#open-fpga-stack-repositories","title":"Open FPGA Stack Repositories","text":"Accessing OFS ingredients to use within the development framework is easy. The github.com/OFS site provides all the hardware and software repositories in one location.
Development Focus Repository Folder Description Hardware ofs-agx7-pcie-attach Provides RTL, unit tests, and build scripts to create an Agilex\u00ae 7 FIM and is leveraged as a starting point for a custom PCIe Attach design. The provided reference FIM can target the following boards:\u00a0\u00a0\u00a0\u00a0\u2022 Intel\u00ae FPGA SmartNIC N6001-PL Platform \u00a0\u00a0\u00a0\u00a0\u2022 Agilex 7 FPGA F-Series Development Kit (2x F-Tile)\u00a0\u00a0\u00a0\u00a0\u2022 Agilex 7 FPGA I-Series Development Kit (2 x R-Tile, 1 x F-Tile) Hardware ofs-f2000x-pl Provides RTL, unit tests, and build scripts to create Agilex\u00ae FIM and is leveraged as a starting point for a custom SoC Attach design. The reference FIM targets an Intel\u00ae FPGA IPU F2000X-PL Platform. Hardware ofs-d5005 Provides RTL, unit tests, and build scripts to create Stratix 10\u00ae FIM and is leveraged as a starting point for a custom PCIe Attach design. The reference FIM targets an Intel\u00ae FPGA PAC D5005 development board. Hardware oneapi-asp Contains the files to generate the support package that works with the reference shells and allows you to use OneAPI. This is an optional repository for developers interested in OneAPI Hardware ofs-fim-common Provides RTL components that are shared among all new platforms that are introduced in OFS. This folder is a subumodule in each platform repository folder. Hardware examples-afu Provides simple Accelerator Functional Unit (AFU) examples you can use as a template for starting your own workload design. Hardware ofs-platform-afu-bbb Contains the hardware code to build a standard interface between the FIM and your workload. Software linux-dfl This repository is a mirror of the linux.org Git site and contains the most up-to-date drivers that are being developed and upstreamed for OFS platforms. Software meta-ofs This repository provides the Linux\u00ae DFL kernel and the OPAE SDK for the Yocto\u00ae Project. Software opae-sdk Contains the ingredients to build the OFS Open Programmable Acceleration Engine (OPAE) Software Development Kit which provides APIs and userspace tools for OFS FPGA management. Software opae-sim This repository is used to build the AFU Hardware/Software Co-Simulation Environment workload developers can use to ensure their AFU can work with the OFS software stack. Software linux-dfl-backport A place for finding and leveraging out-of-tree backported drivers for older OS versions . Software opae-legacy Supports OFS platforms built on the legacy version of OPAE software. Not used in current OFS designs Documentation ofs.github.io Contains the hardware and software collateral that surfaces on the OFS website: https://ofs.github.io "},{"location":"d5005/adp_board_installation_guidelines/","title":"Board Installation Guidelines: Intel\u00ae FPGA SmartNIC N6000/1-PL, Intel\u00ae FPGA PAC D5005","text":"Last updated: November 19, 2024
"},{"location":"d5005/adp_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"d5005/adp_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup - the Intel\u00ae FPGA SmartNIC N6000/1-PL and the Intel\u00ae FPGA PAC D5005. After reading the document a user shall be able to:
- Safely install and remove an ADP
- Set up their server BIOS with the recommended settings
- Learn about thermal cooling requirements for their platform
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to all three platforms.
"},{"location":"d5005/adp_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported ADP platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"d5005/adp_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"d5005/adp_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"d5005/adp_board_installation_guidelines/#table-2-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 2: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platforms you have if you are unsure. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers from sections, whose installation is covered in the [Software Installation Guide: OFS for PCIe Attach FPGAs].
SKU Mapping SKU Value Primary Difference fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table provides a picture reference for the hardware components discussed in the rest of the document.
"},{"location":"d5005/adp_board_installation_guidelines/#table-3-hardware-bkc","title":"Table 3: Hardware BKC","text":"Component Image Intel\u00ae FPGA SmartNIC N6001-PL (SKU2) Supermicro Server SYS-220HE Intel FPGA Download Cable II (Only Required for manual flashing) 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing) In addition to the above, all OFS ADP platforms require an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector. This cable will differ between server vendors - review the pinout of the power connector on the Intel\u00ae FPGA Programmable Acceleration Card D5005 Data Sheet or Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based ADP.
"},{"location":"d5005/adp_board_installation_guidelines/#20-initial-server-setup","title":"2.0 Initial Server Setup","text":""},{"location":"d5005/adp_board_installation_guidelines/#21-server-information-for-intel-fpga-smartnic-n60001-pl","title":"2.1 Server Information for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Both the server BIOS and BMC need to match the versions listed below in Table 4: Supermicro Server BMC BKC. These updates only apply for this specific Best Known Configuration (BKC) - other server manufacturers may require different BIOS updates. Please consult your server's user guide and release notes for update information.
"},{"location":"d5005/adp_board_installation_guidelines/#table-4-supermicro-server-bmc-bkc","title":"Table 4: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4) Information about the server\u2019s currently loaded firmware can be found on the BMC web portal dashboard. Accessing this page requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d. During boot the BMC\u2019s login IP will be presented on the screen.
Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case. It is recommended the user change their BMC\u2019s default username as soon as they are able.
After logging in you should be able to review information about the BMC and BIOS by referring to the System box, visible upon initial loading of the page. Double check that the values match those in Table 4. If they do not, you may download the appropriate versions from the SuperMicro product page by selecting the BIOS option and downloading the most recent \u201cBundled Software File Name\u201d. Follow the BMC and BIOS update instructions included in the SuperMicro manuals page in the document X12/H12 BMC Manual in Appendix A.2 Updating Firmware Using BMC Web GUI.
If using a different server model, refer to that server\u2019s user guide for instructions on remote system management. Ensure that any system you end up using meets all the following requirements:
- Main Board: PCI Express 3.0 (D5005) or 4.0 (N6000/1) compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
"},{"location":"d5005/adp_board_installation_guidelines/#22-server-information-for-intel-fpga-pac-d5005","title":"2.2 Server Information for Intel\u00ae FPGA PAC D5005","text":"Refer to sections 2.1-2.3 of the Intel Acceleration Stack Quick Start Guide: Intel FPGA Programmable Acceleration Card D5005 for a complete overview of the physical installation process and ESD precautions for the D5005 platform.
Ensure that the system meets all the following requirements before proceeding to install the Intel\u00ae FPGA PAC D5005 into a server.
- Main Board: PCI Express 3.0 compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
Detailed mechanical for information can be found on the D5005 Data Sheet and in section 4.0 Mechanical Information of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837).
"},{"location":"d5005/adp_board_installation_guidelines/#30-server-settings","title":"3.0 Server Settings","text":""},{"location":"d5005/adp_board_installation_guidelines/#31-bios-settings","title":"3.1 BIOS Settings","text":"You must enable Intel VT-x/VT-d technologies for the PCIe slot housing your ADP. The following steps are known to work on a SuperMicro SYS-220HE server platform.
-
To enter the Supermicro server\u2019s BIOS setup page, reboot, and press \\<Delete> when prompted. You can browse the tabs / options with a combination of arrow keys along with \\<Escape> and \\<Enter>.
-
Navigate right to the Advanced tab, then select the following menu options: Chipset Configuration -> North Bridge -> IIO Configuration -> Intel VT for Directed I/O (VT-d).
-
If not already, enable the option Intel VT for Directed I/O (VT-d).
"},{"location":"d5005/adp_board_installation_guidelines/#31-server-fan-speed","title":"3.1 Server Fan Speed","text":"The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
- Log in to the SuperMicro server BMC. (This requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d.)
- During boot the BMC\u2019s login IP will be presented on the screen. Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case.
- On the left menu select System -> Component Info, select the Fan tab, under Advanced Settings click the circle next to Full Speed.
"},{"location":"d5005/adp_board_installation_guidelines/#32-cooling-requirements","title":"3.2 Cooling Requirements","text":"Please refer to sections 8.1 and 8.2 of the Intel FPGA Programmable Acceleration Card D5005 Data Sheet or section 6.0 of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) for guidance on cooling specifications that must be met when using these platforms. Failure to adhere to these guidelines may result in thermal runaway and/or performance degradation.
"},{"location":"d5005/adp_board_installation_guidelines/#40-board-installation-procedure","title":"4.0 Board Installation Procedure","text":""},{"location":"d5005/adp_board_installation_guidelines/#41-pcie-slot-mappings-for-intel-fpga-smartnic-n60001-pl","title":"4.1 PCIe Slot Mappings for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"The Intel N6000/1-PL FPGA SmartNIC Platforms are officially verified in the upper middle PCIe x16 slot (Slot 3). If using a different slot, refer to the information in Table 5 PCIe Slot Mapping for which port settings to change in server BIOS.
"},{"location":"d5005/adp_board_installation_guidelines/#table-5-pcie-slot-mapping","title":"Table 5: PCIe Slot Mapping","text":"CPU Number Port Number (in BIOS) PCIe Slot CPU1 Port 2 5 and 6 CPU1 Port 4 7 and 8 CPU2 Port 2 1 and 2 CPU2 Port 4 3 and 4"},{"location":"d5005/adp_board_installation_guidelines/#42-installation-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.2 Installation Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe installation of an ADP platform into a supported server. While an Intel\u00ae FPGA SmartNIC N6001-PL is shown in the images below, this procedure applies to all three platforms.
- Position the board over the selected connector on the motherboard.
- Press down gently and firmly to seat the card in the PCIe slot, and then secure the bracket to the system chassis with the retention screw.
"},{"location":"d5005/adp_board_installation_guidelines/#table-6-adp-installation-procedure","title":"Table 6: ADP Installation Procedure","text":"Callout Description 1 Retention screw 2 Press down here gently 3 Press down here gently 4 Motherboard Do not bend the card while inserting into a slot. Do not apply much pressure in regions 2 or 3 while inserting.
"},{"location":"d5005/adp_board_installation_guidelines/#43-removal-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.3 Removal Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe removal of the platforms from a supported server.
- Disconnect all power cords from the server power supply(s).
- Remove the retention bracket screw.
- Carefully lift the card out of the PCIe slot.
"},{"location":"d5005/adp_board_installation_guidelines/#table-7-adp-removal-procedure","title":"Table 7: ADP Removal Procedure","text":"Callout Description 1 Retention screw 2 Pull up here gently 3 Motherboard Do not bend the card while removing it from the slot.
"},{"location":"d5005/sw_install_pcie_attach/","title":"Software Installation Guide: Open FPGA Stack for PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"d5005/sw_install_pcie_attach/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the PCIe Attach software stack on the host. This document will not cover the process of board installation or platform bring-up. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Build and install the OPAE Software Development Kit (SDK) on the host
- Build and install the Linux DFL driver stack on the host
"},{"location":"d5005/sw_install_pcie_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating a PCIe Attach shell. The PCIe Attach shell design is supported on a number of board offerings, including the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile), Intel\u00ae FPGA SmartNIC N6000/1-PL, and Intel\u00ae FPGA PAC D5005.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"d5005/sw_install_pcie_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"d5005/sw_install_pcie_attach/#table-2-software-and-component-version-summary-for-ofs-pcie-attach","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach","text":"The OFS PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are where the source code resides for each of the components discussed in this document.
Component Version Download Link Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 link OPAE SDK 2.13.0-3 2.13.0-3 Linux DFL intel-1.11.0-2 intel-1.11.0-2"},{"location":"d5005/sw_install_pcie_attach/#table-3-release-pages-for-each-pcie-attach-platform","title":"Table 3: Release Page(s) for each PCIe Attach Platform","text":"This is a comprehensive list of the platform(s) whose software build and installation steps are covered in this document.
Platform Release Page Link Stratix\u00ae 10 FPGA https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Intel\u00ae FPGA SmartNIC N6001-PL https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1"},{"location":"d5005/sw_install_pcie_attach/#12-server-requirements","title":"1.2 Server Requirements","text":""},{"location":"d5005/sw_install_pcie_attach/#121-host-bios","title":"1.2.1 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
- Intel VT for Directed I/O (VT-d) must be enabled
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"d5005/sw_install_pcie_attach/#122-host-server-kernel-and-grub-configuration","title":"1.2.2 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
- OS: RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10
- Kernel: 4.18.0-dfl
"},{"location":"d5005/sw_install_pcie_attach/#20-ofs-software-overview","title":"2.0 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack, on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"d5005/sw_install_pcie_attach/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions, and currently only supported the PCIe Attach release. Its usage is detailed in the relevant Quick Start Demonstration Guideline for your platform and will not be covered here.
"},{"location":"d5005/sw_install_pcie_attach/#31-ofs-dfl-backport-kernel-driver-installation-environment-setup","title":"3.1 OFS DFL Backport Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL Backport GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.2.2 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 3.3 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
-
It is recommended you lock your Red Hat release version to 8.10 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
-
Install the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n\nsudo dnf install kernel-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-headers-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-devel-4.18.0-553.5.1.el8_10.x86_64\n
Now you have the choice to either follow the steps in section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source or 3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages.
"},{"location":"d5005/sw_install_pcie_attach/#32-building-and-installing-the-ofs-dfl-backport-kernel-drivers-from-source","title":"3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Select the 4.18.0-553.5.1.el8_10 kernel as your next boot target on the build machine, then reboot.
# For multiple boots (until overwritten), use the following\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\n# Or select the kernel you want during boot time\nsudo reboot\n
1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n
2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nintel-1.11.0-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n
3. Build the kernel.
```bash\ncd /home/OFS/linux-dfl-backport\nmake && make rpm\n```\n
4. Install the newly compiled RPM package and reboot.
```bash\nsudo rpm -i intel-fpga-dfl-dkms-1.11.0-2.2024.07.25.g276007e.noarch.rpm\n\nsudo reboot\n```\n
5. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/4.18.0-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\n
If an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n
6. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n
7. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n
8. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n
9. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-4.18.0-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\n
A list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"d5005/sw_install_pcie_attach/#33-installing-the-ofs-dfl-backport-kernel-drivers-from-pre-built-packages","title":"3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page under the Artifacts tab. The name will resemble kernel-*.tar.gz. For the backport repository you can also choose to download packages under the GitHub action Build dkms packages, although your version will differ from the release version of the software.
- Download and install the pre-built kernel package.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\ntar xf kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\nsudo rpm -i kernel-4.18.0_dfl_2024.06.14_276007e/intel-fpga-dfl-dkms-1.11.0-2.2024.06.14.g276007e.noarch.rpm\n\n# Set this kernel as your new default boot target\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\nsudo reboot\n
Continue from step 5 of Section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source.
"},{"location":"d5005/sw_install_pcie_attach/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 4.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"d5005/sw_install_pcie_attach/#41-opae-sdk-installation-environment-setup","title":"4.1 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"d5005/sw_install_pcie_attach/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package. -
Remove any currently installed OPAE packages.
sudo dnf remove opae*\n
-
Initialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\n
-
Verify that the correct tag/branch have been checkout out.
git describe --tags\n2.13.0-3\n
-
Set up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\n
The following packages will be built in the same directory as create:
-
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\n
-
Check that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\n
"},{"location":"d5005/sw_install_pcie_attach/#42-installing-the-opae-sdk-with-pre-built-packages","title":"4.2 Installing the OPAE SDK with Pre-Built Packages","text":"You can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.13.0-3.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.13.0-3.x86_64-*.tar.gz\n
For a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\n
"},{"location":"d5005/sw_install_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"d5005/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\n
"},{"location":"d5005/ug_dev_afu_host_software/#11-find-and-connect-to-afu","title":"1.1. Find and connect to AFU","text":"Here is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\n
In main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\n
"},{"location":"d5005/ug_dev_afu_host_software/#12-map-mmio-optional","title":"1.2. Map MMIO (optional)","text":"Mapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\n
The below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\n
"},{"location":"d5005/ug_dev_afu_host_software/#13-allocate-shared-memory-buffers","title":"1.3. Allocate Shared Memory Buffers","text":"The below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\n
In main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\n
"},{"location":"d5005/ug_dev_afu_host_software/#14-perform-acceleration","title":"1.4. Perform Acceleration","text":"The host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\n
"},{"location":"d5005/ug_dev_afu_host_software/#15-cleanup","title":"1.5. Cleanup","text":"When the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\n
"},{"location":"d5005/ug_dev_afu_host_software/#2-building-the-host-application","title":"2. Building the Host Application","text":"A Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
- Path to common_include.mk (from examples-afu)
- TEST name
- Source files: SRCS
- Path to .json file (relative to Makefile directory)
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\n
"},{"location":"d5005/ug_dev_afu_host_software/#3-running-the-host-application","title":"3. Running the Host Application","text":"To run the host application, you will need to:
- Load AFU onto the FIM
- Create VF's
- Bind VF's using the OPAE Drivers
- Run application
See the associated AFU Developer Guide for details.
"},{"location":"d5005/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
- C/C++
- Verilog/SystemVerilog
- RTL simulators such as Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"d5005/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
- Verilog
- SystemVerilog
- VHDL
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"d5005/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
- Software: OPAE API implemented in the C programming language.
- Hardware: PCIe SS TLP specification implemented in SystemVerilog.
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"d5005/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":" - The ASE provides a protocol checker to ensure protocol correctness. The ASE also provides methods to identify potential issues early, before in-system deployment.
- The ASE can help identify certain lock conditions and Configuration and Status Registers (CSR) address mapping and pointer math errors.
- The ASE tracks memory requested from the accelerator. The memory model immediately flags illegal memory transactions to locations outside of requested memory spaces. Consequently, you can fix incorrect memory accesses early, during the simulation phase.
- The ASE does not guarantee that you can synthesize an AFU. After you verify the AFU RTL functionality in the ASE, use the ASE and the Quartus\u00ae Prime Pro Edition software iteratively to generate the Accelerator Function (AF).
- The ASE does not require administrator privileges. After installing all the required tools, you can run the ASE on a plain vanilla user Linux machine.
"},{"location":"d5005/ug_dev_afu_sim_env/#23-ase-limitations","title":"2.3. ASE Limitations","text":"When using ASE in the application development cycle, consider the following limitations:
- The ASE is a transaction-level simulator. It does not model either Intel UPI- or PCIe-specific packet structures and protocol layers.
- The ASE does not simulate caching and is not a cache simulator. It cannot reliably simulate cache collisions or capacity issues.
- Although ASE models some latency parameters, it cannot model real-time system-specific latency. It is also not an accurate timing simulation of the design or latency and bandwidth of the real system. The ASE models enable you to develop functionally correct accelerators.
- The ASE does not simulate multi-AFU or multi-socket configurations.
"},{"location":"d5005/ug_dev_afu_sim_env/#24-ase-based-afu-design-workflow","title":"2.4 ASE-Based AFU Design Workflow","text":"AFU development using the ASE includes the following four stages:
-
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
-
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
-
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
-
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
- The AFU RTL code is synthesizable.
- The AFU RTL code meets timing.
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"d5005/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
- C-Compiler: gcc 8.5.0 or above
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
- CMake: version 3.15 or above
- Python: version 3.6.8 or above
- Intel Quartus Prime Pro 24.1: The ASE must find the
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.
The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
- Compilation of the SystemVerilog Direct Programming Interface (DPI) constructs
- Compilation of the standard examples that are included in the installation
- Support for SystemC
"},{"location":"d5005/ug_dev_afu_sim_env/#4-package-description","title":"4. Package Description","text":"The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\n
This directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.
"},{"location":"d5005/ug_dev_afu_sim_env/#41-ase-scripts","title":"4.1. ASE Scripts","text":"The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
"},{"location":"d5005/ug_dev_afu_sim_env/#411-simulation-tool-set-up","title":"4.1.1. Simulation Tool Set Up","text":"Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
"},{"location":"d5005/ug_dev_afu_sim_env/#412-ase-environment-check","title":"4.1.2. ASE Environment Check","text":"This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\n
"},{"location":"d5005/ug_dev_afu_sim_env/#413-afu-simulation-using-the-ase","title":"4.1.3. AFU Simulation Using the ASE","text":"Before configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
"},{"location":"d5005/ug_dev_afu_sim_env/#4131-afu_sim_setup","title":"4.1.3.1. afu_sim_setup","text":"The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\n
* The only required argument to the afu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.
afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.
generate_ase_environment.py: This script instantiates your simulated platform configuration.
"},{"location":"d5005/ug_dev_afu_sim_env/#4132-rtl_src_configpy","title":"4.1.3.2. rtl_src_config.py","text":"The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
"},{"location":"d5005/ug_dev_afu_sim_env/#4133-generate_ase_environmentpy","title":"4.1.3.3. generate_ase_environment.py","text":"The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
- Synopsys: Creates
synopsys_sim.setup and vcs_run.tcl in the configuration directory. - Siemens: Creates
vsim_run.tcl in the configuration directory.
The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"d5005/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\n
"},{"location":"d5005/ug_dev_afu_sim_env/#5-ase-usage","title":"5. ASE Usage","text":"The AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
- Server: A makefile in the
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code. - Client: The main
cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.
"},{"location":"d5005/ug_dev_afu_sim_env/#51-ase-build-instructions","title":"5.1. ASE Build Instructions","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"d5005/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"d5005/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\n
"},{"location":"d5005/ug_dev_afu_sim_env/#513-install-ase-tools","title":"5.1.3. Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\n
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"d5005/ug_dev_afu_sim_env/#514-ase-simulator-server-build-instructions","title":"5.1.4. ASE Simulator (Server) Build Instructions","text":"ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
Change directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n...
"},{"location":"d5005/ug_dev_afu_sim_env/#514-ase-runtime-instructions","title":"5.1.4. ASE Runtime Instructions","text":"The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
- Terminal 1: In the simulation directroy
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.
Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\n
You can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
- Terminal 2: First set the environment variable
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c.
# Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\n
Note: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\n
Upon completion, the simulation generates the following files:
- Waveform dump:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.
"},{"location":"d5005/ug_dev_afu_sim_env/#52-ase-makefile-targets","title":"5.2. ASE Makefile Targets","text":"COMMAND DESCRIPTION make Build the HW Model using RTL supplied make sim Run simulator - ASE can be run in one of 4 modes set in ase.cfg - A regression mode can be enabled by writing ASE_MODE = 4 in ase.cfg and supplying an ase_regress.sh script make wave Open the waveform (if created) to be run after simulation completes make clean Clean simulation files make distclean Clean ASE sub-distribution"},{"location":"d5005/ug_dev_afu_sim_env/#53-ase-makefile-variables","title":"5.3. ASE Makefile Variables","text":"Makefile switch DESCRIPTION ASE_CONFIG Directly input an ASE configuration file path (ase.cfg) ASE_SCRIPT Directly input an ASE regression file path (ase_regress.sh, for ASE_MODE=4) SIMULATOR Directly input a simulator brand (select between 'VCS' or 'QUESTA') ASE_DISABLE_CHECKER Legacy - Disable CCI-P protocol checker module (set to '1' might speed up simulation) WARNING => NO warnings on hazards, protocol checks, timeouts will be generated. This option must be ONLY used if the design is already CCI-P compliant and fast simulation of app-specific logic is needed"},{"location":"d5005/ug_dev_afu_sim_env/#54-ase-runtime-configuration-options","title":"5.4. ASE Runtime Configuration Options","text":"The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
Switch Name Default Description ASE_MODE 1 ASE mode has the following valid values: 1 : Standard Server-Client Mode2 : Simulator stops after ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"d5005/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
- ASE_INFO: Prints mandatory information messages required to specify operation.
- ASE_ERR: Prints error messages during operation.
- ASE_MSG: Prints general messages indicating check points in the ASE. Suppress these messages by setting the environment variable
ASE_LOG to 0.
Two log levels are supported in ASE, controlled by env(ASE_LOG)
- ASE_LOG=0 | ASE_LOG_SILENT : Only INFO, ERR messages are posted
- ASE_LOG!=0 | ASE_LOG_MESSAGE : All MSG, INFO, ERR messages are posted
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\n
You cannot suppress warnings and errors."},{"location":"d5005/ug_dev_afu_sim_env/#56-troubleshooting-and-error-reference","title":"5.6. Troubleshooting and Error Reference","text":"The following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If using ASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"d5005/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
- AFU Build environment
- Using the PIM to interface with an AFU
- AFU Design
- Software Development
- Packaging the AFU
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"d5005/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X A Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\n
The OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"d5005/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
- ofs_plat_if.vh: PIM's top level wrapper interface for passing all top-level interfaces into an AFU and is copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.
- PIM interfaces are defined in base_ifcs and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).
- PIM modules are defined in ifcs_classes and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support: - host_chan
- local_memory
- hssi
- Other
- PIM utilities are defined in utils and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.
"},{"location":"d5005/ug_dev_pim_based_afu/#3-using-pim-to-interface-with-an-afu","title":"3. Using PIM to interface with an AFU","text":"To interface the PIM with an AFU:
- Create top level module ofs_plat_afu.sv.
- For each Subsystem used by your AFU, create individual channel interfaces using your selected bus protocols and connect the channel PIM Shims based on selected bus protocols.
- PCIe - Host Channel
- Local Memory
- HSSI
- Tie off all unused channels/ports.
- Connect the channel interfaces to the AFU module.
"},{"location":"d5005/ug_dev_pim_based_afu/#31-top-level-module-ofs_plaf_afu","title":"3.1. Top Level Module - ofs_plaf_afu","text":"For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\n
The SystemVerilog interface ofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"d5005/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"d5005/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
- AVMM
- AVMM-RDWR
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n
The Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
- AVMM
- AXI-Lite
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n
"},{"location":"d5005/ug_dev_pim_based_afu/#322-connect-the-host-channel-to-the-pim-shim","title":"3.2.2. Connect the host channel to the PIM Shim","text":"Using the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"d5005/ug_dev_pim_based_afu/#323-avalon-example","title":"3.2.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"d5005/ug_dev_pim_based_afu/#33-local-memory","title":"3.3. Local Memory","text":"Local memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"d5005/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
- AVMM
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\n
"},{"location":"d5005/ug_dev_pim_based_afu/#332-connect-local-memory-to-the-pim-shim","title":"3.3.2. Connect local memory to the PIM Shim","text":"Using the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\n
"},{"location":"d5005/ug_dev_pim_based_afu/#333-avalon-example","title":"3.3.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\n
"},{"location":"d5005/ug_dev_pim_based_afu/#34-high-speed-serial-interface-hssi","title":"3.4. High Speed Serial Interface (HSSI)","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"d5005/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\n
"},{"location":"d5005/ug_dev_pim_based_afu/#35-tie-off-unused-ports","title":"3.5. Tie Off Unused ports","text":"In digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\n
"},{"location":"d5005/ug_dev_pim_based_afu/#36-afu-instantiation","title":"3.6. AFU Instantiation","text":"Instantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\n
"},{"location":"d5005/ug_dev_pim_based_afu/#4-afu","title":"4. AFU","text":"The AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"d5005/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\n
"},{"location":"d5005/ug_dev_pim_based_afu/#42-afu-interfaces","title":"4.2. AFU Interfaces","text":"The AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\n
The Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\n
"},{"location":"d5005/ug_dev_pim_based_afu/#43-csr-interface","title":"4.3. CSR Interface","text":"The MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"d5005/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"d5005/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\n
"},{"location":"d5005/ug_dev_pim_based_afu/#46-systemverilog-packages","title":"4.6. SystemVerilog Packages","text":"The AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
- Host Channel Package: ofs_plat_host_chan_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv - Local Memory Package: local_mem_cfg_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.sv
The HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
- HSSI Channel Package: ofs_fim_eth_if_pkg
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\n
"},{"location":"d5005/ug_dev_pim_based_afu/#5-host-software-development","title":"5. Host Software Development","text":"The host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"d5005/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"d5005/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\n
- power - Accelerator Function power consumption, in watts. Set to 0 for Intel ADP platforms.
- clock-frequency-high - Clock frequency for uclk_usr in MHz. (optional)
- clock-frequency-low - Clock frequency for uclk_usr_div2 in MHz. (optional)
- afu-top-interface:
- class : Set to \"ofs_plat_afu\" for PIM based AFU, \"afu_main\" for native/hybrid AFU's.
- accelerator-clusters:
- name : name of AFU
- total-contexts : Set to '1'
- accelerator-type-uuid : 128-bit Universally Unique Identifier (UUID) used to identify the AFU.
The ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\n
The Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\n
"},{"location":"d5005/ug_dev_pim_based_afu/#62-source-list-file","title":"6.2. Source List File","text":"The source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\n
"},{"location":"d5005/ug_dev_pim_based_afu/#63-directory-structure","title":"6.3. Directory Structure","text":"Below is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\n
"},{"location":"d5005/ug_dev_pim_based_afu/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
- Set up the Intel\u00ae Quartus\u2122 Prime Pro Edition Software in a host server
- Set up the Docker engine
- Build and load your Docker image to the Docker engine
- Run a Docker container with OFS preloaded
The Open FPGA Stack (OFS) Docker image has two main personas:
- Development: You can develop, simulate, and build any component of the OFS. The Docker image enables you to use your laptop or server without having drivers, FPGA Platform, or specific Linux* distribution installed in your host computer. You can follow the development flow provided to run Docker on Linux.
- Deployment: You can program, load binaries, or execute real-time testing using the OPAE and OFS. To do so, the host computer must have the specified software distribution and drivers installed.
"},{"location":"d5005/ug_docker/#12-background-information","title":"1.2 Background Information","text":"A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"d5005/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":" - What is a container?
- Docker vs. Virtual Machines
- Does the Docker container have its own Kernel?
- No, Docker image or Container uses the application layer of the host computer; this functionality is the main reason for docker having lightweight and fast applications.
- Does Docker run on Linux, macOS, and Windows?
- Intel Docker Image can use the PCIe card from the host server?
- Yes, The drivers and additional information could be shared, but this could create potential security concerns (ensure your system is secure).
- Docker security
- Docker subscription
"},{"location":"d5005/ug_docker/#20-prerequisites-and-scope","title":"2.0 Prerequisites and Scope","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"d5005/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"d5005/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"d5005/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"d5005/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\n
-
Add the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\n
-
Install the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"d5005/ug_docker/#ubuntu-2204","title":"Ubuntu 22.04","text":"The Docker installation steps for Ubuntu are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\n
-
Install packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\n
-
Add Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\n
-
The following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\n
-
Update the package manager index again:
sudo apt-get update\n
-
Install Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"d5005/ug_docker/#33-load-docker-image-installation","title":"3.3 Load Docker Image installation","text":"The Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"d5005/ug_docker/#build-the-image","title":"Build the image","text":" -
You can download the Dockefile from OFS GitHub Docker.
-
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\n
Save the file
-
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\n
Note: Never remove --no-cache this could cause issues with your environmental variables inside of the container
-
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\n
"},{"location":"d5005/ug_docker/#volumen-creation","title":"Volumen creation","text":" -
Docker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\n
For more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
-
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\n
-
Inside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\n
The docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
"},{"location":"d5005/ug_docker/#34-create-a-container","title":"3.4 Create a container","text":"Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
- Using the previous example now, you can execute the docker run command.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
-
Now the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\n
Your Container ID is bdc1289fb081.
"},{"location":"d5005/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\n
The container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
-
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the evaluation script.
-
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\n
-
The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n
Now you can control and compile the design. You can use the interactive or de-attach mode. -
If you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\n
all the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
"},{"location":"d5005/ug_docker/#40-deployment","title":"4.0 Deployment","text":"The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"d5005/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
- Follow the steps listed in sections 2.1 to 2.3
- 2.1 Quartus installation
- 2.2 Docker Engine installation
- 2.3 Load Docker Image installation
- The steps required for DFL driver installation are documented Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"d5005/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
-
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Example, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
-
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\n
-
Now, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n
\u200b Your Container ID is 25b41eb4d232.
"},{"location":"d5005/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\n
The container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
-
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the eval script.
-
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n
- The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\n
Now you can control and compile the design using the interactive or de-attach mode.
"},{"location":"d5005/ug_docker/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"d5005/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"d5005/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
- Install the necessary tools, such as virt-manager, on the host machine. This may involve downloading and installing the software from the internet.
- Enable the virtualization feature on the host machine. This may involve going into the BIOS settings and enabling hardware-assisted virtualization or using a command-line tool to enable it in the operating system.
- Use virt-manager to create a new virtual machine and configure its settings. This may involve choosing a name and operating system for the virtual machine and setting the amount of memory and storage it will use.
- Install the OPAE (Open Programmable Acceleration Engine) tool on the virtual machine. This may involve downloading and installing the OPAE software.
- Install the DFL (Data Field Level) drivers on the virtual machine. These drivers allow the virtual machine to access and use the PCIe devices on the host machine. This may involve downloading and installing the drivers from the internet.
- Once all of the steps have been completed, you should be able to use the virtual machine to access and use the PCIe devices on the host machine. You may need to configure the virtual machine's settings to enable it to use the PCIe devices, such as by assigning a specific device to the virtual machine.
"},{"location":"d5005/ug_kvm/#1-modes-of-operation","title":"1. Modes of Operation","text":"Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
-
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
-
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
"},{"location":"d5005/ug_kvm/#2-enable-virtualization","title":"2. Enable Virtualization","text":"To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\n
This command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\n
In this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"d5005/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":" - Open a terminal window and log in as a user with sudo privileges.
- Check if the virtualization kernel modules are loaded by running the following command:
lsmod | grep kvm\n
-
If the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
-
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\n
- If the kernel modules are not loaded, and you cannot load them manually, it may be because virtualization is not supported or enabled in your system's BIOS or UEFI settings. You must reboot your system and enter the BIOS or UEFI settings menu to enable virtualization. The exact steps for doing this may vary depending on your system's hardware and BIOS/UEFI version, so consult your motherboard or system documentation for specific instructions.
"},{"location":"d5005/ug_kvm/#4-install-virtual-machine-manager","title":"4. Install Virtual Machine Manager","text":"Virtual Machine Manager (also known as libvirt) can be installed by following the below steps:
- Open a terminal window and log in as a user with sudo privileges.
-
Update your system package index by running the following command:
- Redhat
sudo dnf update\n
* Ubuntu sudo apt update\n
-
Install the libvirt package and any required dependencies by running the following command:
- Redhat
sudo dnf install @virtualization\n
- Ubuntu
sudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\n
-
Start the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\n
- Optional: Install the virt-manager package, which provides a GUI application for managing virtual machines, by running the following command:
sudo dnf install virt-manager\n
- Optional: If you want to be able to run virtual machines as a non-root user, add your user to the libvirt group by running the following command, replacing \"USERNAME\" with your username:
sudo usermod -a -G libvirt USERNAME\n
- You can now launch virt-manager by running the command
virt-manager as the non-root user.
Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\n
You will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\n
- Reboot your server to apply the changes
reboot\n
After completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"d5005/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
- Start the
virt-manager GUI by running the following command:
sudo virt-manager&\n
-
Create a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
-
In the virt-manager window, click the \"New virtual machine\" button.
-
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
-
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
-
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
-
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
-
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
-
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
-
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
-
Click on the \"+\" icon (Bottom left) to create the storage pool.
-
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
-
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
-
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
-
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
-
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
"},{"location":"d5005/ug_kvm/#51-passing-devices-to-the-vm","title":"5.1 Passing Devices to the VM","text":"In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"d5005/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n
\u200b
"},{"location":"d5005/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
The following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"d5005/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Ensure you add the desired VF in your PCIE devices list.
"},{"location":"d5005/ug_kvm/#513-2-pf-mode","title":"5.1.3 2 PF Mode","text":"For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
-
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n
-
Pass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
-
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\n
"},{"location":"d5005/ug_kvm/#52-virt-manager-configuration-changes","title":"5.2 Virt-Manager Configuration Changes","text":" -
Edit the XML file for your machine and include the following
-
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\n
-
Inside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\n
-
Ensure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\n
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Save the changes \"Apply\"
-
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\n
-
Ensure your devices are enumerated properly.
-
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n
2. Deployment Mode:
B1:00.5\n
-
Under the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n
2. Deployment Mode:
177:00.0\n
-
Click on \"Begin Installation.\" and follow the wizard installation of the OS.
-
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
-
Under your virtual machine, configure your VM proxy:
- Redhat How to apply a system-wide proxy?
- Ubuntu Define proxy settings
- Configure Git to use a proxy
-
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
-
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\n
-
Use the Virtual function (Not supported at management mode)
-
Ensure the DFL kernel drivers is install in your VM system
-
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\n
-
Verify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n
-
Test the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\n
After the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
"},{"location":"d5005/ug_kvm/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/sw_install_soc_attach/","title":"Software Installation Guide: Intel Agilex 7 SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"f2000x/sw_install_soc_attach/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the OFS SoC Attach software stack on the host and platform. After reviewing this document, a user shall be able to:
- Set up their server environment according to the Best Known Configuration (BKC)
- Build and install the OPAE Software Development Kit (SDK) on the host
- Build and load a Yocto image with the OPAE SDK and Linux DFL Drivers included on the SoC
This document does not cover first time setup of the IPU Platform F2000X-PL platform.
"},{"location":"f2000x/sw_install_soc_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating an SoC Attach release. This document will cover key topics related to initial bring up of the IPU Platform F2000X-PL software stack, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"f2000x/sw_install_soc_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"f2000x/sw_install_soc_attach/#table-2-software-component-version-summary-for-soc-attach","title":"Table 2: Software Component Version Summary for SoC Attach","text":"Name Location META-OFS https://github.com/OFS/meta-ofs, tag: ofs-2024.1-2 Host Operating System Ubuntu 22.04 LTS Host OPAE SDK 2.12.0-5 SoC OS meta-intel-ese Reference Distro 1.0-ESE (kirkstone) SoC Kernel Version 6.1.78-dfl SoC OPAE SDK 2.12.0-5 SoC Linux DFL ofs-2024.1-6.1-2 Not all components shown in Table 2 will have an update available upon release. The OPAE SDK and Linux DFL software stacks are incorporated into a Yocto image and do not need to be downloaded separately.
"},{"location":"f2000x/sw_install_soc_attach/#20-updating-the-ipu-platform-f2000x-pl","title":"2.0 Updating the IPU Platform F2000X-PL","text":"Every IPU Platform F2000X-PL ships with pre-programmed firmware for the FPGA user1, user2, and factory images, the Cyclone 10 BMC RTL and FW, the SoC NVMe, and the SoC BIOS. In this software installation guide, we will only be focusing on the building and loading of a new SoC NVMe image. Board setup and configuration for the IPU Platform F2000X-PL is discussed in the Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"f2000x/sw_install_soc_attach/#30-compiling-a-custom-yocto-soc-image","title":"3.0 Compiling a Custom Yocto SoC Image","text":"Current Yocto image architecture for SoC Attach is based off of the IOTG Yoct-based ESE BSP, with the addition of the Linux DFL kernel including the latest DFL drivers for FPGA devices alongside the OPAE SDK user space software. The image targets x86_64 SoC FPGA devices but should boot on most UEFI-based machines. The source code and documentation for this image is hosted on the meta-ofs repository.
Build requirements exceed 100 GiB of disk space, depending on which image is built. As a reference point, on a system with two Intel(R) Xeon(R) E5-2699 v4 for a total of 44 CPU cores, the initial, non-incremental build takes less than an hour of wall time.
The repo tool is needed to clone the various Yocto layer repositories used in this example.
Note: If you are behind a firewall that prevents you from accessing references using the git:// protocol, you can use the following to redirect Git to use the corresponding https repositories for Yocto only: git config --global url.https://git.yoctoproject.org/.insteadOf git://git.yoctoproject.org/.
To compile the image as-is, use the following steps (as provided in meta-ofs):
-
Create and initialize the source directory.
mkdir ofs-yocto && cd ofs-yocto\ngit clone --recurse-submodules --shallow-submodules https://github.com/OFS/meta-ofs\ncd meta-ofs\ngit checkout tags/ofs-2024.1-2\n
-
Build packages and create an image.
cd examples/iotg-yocto-ese\nTEMPLATECONF=$PWD/conf source openembedded-core/oe-init-build-env build\nbitbake mc:x86-2022-minimal:core-image-full-cmdline\n
The resulting GPT disk image is available in uncompressed (.wic) and compressed form (.wic.gz) in meta-ofs/examples/iotg-yocto-ese/build/tmp-x86-2021-minimal-glibc/deploy/images/intel-corei7-64/. With no changes the uncompressed image size is ~21 GB.
The image type core-image-full-cmdline includes the familiar GNU core utilities, as opposed to core-image-minimal which uses BusyBox instead.
The example build configuration files under build/conf/ are symlinked from examples/iotg-yocto-ese/. To customize the image, start by modifying local.conf and bblayers.conf.
The uncompressed Yocto image can be loaded onto a flash drive as discussed in section 1.5.5 Creating a Bootable USB Flash Drive for the SoC and written to NVMe as the default boot target for the SoC as demonstrated in section 2.1 Updating the F2000X-PL ICX-D SoC NVMe.
"},{"location":"f2000x/sw_install_soc_attach/#40-verifying-the-icx-d-soc-software-stack","title":"4.0 Verifying the ICX-D SoC Software Stack","text":"The reference SoC Attach FIM and unaltered FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Full supported functionality of the HEMs is documented in this platform's associated Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs and will not be covered here.
"},{"location":"f2000x/sw_install_soc_attach/#50-setting-up-the-host","title":"5.0 Setting up the Host","text":"External SoC Attach supports testing Host to FPGA latency, MMIO latency, and MMIO bandwidth. This testing is accomplished using the utility host_exerciser on the host, which is included as a part of OPAE. This section will cover the installation and verification flow for a host interacting with the SoC Attach workload.
Review Section 1.2 Server Requirements of the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL for a list of changes required on the host to support an IPU Platform F2000X-PL and for a list of supported OS distributions. Installation will require an active internet connection to resolve dependencies.
The following software checks may be run on the host to verify the FPGA has been detected and has auto-negotiated the correct PCIe link width/speed. These commands do not require any packages to be installed. We are using PCIe BDF b1:00.0 as an example address.
# Check that the board has enumerated successfully.\n# Your PCIe BDF may differ from what is shown below.\n$ lspci | grep accel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce b1:00.1 Processing accelerators: Intel Corporation Device bcce\n\n# Check PCIe link status and speed. Width should be x16, and speed whould be 16GT/s\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\n
"},{"location":"f2000x/sw_install_soc_attach/#60-installing-the-opae-sdk-on-the-host","title":"6.0 Installing the OPAE SDK On the Host","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit the opae.io page, and the Software Reference Manual
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access. If you wish to install pre-built artifacts instead of building the release yourself, skip steps 3 and 4.
-
Before Installing the newest version of OPAE you must remove any prior OPAE framework installations.
$ sudo apt-get remove opae*\n
-
The following system and Python3 package dependencies must be installed before OPAE may be built.
$ sudo apt-get install bison flex git ssh pandoc devscripts debhelper cmake python3-dev libjson-c-dev uuid-dev libhwloc-dev doxygen libtbb-dev libncurses-dev libspdlog-dev libspdlog1 python3-pip libedit-dev pkg-config libcli11-dev libssl-dev dkms libelf-dev gawk openssl libudev-dev libpci-dev libiberty-dev autoconf llvm\n\n$ python3 -m pip install setuptools pybind11 jsonschema\n
-
Clone the OPAE SDK repo. In this example we will use the top level directory OFS for our package installs.
$ mkdir OFS && cd OFS\n$ git init\n$ git clone https://github.com/OFS/opae-sdk\n$ cd opae-sdk\n$ git checkout tags/2.12.0-5\n\n# Verifying we are on the correct release tag\n$ git describe --tags\n2.12.0-5\n
-
Navigate to the automatic DEB package build script location and execute.
$ cd OFS/opae-sdk/packaging/opae/deb $ ./create\n\n# Verify all packages are present\n$ ls | grep opae.*.deb\nopae_2.12.0-5_amd64.deb\nopae-dbgsym_2.12.0-5_amd64.ddeb\nopae-devel_2.12.0-5_amd64.deb\nopae-devel-dbgsym_2.12.0-5_amd64.ddeb\nopae-extra-tools_2.12.0-5_amd64.deb\nopae-extra-tools-dbgsym_2.12.0-5_amd64.ddeb\n
-
Install your newly built OPAE SDK packages.
$ cd OFS/opae-sdk/packaging/opae/deb\n$ sudo dpkg -i opae*.deb\n
The OPAE SDK version installed the host are identical to those installed on the SoC. A set of pre-compiled OPAE SDK artifacts are included in this release. These can be downloaded from ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell and installed without building/configuring.
$ tar xf opae-*.tar.gz\n$ sudo dpkg -i opae*.deb\n
-
Enable iommu=on, pcie=realloc, and set hugepages as host kernel parameters.
# Check if parameters are already enabled\n$ cat /proc/cmdline\n
If you do not see intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200, then add them manually.
$ sudo vim /etc/default/grub\n\n# Edit the value for GRUB_CMDLINE_LINUX, add the values at the end of the variable inside of the double quotes. Example: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/rhel00-swap rd.lvm.lv=rhel00/root rd.lvm.lv=rhel00/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\"\n# Save your changes, then apply them with the following\n$ sudo grub2-mkconfig\n$ sudo reboot now\n
After rebooting, check that proc/cmdline reflects your changes.
"},{"location":"f2000x/sw_install_soc_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"f2000x/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\n
"},{"location":"f2000x/ug_dev_afu_host_software/#11-find-and-connect-to-afu","title":"1.1. Find and connect to AFU","text":"Here is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\n
In main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\n
"},{"location":"f2000x/ug_dev_afu_host_software/#12-map-mmio-optional","title":"1.2. Map MMIO (optional)","text":"Mapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\n
The below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\n
"},{"location":"f2000x/ug_dev_afu_host_software/#13-allocate-shared-memory-buffers","title":"1.3. Allocate Shared Memory Buffers","text":"The below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\n
In main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\n
"},{"location":"f2000x/ug_dev_afu_host_software/#14-perform-acceleration","title":"1.4. Perform Acceleration","text":"The host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\n
"},{"location":"f2000x/ug_dev_afu_host_software/#15-cleanup","title":"1.5. Cleanup","text":"When the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\n
"},{"location":"f2000x/ug_dev_afu_host_software/#2-building-the-host-application","title":"2. Building the Host Application","text":"A Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
- Path to common_include.mk (from examples-afu)
- TEST name
- Source files: SRCS
- Path to .json file (relative to Makefile directory)
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\n
"},{"location":"f2000x/ug_dev_afu_host_software/#3-running-the-host-application","title":"3. Running the Host Application","text":"To run the host application, you will need to:
- Load AFU onto the FIM
- Create VF's
- Bind VF's using the OPAE Drivers
- Run application
See the associated AFU Developer Guide for details.
"},{"location":"f2000x/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
- C/C++
- Verilog/SystemVerilog
- RTL simulators such as Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"f2000x/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
- Verilog
- SystemVerilog
- VHDL
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"f2000x/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
- Software: OPAE API implemented in the C programming language.
- Hardware: PCIe SS TLP specification implemented in SystemVerilog.
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"f2000x/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":" - The ASE provides a protocol checker to ensure protocol correctness. The ASE also provides methods to identify potential issues early, before in-system deployment.
- The ASE can help identify certain lock conditions and Configuration and Status Registers (CSR) address mapping and pointer math errors.
- The ASE tracks memory requested from the accelerator. The memory model immediately flags illegal memory transactions to locations outside of requested memory spaces. Consequently, you can fix incorrect memory accesses early, during the simulation phase.
- The ASE does not guarantee that you can synthesize an AFU. After you verify the AFU RTL functionality in the ASE, use the ASE and the Quartus\u00ae Prime Pro Edition software iteratively to generate the Accelerator Function (AF).
- The ASE does not require administrator privileges. After installing all the required tools, you can run the ASE on a plain vanilla user Linux machine.
"},{"location":"f2000x/ug_dev_afu_sim_env/#23-ase-limitations","title":"2.3. ASE Limitations","text":"When using ASE in the application development cycle, consider the following limitations:
- The ASE is a transaction-level simulator. It does not model either Intel UPI- or PCIe-specific packet structures and protocol layers.
- The ASE does not simulate caching and is not a cache simulator. It cannot reliably simulate cache collisions or capacity issues.
- Although ASE models some latency parameters, it cannot model real-time system-specific latency. It is also not an accurate timing simulation of the design or latency and bandwidth of the real system. The ASE models enable you to develop functionally correct accelerators.
- The ASE does not simulate multi-AFU or multi-socket configurations.
"},{"location":"f2000x/ug_dev_afu_sim_env/#24-ase-based-afu-design-workflow","title":"2.4 ASE-Based AFU Design Workflow","text":"AFU development using the ASE includes the following four stages:
-
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
-
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
-
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
-
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
- The AFU RTL code is synthesizable.
- The AFU RTL code meets timing.
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"f2000x/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
- C-Compiler: gcc 8.5.0 or above
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
- CMake: version 3.15 or above
- Python: version 3.6.8 or above
- Intel Quartus Prime Pro 24.1: The ASE must find the
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.
The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
- Compilation of the SystemVerilog Direct Programming Interface (DPI) constructs
- Compilation of the standard examples that are included in the installation
- Support for SystemC
"},{"location":"f2000x/ug_dev_afu_sim_env/#4-package-description","title":"4. Package Description","text":"The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\n
This directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.
"},{"location":"f2000x/ug_dev_afu_sim_env/#41-ase-scripts","title":"4.1. ASE Scripts","text":"The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
"},{"location":"f2000x/ug_dev_afu_sim_env/#411-simulation-tool-set-up","title":"4.1.1. Simulation Tool Set Up","text":"Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
"},{"location":"f2000x/ug_dev_afu_sim_env/#412-ase-environment-check","title":"4.1.2. ASE Environment Check","text":"This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\n
"},{"location":"f2000x/ug_dev_afu_sim_env/#413-afu-simulation-using-the-ase","title":"4.1.3. AFU Simulation Using the ASE","text":"Before configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
"},{"location":"f2000x/ug_dev_afu_sim_env/#4131-afu_sim_setup","title":"4.1.3.1. afu_sim_setup","text":"The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\n
* The only required argument to the afu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.
afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.
generate_ase_environment.py: This script instantiates your simulated platform configuration.
"},{"location":"f2000x/ug_dev_afu_sim_env/#4132-rtl_src_configpy","title":"4.1.3.2. rtl_src_config.py","text":"The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
"},{"location":"f2000x/ug_dev_afu_sim_env/#4133-generate_ase_environmentpy","title":"4.1.3.3. generate_ase_environment.py","text":"The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
- Synopsys: Creates
synopsys_sim.setup and vcs_run.tcl in the configuration directory. - Siemens: Creates
vsim_run.tcl in the configuration directory.
The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"f2000x/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\n
"},{"location":"f2000x/ug_dev_afu_sim_env/#5-ase-usage","title":"5. ASE Usage","text":"The AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
- Server: A makefile in the
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code. - Client: The main
cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.
"},{"location":"f2000x/ug_dev_afu_sim_env/#51-ase-build-instructions","title":"5.1. ASE Build Instructions","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"f2000x/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"f2000x/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\n
"},{"location":"f2000x/ug_dev_afu_sim_env/#513-install-ase-tools","title":"5.1.3. Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\n
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"f2000x/ug_dev_afu_sim_env/#514-ase-simulator-server-build-instructions","title":"5.1.4. ASE Simulator (Server) Build Instructions","text":"ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
Change directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n...
"},{"location":"f2000x/ug_dev_afu_sim_env/#514-ase-runtime-instructions","title":"5.1.4. ASE Runtime Instructions","text":"The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
- Terminal 1: In the simulation directroy
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.
Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\n
You can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
- Terminal 2: First set the environment variable
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c.
# Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\n
Note: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\n
Upon completion, the simulation generates the following files:
- Waveform dump:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.
"},{"location":"f2000x/ug_dev_afu_sim_env/#52-ase-makefile-targets","title":"5.2. ASE Makefile Targets","text":"COMMAND DESCRIPTION make Build the HW Model using RTL supplied make sim Run simulator - ASE can be run in one of 4 modes set in ase.cfg - A regression mode can be enabled by writing ASE_MODE = 4 in ase.cfg and supplying an ase_regress.sh script make wave Open the waveform (if created) to be run after simulation completes make clean Clean simulation files make distclean Clean ASE sub-distribution"},{"location":"f2000x/ug_dev_afu_sim_env/#53-ase-makefile-variables","title":"5.3. ASE Makefile Variables","text":"Makefile switch DESCRIPTION ASE_CONFIG Directly input an ASE configuration file path (ase.cfg) ASE_SCRIPT Directly input an ASE regression file path (ase_regress.sh, for ASE_MODE=4) SIMULATOR Directly input a simulator brand (select between 'VCS' or 'QUESTA') ASE_DISABLE_CHECKER Legacy - Disable CCI-P protocol checker module (set to '1' might speed up simulation) WARNING => NO warnings on hazards, protocol checks, timeouts will be generated. This option must be ONLY used if the design is already CCI-P compliant and fast simulation of app-specific logic is needed"},{"location":"f2000x/ug_dev_afu_sim_env/#54-ase-runtime-configuration-options","title":"5.4. ASE Runtime Configuration Options","text":"The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
Switch Name Default Description ASE_MODE 1 ASE mode has the following valid values: 1 : Standard Server-Client Mode2 : Simulator stops after ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"f2000x/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
- ASE_INFO: Prints mandatory information messages required to specify operation.
- ASE_ERR: Prints error messages during operation.
- ASE_MSG: Prints general messages indicating check points in the ASE. Suppress these messages by setting the environment variable
ASE_LOG to 0.
Two log levels are supported in ASE, controlled by env(ASE_LOG)
- ASE_LOG=0 | ASE_LOG_SILENT : Only INFO, ERR messages are posted
- ASE_LOG!=0 | ASE_LOG_MESSAGE : All MSG, INFO, ERR messages are posted
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\n
You cannot suppress warnings and errors."},{"location":"f2000x/ug_dev_afu_sim_env/#56-troubleshooting-and-error-reference","title":"5.6. Troubleshooting and Error Reference","text":"The following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If using ASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"f2000x/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
- AFU Build environment
- Using the PIM to interface with an AFU
- AFU Design
- Software Development
- Packaging the AFU
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"f2000x/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X A Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\n
The OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"f2000x/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
- ofs_plat_if.vh: PIM's top level wrapper interface for passing all top-level interfaces into an AFU and is copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.
- PIM interfaces are defined in base_ifcs and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).
- PIM modules are defined in ifcs_classes and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support: - host_chan
- local_memory
- hssi
- Other
- PIM utilities are defined in utils and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.
"},{"location":"f2000x/ug_dev_pim_based_afu/#3-using-pim-to-interface-with-an-afu","title":"3. Using PIM to interface with an AFU","text":"To interface the PIM with an AFU:
- Create top level module ofs_plat_afu.sv.
- For each Subsystem used by your AFU, create individual channel interfaces using your selected bus protocols and connect the channel PIM Shims based on selected bus protocols.
- PCIe - Host Channel
- Local Memory
- HSSI
- Tie off all unused channels/ports.
- Connect the channel interfaces to the AFU module.
"},{"location":"f2000x/ug_dev_pim_based_afu/#31-top-level-module-ofs_plaf_afu","title":"3.1. Top Level Module - ofs_plaf_afu","text":"For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\n
The SystemVerilog interface ofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"f2000x/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"f2000x/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
- AVMM
- AVMM-RDWR
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n
The Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
- AVMM
- AXI-Lite
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#322-connect-the-host-channel-to-the-pim-shim","title":"3.2.2. Connect the host channel to the PIM Shim","text":"Using the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#323-avalon-example","title":"3.2.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#33-local-memory","title":"3.3. Local Memory","text":"Local memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"f2000x/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
- AVMM
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#332-connect-local-memory-to-the-pim-shim","title":"3.3.2. Connect local memory to the PIM Shim","text":"Using the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#333-avalon-example","title":"3.3.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#34-high-speed-serial-interface-hssi","title":"3.4. High Speed Serial Interface (HSSI)","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"f2000x/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#35-tie-off-unused-ports","title":"3.5. Tie Off Unused ports","text":"In digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#36-afu-instantiation","title":"3.6. AFU Instantiation","text":"Instantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#4-afu","title":"4. AFU","text":"The AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"f2000x/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#42-afu-interfaces","title":"4.2. AFU Interfaces","text":"The AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\n
The Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#43-csr-interface","title":"4.3. CSR Interface","text":"The MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"f2000x/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"f2000x/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#46-systemverilog-packages","title":"4.6. SystemVerilog Packages","text":"The AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
- Host Channel Package: ofs_plat_host_chan_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv - Local Memory Package: local_mem_cfg_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.sv
The HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
- HSSI Channel Package: ofs_fim_eth_if_pkg
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#5-host-software-development","title":"5. Host Software Development","text":"The host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"f2000x/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"f2000x/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\n
- power - Accelerator Function power consumption, in watts. Set to 0 for Intel ADP platforms.
- clock-frequency-high - Clock frequency for uclk_usr in MHz. (optional)
- clock-frequency-low - Clock frequency for uclk_usr_div2 in MHz. (optional)
- afu-top-interface:
- class : Set to \"ofs_plat_afu\" for PIM based AFU, \"afu_main\" for native/hybrid AFU's.
- accelerator-clusters:
- name : name of AFU
- total-contexts : Set to '1'
- accelerator-type-uuid : 128-bit Universally Unique Identifier (UUID) used to identify the AFU.
The ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\n
The Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#62-source-list-file","title":"6.2. Source List File","text":"The source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#63-directory-structure","title":"6.3. Directory Structure","text":"Below is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\n
"},{"location":"f2000x/ug_dev_pim_based_afu/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
- Set up the Intel\u00ae Quartus\u2122 Prime Pro Edition Software in a host server
- Set up the Docker engine
- Build and load your Docker image to the Docker engine
- Run a Docker container with OFS preloaded
The Open FPGA Stack (OFS) Docker image has two main personas:
- Development: You can develop, simulate, and build any component of the OFS. The Docker image enables you to use your laptop or server without having drivers, FPGA Platform, or specific Linux* distribution installed in your host computer. You can follow the development flow provided to run Docker on Linux.
- Deployment: You can program, load binaries, or execute real-time testing using the OPAE and OFS. To do so, the host computer must have the specified software distribution and drivers installed.
"},{"location":"f2000x/ug_docker/#12-background-information","title":"1.2 Background Information","text":"A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"f2000x/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":" - What is a container?
- Docker vs. Virtual Machines
- Does the Docker container have its own Kernel?
- No, Docker image or Container uses the application layer of the host computer; this functionality is the main reason for docker having lightweight and fast applications.
- Does Docker run on Linux, macOS, and Windows?
- Intel Docker Image can use the PCIe card from the host server?
- Yes, The drivers and additional information could be shared, but this could create potential security concerns (ensure your system is secure).
- Docker security
- Docker subscription
"},{"location":"f2000x/ug_docker/#20-prerequisites-and-scope","title":"2.0 Prerequisites and Scope","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"f2000x/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"f2000x/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"f2000x/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"f2000x/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\n
-
Add the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\n
-
Install the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"f2000x/ug_docker/#ubuntu-2204","title":"Ubuntu 22.04","text":"The Docker installation steps for Ubuntu are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\n
-
Install packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\n
-
Add Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\n
-
The following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\n
-
Update the package manager index again:
sudo apt-get update\n
-
Install Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"f2000x/ug_docker/#33-load-docker-image-installation","title":"3.3 Load Docker Image installation","text":"The Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"f2000x/ug_docker/#build-the-image","title":"Build the image","text":" -
You can download the Dockefile from OFS GitHub Docker.
-
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\n
Save the file
-
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\n
Note: Never remove --no-cache this could cause issues with your environmental variables inside of the container
-
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\n
"},{"location":"f2000x/ug_docker/#volumen-creation","title":"Volumen creation","text":" -
Docker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\n
For more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
-
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\n
-
Inside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\n
The docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
"},{"location":"f2000x/ug_docker/#34-create-a-container","title":"3.4 Create a container","text":"Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
- Using the previous example now, you can execute the docker run command.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
-
Now the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\n
Your Container ID is bdc1289fb081.
"},{"location":"f2000x/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\n
The container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
-
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the evaluation script.
-
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\n
-
The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n
Now you can control and compile the design. You can use the interactive or de-attach mode. -
If you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\n
all the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
"},{"location":"f2000x/ug_docker/#40-deployment","title":"4.0 Deployment","text":"The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"f2000x/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
- Follow the steps listed in sections 2.1 to 2.3
- 2.1 Quartus installation
- 2.2 Docker Engine installation
- 2.3 Load Docker Image installation
- The steps required for DFL driver installation are documented Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"f2000x/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
-
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Example, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
-
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\n
-
Now, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n
\u200b Your Container ID is 25b41eb4d232.
"},{"location":"f2000x/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\n
The container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
-
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the eval script.
-
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n
- The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\n
Now you can control and compile the design using the interactive or de-attach mode.
"},{"location":"f2000x/ug_docker/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"f2000x/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"f2000x/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
- Install the necessary tools, such as virt-manager, on the host machine. This may involve downloading and installing the software from the internet.
- Enable the virtualization feature on the host machine. This may involve going into the BIOS settings and enabling hardware-assisted virtualization or using a command-line tool to enable it in the operating system.
- Use virt-manager to create a new virtual machine and configure its settings. This may involve choosing a name and operating system for the virtual machine and setting the amount of memory and storage it will use.
- Install the OPAE (Open Programmable Acceleration Engine) tool on the virtual machine. This may involve downloading and installing the OPAE software.
- Install the DFL (Data Field Level) drivers on the virtual machine. These drivers allow the virtual machine to access and use the PCIe devices on the host machine. This may involve downloading and installing the drivers from the internet.
- Once all of the steps have been completed, you should be able to use the virtual machine to access and use the PCIe devices on the host machine. You may need to configure the virtual machine's settings to enable it to use the PCIe devices, such as by assigning a specific device to the virtual machine.
"},{"location":"f2000x/ug_kvm/#1-modes-of-operation","title":"1. Modes of Operation","text":"Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
-
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
-
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
"},{"location":"f2000x/ug_kvm/#2-enable-virtualization","title":"2. Enable Virtualization","text":"To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\n
This command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\n
In this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"f2000x/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":" - Open a terminal window and log in as a user with sudo privileges.
- Check if the virtualization kernel modules are loaded by running the following command:
lsmod | grep kvm\n
-
If the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
-
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\n
- If the kernel modules are not loaded, and you cannot load them manually, it may be because virtualization is not supported or enabled in your system's BIOS or UEFI settings. You must reboot your system and enter the BIOS or UEFI settings menu to enable virtualization. The exact steps for doing this may vary depending on your system's hardware and BIOS/UEFI version, so consult your motherboard or system documentation for specific instructions.
"},{"location":"f2000x/ug_kvm/#4-install-virtual-machine-manager","title":"4. Install Virtual Machine Manager","text":"Virtual Machine Manager (also known as libvirt) can be installed by following the below steps:
- Open a terminal window and log in as a user with sudo privileges.
-
Update your system package index by running the following command:
- Redhat
sudo dnf update\n
* Ubuntu sudo apt update\n
-
Install the libvirt package and any required dependencies by running the following command:
- Redhat
sudo dnf install @virtualization\n
- Ubuntu
sudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\n
-
Start the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\n
- Optional: Install the virt-manager package, which provides a GUI application for managing virtual machines, by running the following command:
sudo dnf install virt-manager\n
- Optional: If you want to be able to run virtual machines as a non-root user, add your user to the libvirt group by running the following command, replacing \"USERNAME\" with your username:
sudo usermod -a -G libvirt USERNAME\n
- You can now launch virt-manager by running the command
virt-manager as the non-root user.
Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\n
You will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\n
- Reboot your server to apply the changes
reboot\n
After completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"f2000x/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
- Start the
virt-manager GUI by running the following command:
sudo virt-manager&\n
-
Create a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
-
In the virt-manager window, click the \"New virtual machine\" button.
-
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
-
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
-
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
-
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
-
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
-
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
-
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
-
Click on the \"+\" icon (Bottom left) to create the storage pool.
-
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
-
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
-
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
-
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
-
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
"},{"location":"f2000x/ug_kvm/#51-passing-devices-to-the-vm","title":"5.1 Passing Devices to the VM","text":"In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"f2000x/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n
\u200b
"},{"location":"f2000x/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
The following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"f2000x/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Ensure you add the desired VF in your PCIE devices list.
"},{"location":"f2000x/ug_kvm/#513-2-pf-mode","title":"5.1.3 2 PF Mode","text":"For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
-
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n
-
Pass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
-
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\n
"},{"location":"f2000x/ug_kvm/#52-virt-manager-configuration-changes","title":"5.2 Virt-Manager Configuration Changes","text":" -
Edit the XML file for your machine and include the following
-
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\n
-
Inside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\n
-
Ensure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\n
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Save the changes \"Apply\"
-
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\n
-
Ensure your devices are enumerated properly.
-
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n
2. Deployment Mode:
B1:00.5\n
-
Under the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n
2. Deployment Mode:
177:00.0\n
-
Click on \"Begin Installation.\" and follow the wizard installation of the OS.
-
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
-
Under your virtual machine, configure your VM proxy:
- Redhat How to apply a system-wide proxy?
- Ubuntu Define proxy settings
- Configure Git to use a proxy
-
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
-
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\n
-
Use the Virtual function (Not supported at management mode)
-
Ensure the DFL kernel drivers is install in your VM system
-
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\n
-
Verify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n
-
Test the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\n
After the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
"},{"location":"f2000x/ug_kvm/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/","title":"Board Installation Guidelines: Intel\u00ae FPGA SmartNIC N6000/1-PL, Intel\u00ae FPGA PAC D5005","text":"Last updated: November 19, 2024
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup - the Intel\u00ae FPGA SmartNIC N6000/1-PL and the Intel\u00ae FPGA PAC D5005. After reading the document a user shall be able to:
- Safely install and remove an ADP
- Set up their server BIOS with the recommended settings
- Learn about thermal cooling requirements for their platform
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to all three platforms.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported ADP platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-2-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 2: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platforms you have if you are unsure. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers from sections, whose installation is covered in the [Software Installation Guide: OFS for PCIe Attach FPGAs].
SKU Mapping SKU Value Primary Difference fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table provides a picture reference for the hardware components discussed in the rest of the document.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-3-hardware-bkc","title":"Table 3: Hardware BKC","text":"Component Image Intel\u00ae FPGA SmartNIC N6001-PL (SKU2) Supermicro Server SYS-220HE Intel FPGA Download Cable II (Only Required for manual flashing) 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing) In addition to the above, all OFS ADP platforms require an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector. This cable will differ between server vendors - review the pinout of the power connector on the Intel\u00ae FPGA Programmable Acceleration Card D5005 Data Sheet or Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based ADP.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#20-initial-server-setup","title":"2.0 Initial Server Setup","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#21-server-information-for-intel-fpga-smartnic-n60001-pl","title":"2.1 Server Information for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Both the server BIOS and BMC need to match the versions listed below in Table 4: Supermicro Server BMC BKC. These updates only apply for this specific Best Known Configuration (BKC) - other server manufacturers may require different BIOS updates. Please consult your server's user guide and release notes for update information.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-4-supermicro-server-bmc-bkc","title":"Table 4: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4) Information about the server\u2019s currently loaded firmware can be found on the BMC web portal dashboard. Accessing this page requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d. During boot the BMC\u2019s login IP will be presented on the screen.
Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case. It is recommended the user change their BMC\u2019s default username as soon as they are able.
After logging in you should be able to review information about the BMC and BIOS by referring to the System box, visible upon initial loading of the page. Double check that the values match those in Table 4. If they do not, you may download the appropriate versions from the SuperMicro product page by selecting the BIOS option and downloading the most recent \u201cBundled Software File Name\u201d. Follow the BMC and BIOS update instructions included in the SuperMicro manuals page in the document X12/H12 BMC Manual in Appendix A.2 Updating Firmware Using BMC Web GUI.
If using a different server model, refer to that server\u2019s user guide for instructions on remote system management. Ensure that any system you end up using meets all the following requirements:
- Main Board: PCI Express 3.0 (D5005) or 4.0 (N6000/1) compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#22-server-information-for-intel-fpga-pac-d5005","title":"2.2 Server Information for Intel\u00ae FPGA PAC D5005","text":"Refer to sections 2.1-2.3 of the Intel Acceleration Stack Quick Start Guide: Intel FPGA Programmable Acceleration Card D5005 for a complete overview of the physical installation process and ESD precautions for the D5005 platform.
Ensure that the system meets all the following requirements before proceeding to install the Intel\u00ae FPGA PAC D5005 into a server.
- Main Board: PCI Express 3.0 compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
Detailed mechanical for information can be found on the D5005 Data Sheet and in section 4.0 Mechanical Information of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837).
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#30-server-settings","title":"3.0 Server Settings","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#31-bios-settings","title":"3.1 BIOS Settings","text":"You must enable Intel VT-x/VT-d technologies for the PCIe slot housing your ADP. The following steps are known to work on a SuperMicro SYS-220HE server platform.
-
To enter the Supermicro server\u2019s BIOS setup page, reboot, and press \\<Delete> when prompted. You can browse the tabs / options with a combination of arrow keys along with \\<Escape> and \\<Enter>.
-
Navigate right to the Advanced tab, then select the following menu options: Chipset Configuration -> North Bridge -> IIO Configuration -> Intel VT for Directed I/O (VT-d).
-
If not already, enable the option Intel VT for Directed I/O (VT-d).
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#31-server-fan-speed","title":"3.1 Server Fan Speed","text":"The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
- Log in to the SuperMicro server BMC. (This requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d.)
- During boot the BMC\u2019s login IP will be presented on the screen. Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case.
- On the left menu select System -> Component Info, select the Fan tab, under Advanced Settings click the circle next to Full Speed.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#32-cooling-requirements","title":"3.2 Cooling Requirements","text":"Please refer to sections 8.1 and 8.2 of the Intel FPGA Programmable Acceleration Card D5005 Data Sheet or section 6.0 of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) for guidance on cooling specifications that must be met when using these platforms. Failure to adhere to these guidelines may result in thermal runaway and/or performance degradation.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#40-board-installation-procedure","title":"4.0 Board Installation Procedure","text":""},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#41-pcie-slot-mappings-for-intel-fpga-smartnic-n60001-pl","title":"4.1 PCIe Slot Mappings for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"The Intel N6000/1-PL FPGA SmartNIC Platforms are officially verified in the upper middle PCIe x16 slot (Slot 3). If using a different slot, refer to the information in Table 5 PCIe Slot Mapping for which port settings to change in server BIOS.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-5-pcie-slot-mapping","title":"Table 5: PCIe Slot Mapping","text":"CPU Number Port Number (in BIOS) PCIe Slot CPU1 Port 2 5 and 6 CPU1 Port 4 7 and 8 CPU2 Port 2 1 and 2 CPU2 Port 4 3 and 4"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#42-installation-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.2 Installation Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe installation of an ADP platform into a supported server. While an Intel\u00ae FPGA SmartNIC N6001-PL is shown in the images below, this procedure applies to all three platforms.
- Position the board over the selected connector on the motherboard.
- Press down gently and firmly to seat the card in the PCIe slot, and then secure the bracket to the system chassis with the retention screw.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-6-adp-installation-procedure","title":"Table 6: ADP Installation Procedure","text":"Callout Description 1 Retention screw 2 Press down here gently 3 Press down here gently 4 Motherboard Do not bend the card while inserting into a slot. Do not apply much pressure in regions 2 or 3 while inserting.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#43-removal-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.3 Removal Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe removal of the platforms from a supported server.
- Disconnect all power cords from the server power supply(s).
- Remove the retention bracket screw.
- Carefully lift the card out of the PCIe slot.
"},{"location":"hw/common/board_installation/adp_board_installation/adp_board_installation_guidelines/#table-7-adp-removal-procedure","title":"Table 7: ADP Removal Procedure","text":"Callout Description 1 Retention screw 2 Pull up here gently 3 Motherboard Do not bend the card while removing it from the slot.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/","title":"Board Installation Guidelines: Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)","text":"Last updated: November 19, 2024
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup for the F-Series (2x F-Tile) and I-Series (2x R-Tile and 1xF-Tile) Development Kits. After reading the document a user shall be able to:
- Safely install and remove a development kit
- Set up their server BIOS with the recommended settings
- Learn about thermal cooling requirements for their platform
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to both platforms.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported development kit platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#table-2-hardware-bkc-for-ofs-pcie-attach-targeting-the-f-series-development-kit","title":"Table 2: Hardware BKC for OFS PCIe Attach targeting the F-Series Development Kit","text":"The following table highlights the hardware which composes the Best Known Configuation (BKC) for the OFS 2024.2-1 PCIe Attach release targeting F-Series Development Kit.
Note: The Dell R750 server product line is known not to work with this release.
Component Link Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/agilex/agf027-and-agf023.html Intel FPGA Download Cable II (optional) https://www.intel.com/content/www/us/en/products/sku/215664/intel-fpga-download-cable-ii/specifications.html SuperMicro SYS-220HE-FTNR https://www.supermicro.com/en/products/system/hyper/2u/sys-220he-ftnr In addition to the above, both OFS enabled development kit platforms require either an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector, or the standalone AC Power Supply with associated 2x4 power connector. You must choose one of these modes to operate your device in.
A server ATX power cable will differ between vendors - review the pinout of the power connector on the Intel Agilex\u00ae 7 FPGA I-Series Development Kit User Guide or Intel Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit User Guide as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based development kit.
In addition, the server used to house either dev kit must meet the following guidelines:
- The server platform must be able to fit, power, and cool an F-Series Dev Kit or I-Series Dev Kit as described in their mechanical details
- The server should be able to run PCIe at Gen 4.0 speeds to properly test designs and demos
Both platforms ship with an embedded Intel\u00ae FPGA Download Cable II on the rear of the device which will be used for device programming. An on-board micro-USB connector on the rear of the development kit provides the data to the Intel\u00ae MAX\u00ae 10. This allows configuration of the FPGA using a USB cable directly connected to a PC running the Intel\u00ae Quartus\u00ae Prime software without requiring the external download cable dongle. An external download cable dongle can also be used on J10 to configure the FPGA, although both cannot be used at the same time.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#20-server-settings","title":"2.0 Server Settings","text":""},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#21-bios-settings","title":"2.1 BIOS Settings","text":"These are the host BIOS settings known to work with the either dev kit. Information about the server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings.
- PCIe slot width must be set to x16
- PCIe slot speed must be set to 4
- PCIe slot must have iommu enabled
- Intel VT for Directed I/O (VT-d) must be enabled
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#22-server-fan-speed","title":"2.2 Server Fan Speed","text":"The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
- Log in to the SuperMicro server BMC. (This requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d.)
- During boot the BMC\u2019s login IP will be presented on the screen. Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case.
- On the left menu select System -> Component Info, select the Fan tab, under Advanced Settings click the circle next to Full Speed.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#30-development-kit-installation","title":"3.0 Development Kit Installation","text":""},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#31-preparing-the-f-series-development-kit-for-installation-into-a-server","title":"3.1 Preparing the F-Series Development Kit for Installation into a Server","text":"Light pipes located on the top of the QSFP cages for the F-Series Dev Kit may or may not cause physical fit issues for some server platforms. If you run into any issues during installation you may remove the light pipes:
-
The DK-DEV-AGF027F1ES (or it is called the F - tile Dev Kit, or FM86 Dev Kit) has LED light pipes on top of the QSFP cages.
These light pipes interfere with the server PCIe slot faceplate.
-
The light pipes can be easily removed by prying them off using a small screwdriver for leverage, then pushing the light pipes back to remove the retaining clips from the QSFP cage.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#32-default-switch-settings","title":"3.2 Default Switch Settings","text":"Double check that your development kit switch settings match those listed as the default positions in the user guide prior to installation. An F Tile Dev Kit is used as an example in this section.
-
Board switch definitions can be found in the Intel Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit User Guide or Intel Agilex\u00ae 7 FPGA I-Series Development Kit User Guide.
See the image below for SW1, SW4 and SW3.
Before inserting into a server, set SW5 to 'ON'.
-
Below shows an F-Series Dev Kit installed into a PCIe riser with the light pipes removed.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#33-physical-installation-procedure","title":"3.3 Physical Installation Procedure","text":"The following instructions will help ensure safe installation of the Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) into a supported server platform. Safety and Regulatory information can be found under the product page for either development kit. It is assumed you have previously removed the light pipes mounted above the F-Series Dev Kit's QSFP cages before attempting to slot into a server mounted riser if they are an issue.
- Position the board over the selected connector on the motherboard
- Press down gently and firmly seat the card in a PCIe slot. Depending on the server model being used, you may need to secure a retention screw or rotate retention clips over the development kit's faceplate.
- Do not bend the card while inserting in a slot. Do not apply too much pressure while inserting.
- Plug a standard 2x4 auxiliary power cord available from the server's ATX power supply or from the riser itself to the respective matching power connected on the board (J11). Both the PCIe slot and auxiliary PCIe power cable are required to power the entire board.
"},{"location":"hw/common/board_installation/devkit_board_installation/devkit_board_installation_guidelines/#34-jtag-setup","title":"3.4 JTAG Setup","text":"Both Development Kits have an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG. Perform the following steps to establish a JTAG connection:
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Software Installation Guide: OFS for PCIe Attach FPGAs] for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
Steps:
-
Refer to the following figure for Steps 2 and 3.
-
Locate Single DIP Switch SW2 and 4-position DIP switch SW3 on the fseries-dk. These switches control the JTAG setup for the board. Ensure that both SW2 and SW3.3 are set to ON.
-
Locate the J10 Micro-USB port on the fseries-dk. Connect a Micro-USB to USB-A cable between the J10 port and the workstation that has Quartus Prime Pro tools installed.
-
Use the jtagconfig tool to check that the JTAG chain contains the AGFB027R24C2E2VR2 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
1) Agilex F-Series FPGA Dev Kit [1-6]\n0343B0DD AGFB027R24C(.|R2|R0)/..\n020D10DD VTAP10\n
This concludes the walkthrough for establishing a JTAG connection on the fseries-dk.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/","title":"Board Installation Guidelines: IPU Platform F2000X-PL","text":"Last updated: November 19, 2024
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users prepare their server and install the IPU Platform F2000X-PL. After reviewing this document, a user shall be able to:
- Set up their server environment according to the Best Known Configuration (BKC)
- Install an F2000X device into a supported server platform
- Attach all required peripherals
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the IPU Platform F2000X-PL. The card is an acceleration development platform (ADP) intended to be used as a starting point for evaluation and development. This document will cover key topics related to server bring-up and physical platform installation, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-2-related-documentation","title":"Table 2: Related Documentation","text":"Name [Automated Evaluation User Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Software Reference Manual: Open FPGA Stack] [Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Workload Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs] [Security User Guide: Open FPGA Stack] [KVM User Guide: Open FPGA Stack] [Docker User Guide: Open FPGA Stack] [OFS 2024.1 F2000X-PL Release Notes] - under \"Important Notes\" Cyclone\u00ae 10 LP Board Management Controller (BMC) User Guide for Intel\u00ae IPU F2000X-PL v1.2.4 (Document ID: 789706) 1 1 Work with your Altera sales representative for access.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#12-server-requirements","title":"1.2 Server Requirements","text":"The following requirements must be met when purchasing a server to support the IPU Platform F2000X-PL.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#121-host-server-specifications","title":"1.2.1 Host Server Specifications","text":"The host server must meet the following minimal specifications:
- The server platform must contain at least 64GiB of RAM to to compile the Yocto or FIM images
- If using the server platform to build a Yocto image, it is recommended to have at least 200 GB of free storage space
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#122-host-bios","title":"1.2.2 Host BIOS","text":"Te Host BIOS settings known to work with the IPU Platform F2000X-PL:
- PCIe slot width must be x16
- PCIe slot speed must be 4
- PCIe slot must have iommu enabled
- Intel VT for Directed I/O (VT-d) must be enabled
Specific BIOS paths are not listed here, as they can differ between BIOS vendors and versions.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#123-host-server-kernel-and-grub-configuration","title":"1.2.3 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested:
- Ubuntu 22.04 LTS, 6.1.78-dfl
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#13-server-forced-air-cooling","title":"1.3 Server Forced Air Cooling","text":"The IPU Platform F2000X-PL is a high-performance processing card with a passive heat sink to dissipate device heat and must be installed in a server with sufficient forced airflow cooling to keep all devices operating below maximum temperature. The table below lists the thermal terms and descriptions used in thermal analysis.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-5-thermal-terms-and-descriptions","title":"Table 5: Thermal Terms and Descriptions","text":"Term Description Cubic Feet per Minute (CFM) Volumetric airflow rate, in cubic feet per minute, of air passing through faceplate. Tj FPGA Junction Temperature TLA Local Ambient temperature. Temperature of forced air as it enters the IPU Platform F2000X-PL. \u00a0 Note: In many systems, this is higher than the room ambient due to heating effects of chassis components. Note: The FPGA junction temperature must not exceed 100\u00b0C. The case temperature of the QSFP modules must meet the module vendor's specification.
Note: The table below provides the thermal targets for which the IPU Platform F2000X-PL was designed. As a card manufacturer, you must qualify your own production cards.
The maximum card inlet air temperatures must support continuous operation under the worst-case power scenario of 150W TDP.
The airflow requirements for optimal heat sink performance at minimum is characteristic of CAT 3 servers or PCIe SIG Level 7 thermal profiles, in both, forward & reverse flow, see figure below:
As the IPU Platform F2000X-PL is a development platform, it is not integrated into the server baseband management controller closed loop cooling control. It is strongly recommended that you set your server's fan settings to run constantly at 100% with the server chassis lid closed to prevent unwanted IPU Platform F2000X-PL thermal shutdown.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#14-external-connections","title":"1.4 External Connections","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-1-external-connections","title":"Figure 1: External Connections","text":"The items listed Table 6 in are known to work for external connectivity. Specific links are given for convenience, other products may be used but have not been tested.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-6-external-connection-cables","title":"Table 6: External Connection Cables","text":"Item Part Number Link to source RS-232 to USB Adapter DTECH FTDI USB to TTL Serial Adapter, 3 m USB to TTL Serial USB to Ethernet Adapter, Aluminum 3 Port USB 3.0 Teknet USB Hub with Ethernet adapter Flash Drive, 64 GB or larger SanDisk QSFP DAC Cable \u00a0FS.com Generic 2m 100G QSP28 Passive Direct Attach Copper QSFP28 DAC (optional) Intel FPGA Download Cable II PL-USB2-BLASTER USB-Blaster II"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#15-preparing-the-ipu-platform-f2000x-pl-for-installation","title":"1.5 Preparing the IPU Platform F2000X-PL for Installation","text":"Turn the board over to back side and remove the Kapton tape covering switches SW2 and SW3 and make sure the switches are set as shown in Figure 1.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-7-switch-settings","title":"Table 7: Switch Settings","text":"Name Value SW3.1 off SW3.2 off SW3.2 on SW3.2 off SW2 off"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-2-board-switch-settings","title":"Figure 2: Board Switch Settings","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#151-usb-to-serial-adapter","title":"1.5.1 USB to Serial Adapter","text":"The IPU Platform F2000X-PL has a serial UART for access located on back edge of the board. This connection is useful for making BIOS and boot settings and for monitoring the SoC. In most servers, you will need to remove a riser card and route the USB to serial cable and (optional) Intel FPGA USB Blaster through an unused PCIe slot above or below where the IPU is installed. See Figure 3 for an example of cable routing.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-3-cable-routing","title":"Figure 3: Cable Routing","text":"The USB to serial connection is shown in Figure 4 where the White wire is TXD, Black wire is ground and Green wire is RXD.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-4-usb-to-serial-adapter-connection","title":"Figure 4: USB to Serial Adapter connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#152-ipu-jtag","title":"1.5.2 IPU JTAG","text":"The IPU Platform F2000X-PL provides a 10 pin JTAG header for FPGA and Cyclone 10 Board Management Controller development work using a Intel FPGA Download Cable II. This JTAG connection is optional for initial bring-up but is useful for manual image reprogramming and debug. See Figure 5 noting the orientation of the connection. The orientation of the USB Blaster II requires careful installation in a PCIe bay that has additional room in the adjacent bay. This may require you to either install the board over the PSU of the server, or to temporarily remove an adjacent riser while programming.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-5-usb-blaster-ii-connection","title":"Figure 5: USB Blaster II Connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-6-usb-blaster-ii-installation","title":"Figure 6: USB Blaster II Installation","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#153-power","title":"1.5.3 Power","text":"The IPU Platform F2000X-PL must receive power from both the 12 V and 3.3V PCIe slot and the 12 V Auxiliary 2\u00d74 power connector. The board does not power up if any of the 12 V and 3.3 V PCIe slot, or 12 V Auxiliary power sources are disconnected.
PCIe specifications define 12 V Auxiliary power connector pin assignment. The IPU Platform F2000X-PL implements an 8-position right angle (R/A) through-hole PCB header assembly on the top right side of the board as depicted in the picture below.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-7-12v-pcie-aux-connector-location","title":"Figure 7: 12V PCIe AUX Connector Location","text":"Refer the table below for pinout details.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#table-8-12v-2x3-aux-connector-pin-out","title":"Table 8: 12V (2x3) AUX Connector Pin Out","text":"Pin Description 1 +12V 2 +12V 3 +12V 4 Sense 1 5 Ground 6 Sense 0 7 Ground 8 Ground See Auxiliary power connection in Figure 8.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-8-auxiliary-power-connection","title":"Figure 8: Auxiliary Power Connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#154-usb-hub-connection","title":"1.5.4 USB Hub Connection","text":"The USB Hub is connected to the USB type A connector on the front panel. Additionally, attach a network connected Ethernet connection to the USB hub. See Figure 9.
"},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#figure-9-usb-hub-connection","title":"Figure 9: USB Hub Connection","text":""},{"location":"hw/common/board_installation/f2000x_board_installation/f2000x_board_installation/#155-creating-a-bootable-usb-flash-drive-for-the-soc","title":"1.5.5 Creating a Bootable USB Flash Drive for the SoC","text":"Connect your flash drive to an available Linux host. In this section the USB will set up to be used as a secondary boot source for the SoC and will also be used to update the NVMe from which the ICX-D SoC boots in section 2.1 Updating the F2000X-PL ICX-D SoC NVMe.
You will load the latest pre-compiled Yocto core-image-minimal WIC image into USB flash. This image can be downloaded from ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell, under assets, or compiled from meta-ofs. Compilation is discussed in section 4.0 Compiling a Custom Yocto SoC Image.
-
Insert a 64 GB or larger USB Flash Drive into the USB slot of a computer/server you can use to format the drive. The following instructions assume you are using some flavor of GNU+Linux. You need sudo access privileges on this machine.
-
In a terminal window, find the device name of the USB flash drive and unmount the device:
$ lsblk\n\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\nsda 8:0 0 1.8T 0 disk\n\u251c\u2500sda1 8:1 0 600M 0 part /boot/efi\n\u251c\u2500sda2 8:2 0 1G 0 part /boot\n\u2514\u2500sda3 8:3 0 1.8T 0 part\n\u251c\u2500rhel-root 253:0 0 50G 0 lvm /\n\u251c\u2500rhel-swap 253:1 0 4G 0 lvm \\[SWAP\\]\n\u2514\u2500rhel-home 253:6 0 1.7T 0 lvm /home\nsdb 8:16 0 447.1G 0 disk\n\u251c\u2500sdb1 8:17 0 600M 0 part\n\u251c\u2500sdb2 8:18 0 1G 0 part\n\u2514\u2500sdb3 8:19 0 445.5G 0 part\n\u251c\u2500fedora_localhost\\--live-swap 253:2 0 4G 0 lvm\n\u251c\u2500fedora_localhost\\--live-home 253:3 0 301G 0 lvm\n\u251c\u2500fedora_localhost\\--live-root 253:4 0 70G 0 lvm\n\u2514\u2500fedora_localhost\\--live-centos_root 253:5 0 70.5G 0 lvm\nsdd 8:48 1 57.3G 0 disk\n\u2514\u2500sdd1 8:49 1 57.3G 0 part\n
In the above example, the 64 GB USB Flash device is designated sdd. Note, your device file name may be different. You are looking for an entry that matches the size of your USB Flash. You can also check the output of dmesg after manually plugging in your USB Flash device to view the name the kernel has given it in an auto-generated event.
-
Unmount the USB flash (if not already unmounted).
$ sudo umount /dev/sdd1\numount: /dev/sdd1: not mounted.\n
-
Download the Yocto WIC image. To prevent boot errors that may arise when using the same boot image loaded in both USB flash and on-board NVMe, you must choose an older version of the Yocto WIC to load onto the USB. Browse the tagged Yocto release images on GitHub and choose the second newest release image as the temporary USB boot target. In this example we will use the OFS 2023.1 RC3 release. You will also need to download the newest Yocto release image (core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.gz).
# Download an older version of the Yocto release image to use as the USB boot target, version 2023.1 RC3 shown here\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2023.1-3/core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2023.1-3/core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz.sha256\n# Verify the checksum of the downloaded image\n$ sha256sum -c https://github.com/OFS/meta-ofs/releases/download/ofs-2023.1-3/core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz.sha256\n# Uncompress the package\n$ gzip -d core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic.gz\n\n# Download the most recent Yocto release image, which will overwrite on-board NVMe\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2024.1-2/core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic\n$ wget https://github.com/OFS/meta-ofs/releases/download/ofs-2024.1-2/core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.sha256\n# Verify the checksum of the downloaded image\nsha256sum -c core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.sha256\n# Uncompress the package\n$ gzip -d core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic\n
-
Copy core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic to the USB flash. This process may take several minutes.
$ sudo dd if=core-image-full-cmdline-intel-corei7-64-20230505161810.rootfs.wic of=/dev/sdd1 bs=512k status=progress conv=sync $ sgdisk -e /dev/sdd\n
-
Create a partition to store the Yocto image, which will be used to overwrite on-board NVMe as the default boot target.
$ sudo fdisk /dev/sdd\n Command (m for help): p\n Command (m for help): n\n Partition number (4-128, default 4): <<press enter>>\n First sector (14617908-125045390, default 14618624): <<press enter>>\n Last sector, +/-sectors or +/-size{K,M,G,T,P} (14618624-125045390, default\n 125045390): <<press enter>>\n Created a new partition 4 of type 'Linux filesystem' and of size 92 GiB.\n Command (m for help): p\nCommand (m for help): w\n
-
Verify USB flash is partitioned.
$ lsblk\n NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\n sdd 8:0 1 114.6G 0 disk\n |-sdd1 8:1 1 300M 0 part\n |-sdd2 8:2 1 22.1G 0 part\n |-sdd3 8:3 1 44M 0 part\n `-sdd4 8:4 1 92.2G 0 part\n
-
Format the new partition.
$ mkfs -t ext4 /dev/sdd4\n$ mount /dev/sda4 /mnt\n
-
Copy compressed core-image-minimal WIC into /mnt.
$ cp core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic /mnt\n
Remove the USB flash from the Linux computer and install the flash drive in the USB hub attached to the IPU Platform F2000X-PL.
"},{"location":"hw/common/doc_modules/links/","title":"AFU Dev","text":""},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/","title":"Driver Development Quick Start Guidelines for Linux DFL","text":""},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#11-audience","title":"1.1 Audience","text":"The intended audience for this document is for developers looking to get starting in creating their own custom drivers with Linux DFL/DFH integration. It is assumed you have some prior knowledge of FPGA-based terminology and Linux Kernel Module development. It will be extremely helpful to have knowledge of the FPGA OPAE framework including plugins, SR-IOV, and MMIO. This information is intended as a starting point, with links where users can dive deeper.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#12-terminology","title":"1.2 Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#13-linux-dfl","title":"1.3 Linux DFL","text":"The Device Feature List is a kernel-based framework that provides support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in the PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.
A detailed explanation of the DFL framework and its parts and their functions has been upstreamed in the Linux Kernel documentation at dfl.html. Here you can find an overview of the DFL linked list structure, a breakdown of DFHv0 and DFHv1 header layouts, and introduction to the FPGA Management Engine (FME), FIU-Ports, Partial Reconfiguration, virtualization using SR-IOV, device enumeration, performance counters, interrupt support, and more. This is considered required reading before moving forward as terms and concepts will be used through the rest of the document.
There are different flavors of the DFL framework currently on offer on GitHub - an explanation of their purpose and branch structure can be found on the linux-dfl GitHub wiki pages.
If you wish to build existing DFL offerings from source, these steps are provided in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
The examples shown in this document are referencing values from the DFL implementation of a Virtual UART 8250_dfl.c, which demonstrates a minimal level implementation of a UART driver which plugs into the DFL framework. You may also find it helpful to review spi-altera-dfl.c, which similarly glues existing kernel SPI logic together with the DFL framework.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#20-information-handoff-with-workload-developer","title":"2.0 Information Handoff with Workload Developer","text":"When writing DFL enabled kernel-space drivers there are several key pieces of information you will need to gather from the RTL developer as a part of their design process. For a specific feature that is exposed to kernel-space that you are writing a driver for, you need the following:
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#table-1-hw-to-sw-id-mappings","title":"Table 1 - HW to SW ID Mappings","text":"Variable Description FME_FEATURE_ID The current value set can be seen in Device Feature List (DFL) Feature IDs. Create a pull request to merge your feature IDs with the current ID table. The RTL developer may pick any unused value for testing purposes while building their design as described in the Shell Technical Reference Manual. PCIe VID/DID/sVID/sDID The PCIe Vendor ID, Device ID, Subsystem Vendor ID, and Subsystem Device ID. A list of all values used by Altera is found in PCI IDs Containing Device Feature Lists. Information for modifying these values in your design are found in the Shell Developer Guide. param_id ID of the DFL paramater for the feature for features exposed to software in the RTL design, as set in FeatureID. Ensure the ID is not already reserved. In addition to the above, your driver logic will depend on bitmasks, register offsets, register configurations, data widths, and other general values which will be defined at the top of your driver file in the form of C preprocessor macros. These are entirely dependent on the workload being developed and will need to be communicated from the workload to the driver developer.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#30-module-definitions","title":"3.0 Module Definitions","text":"The following Macros are required when writing your own kernel driver code.
- MODULE_DEVICE_TABLE(type, name) - Describes all supported devices for this specific driver. Used to build out a table for all PCI and USB devices used by the kernel. type = FME_ID. FME_ID represents the DFL IU type which corresponds with the FPGA Management Engine. name = FME_FEATURE_ID. FME_FEATURE_ID comes from the reserved list of all FME Feature IDs. In this case, we use 0x24 for the Virtual UART. Guidance on submitting a pull request for new Feature ID(s) is found in that repository's README.
- module_dfl_driver(__dfl_driver) - Helper Macro for drivers that don't do anything special in module init/exit Macros (This replaces the module_init() and module_exit()). In this case, we pass a dfl_uart_driver struct which is an instantiation of a dfl_driver where:.drv = struct * device_driver, device_driver* drv uses the value \"dfl-uart\" as its name, which will display to the user when loaded. .id_table = dfl_uart_ids, which is an instantiation of dfl_device_id and is a 2D array of IDs from FME Feature IDs this driver should match to (in this case just 0x24) in the form of { {FME_ID, FME_FEATURE_ID}, ..., {} }. The value .guid is a DFHv1 specific matching criteria that does not require knowledge of the feature ID ahead of time. You need at least one of .guid/FME_FEATURE_ID in the dfl_device_id table. .probe = dfl_uart_probe which is the UART's probe function as locally defined. This function has a more in-depth description in section 4.1 Driver Probing. .remove = dfl_uart_remove which is the UART's removal function as locally defined. This function has a more in-depth description in section 4.2 Driver Removal.
- GUID_INIT() - describes the PCIe device IDs used to match this driver against the device(s) it should bind to. This is a built-in kernel Macro.
- MODULE_DESCRIPTION(\"DFL Intel UART driver\") - describes what a module does and has no value dependency.
- MODULE_AUTHOR(\"Intel Corporation\") - describes the author of the module and has no value dependency.
- MODULE_LICENSE(\"GPL\") - determines whether or not the module is free software or proprietary for the kernel module loader and for user space tools and must pull from a list of valid strings.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#40-commonly-used-dfl-structures-and-functions","title":"4.0 Commonly Used DFL Structures and Functions","text":"Creation of driver software involves interfacing with the same DFL-specific data structures and functions. The following is not a fully inclusive list:
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_device","title":"dfl_device*","text":"/**\n * struct dfl_device - represent an dfl device on dfl bus\n *\n * @dev: generic device interface.\n * @id: id of the dfl device.\n * @type: type of DFL FIU of the device. See enum dfl_id_type.\n * @feature_id: feature identifier local to its DFL FIU type.\n * @revision: revision of this dfl device feature.\n * @mmio_res: mmio resource of this dfl device.\n * @irqs: list of Linux IRQ numbers of this dfl device.\n * @num_irqs: number of IRQs supported by this dfl device.\n * @cdev: pointer to DFL FPGA container device this dfl device belongs to.\n * @id_entry: matched id entry in dfl driver's id table.\n * @dfh_version: version of DFH for the device\n * @param_size: size of the block parameters in bytes\n * @params: pointer to block of parameters copied memory\n */\nstruct dfl_device {\nstruct device dev;\nint id;\nu16 type;\nu16 feature_id;\nu8 revision;\nstruct resource mmio_res;\nint *irqs;\nunsigned int num_irqs;\nstruct dfl_fpga_cdev *cdev;\nconst struct dfl_device_id *id_entry;\nu8 dfh_version;\nunsigned int param_size;\nvoid *params;\n};\n
Represents a DFL device on the DFL bus. This value is automatically passed to drivers that register their devices in the Macro module_dfl_driver(__dfl_driver). The struct device dev local parameter is a pointer to the standard Linux container for devices. Other values are auto-populated in DFH by the RTL Developer as a part of workload development and exposed to you by the DFL framework.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_driver","title":"dfl_driver*","text":"/**\n * struct dfl_driver - represent an dfl device driver\n *\n * @drv: driver model structure.\n * @id_table: pointer to table of device IDs the driver is interested in.\n * { } member terminated.\n * @probe: mandatory callback for device binding.\n * @remove: callback for device unbinding.\n */\nstruct dfl_driver {\nstruct device_driver drv;\nconst struct dfl_device_id *id_table;\nint (*probe)(struct dfl_device *dfl_dev);\nvoid (*remove)(struct dfl_device *dfl_dev);\n};\n
Represents a DFL device driver as required by the kernel. The struct device_driver drv local parameter is a pointer to the standard Linux device_driver. Importantly it is up to the user to populate the dfl_device_id table, and provide a pointer to both the probe and remove functions for their DFL driver. dfl_device_id is an upstreamed data structure which identifies DFL devices to the kernel.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_device_id","title":"dfl_device_id*","text":"/**\n * struct dfl_device_id - dfl device identifier\n * @type: DFL FIU type of the device. See enum dfl_id_type.\n * @feature_id: feature identifier local to its DFL FIU type.\n * @driver_data: driver specific data.\n */\nstruct dfl_device_id {\n__u16 type;\n__u16 feature_id;\nkernel_ulong_t driver_data;\n};\n
Similarly, dfl_device_id is an upstreamed data structure used to identify devices as part of the DFL bus to the kernel.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#dfl_find_param","title":"dfl_find_param()","text":"/**\n * dfh_find_param() - find parameter block for the given parameter id\n * @dfl_dev: dfl device\n * @param_id: id of dfl parameter\n * @psize: destination to store size of parameter data in bytes\n *\n * Return: pointer to start of parameter data, PTR_ERR otherwise.\n */\nvoid *dfh_find_param(struct dfl_device *dfl_dev, int param_id, size_t *psize)\n
This function will find and return the value of a parameter as identified by the dfl_dev and param_id. This functions similarly to the PCIe find capability in that it scans PCIe bar space for the correct DFL information offsets to read off feature capabilities. Ultimately this function relies on the Linux Kernel Macro FIELD_GET(mask, reg) to extract a value from a hardware register give a bitmask and register value. This function iterates through all DFH offsets and returns NULL if it is unable to locate the param_id requested.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#50-probe-and-remove-functions-using-dfh","title":"5.0 Probe and Remove Functions Using DFH","text":"Every DFL driver requires a probe and removal function to be defined for matching devices. The examples shown here are referencing values from the DFL implementation of a Virtual UART 8250_dfl.c, which demonstrates a minimal level implementation of a UART driver which plugs into the DFL framework.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#51-driver-probing","title":"5.1 Driver Probing","text":"dfl_uart_probe() = The DFL enabled vUART probe function. A probe function is passed a struct * pcie_dev for any devices matching its ID table in most cases. Because we have registered our device as a dfl_uart_driver using module_dfl_driver_() Macro, we are instead passed a dfl_device * dev. In this example we use the pre-existing Serial 8250 driver implementation in the kernel while also plugging the device into the DFL framework.
static int dfl_uart_probe(struct dfl_device *dfl_dev)\n
To probe our device we need to accomplish the following:
- Allocate the memory in the kernel for our device driver using static inline void *devm_kzalloc(struct device *dev, size_t size, gfp_t gfp)
- Initialize our serial port using serial8250_register_8250_port() which will ready our port and return the line number.
- Set a pointer to this driver's current state using dev_set_drvdata(struct device *dev, void *data)
- Return 0 if the driver probe has completed without error.
Initializing the serial port is the only vUART specific function in this flow and can be replaced with the initialization functionality of the driver you are implementing.
Because the probe function is passed a (dfl_dev ) struct we have access to DFL specific functions for accessing device parameters via the Device Feature Header (DFH) structure. The DFL function (void *dfh_find_param(struct dfl_device *dfl_dev, int param_id, size_t *psize) can be used to read parameters as defined in the DFH given a **param_id*. This information allows us to read configuration information for our driver directly from DFH registers and pass it along to our initialization function.
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#52-driver-removal","title":"5.2 Driver Removal","text":"dfl_uart_remove() is the DFL enabled vUART driver remove function. The logic in this function is related to the closure of any non-DFL structures from the kernel used by our driver. In this case we only need to worry about freeing the UART serial line that was previously claimed by our driver. If you have no logic tied to the removal of our driver this function is not necessary; the kernel will free memory automatically because of the work we put in to integrating our software with the driver framework.
static void dfl_uart_remove(struct dfl_device *dfl_dev)\n{\nstruct dfl_uart *dfluart = dev_get_drvdata(&dfl_dev->dev);\nserial8250_unregister_port(dfluart->line);\n}\n
"},{"location":"hw/common/reference_manual/driver_dev_qs/driver_dev_qs/#60-driver-skeleton-example","title":"6.0 Driver Skeleton Example","text":"This example provided below instantiates a DFL compatible driver without any feature-specific logic. This code will build an out-of-tree module and insert into a kernel, although without modification it doesn't accomplish anything. It will also not register itself with any known devices. Comments are provided in-line on where you can add in extra logic for your feature's use case, alongside some printk statements which will display in dmesg after the driver has been build and inserted with insmod <file>.ko.
You will need to install the Linux DFL driver framework before attempting to build as shown in Software Installation Guide: Open FPGA Stack for PCIe Attach alongside linux-headers-, linux-devel-, and any other minimal requirements. If you do not have any DFL compatible devices installed on your build machine, you will need to load the DFL kernel module code with sudo modprobe dfl before building.
- Create a file skeleton.c and copy the following:
#include <linux/bitfield.h>\n#include <linux/device.h>\n#include <linux/dfl.h>\n#include <linux/errno.h>\n#include <linux/ioport.h>\n#include <linux/module.h>\n#include <linux/mod_devicetable.h>\n#include <linux/types.h>\n/*\nInsert all build dependent #includes.\n*/\n/*\nInsert all feature-specific Macros that will be referenced as a part of your kernel probe, removal, and logic implementation. These values can be pulled from the RTL developer's build of your design and vary depending on requirements.\n*/\nstatic int dfh_get_u64_param_val(struct dfl_device *dfl_dev, int param_id, u64 *pval)\n{\nsize_t psize;\nu64 *p;\np = dfh_find_param(dfl_dev, param_id, &psize);\nif (IS_ERR(p))\nreturn PTR_ERR(p);\nif (psize != sizeof(*pval))\nreturn -EINVAL;\n*pval = *p;\nreturn 0;\n}\n/*\nUsing dfh_get_u64_param_val above, create a function that reads all relevant information from DFH for your design.\n*/\n/*\nYou will need to insert driver probe logic for your specific feature in this function. We use a stand-in struct 'driver_data' with bogus data.\n*/\nstruct driver_data {\nint field1;\nint field2;\nunsigned char field3;\nu64 size;\nu32 capabilities;\n};\nstatic int dfl_skeleton_probe(struct dfl_device *dfl_dev)\n{\nstruct device *dev = &dfl_dev->dev;\nstruct driver_data *data;\ndata = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);\nif (!data) {\nreturn -ENOMEM;\n}\nint ret;\ndev_set_drvdata(dev, data);\nprintk(KERN_INFO \"DFL Skeleton Driver probe function has completed!\\n\");\nreturn 0;\n}\nstatic void dfl_skeleton_remove(struct dfl_device *dfl_dev)\n{\nprintk(KERN_INFO \"DFL Skeleton Driver probe remove has completed!\\n\");\n}\n#define FME_FEATURE_ID_SKELETON 0x27\n//#define FME_GUIDS \\\n// GUID_INIT() \n// This section commented out to prevent build errors. Fill this in with your device's PCIe addressing information.\n// Add FME_GUIDS as the third item after FME_FEATURE_ID_SKELETON if you have used the Kernel Macro\nstatic const struct dfl_device_id dfl_driver_ids[] = {\n{ FME_ID, FME_FEATURE_ID_SKELETON },\n{ }\n};\nMODULE_DEVICE_TABLE(dfl, dfl_driver_ids);\nstatic struct dfl_driver dfl_skeleton_driver = {\n.drv = {\n.name = \"dfl-skeleton\",\n},\n.id_table = dfl_driver_ids,\n.probe = dfl_skeleton_probe,\n.remove = dfl_skeleton_remove,\n};\nmodule_dfl_driver(dfl_skeleton_driver);\nMODULE_DESCRIPTION(\"DFL Skeleton driver\");\nMODULE_AUTHOR(\"Your Name\");\nMODULE_LICENSE(\"GPL\");\n
- Create the below Makefile in the same directory as your driver code and build with
make:
obj-m += skeleton.o\n\nall:\nmake -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules\n\nclean:\nmake -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/","title":"Software Reference Manual: Open FPGA Stack","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#11-audience","title":"1.1 Audience","text":"The information presented in this document is intended to be used by software developers looking to increase their knowledge of the OPAE SDK user-space software stack and the kernel-space linux-dfl drivers. This information is intended as a starting point, with links to where users can deep dive on specific topics.
Former OPAE and DFL software documents were combined with the Software Reference Manual to reduce clutter and more cleanly document OFS software capabilities. The following documents are no longer available as standalone as of ofs-2023.3:
- Quick Start Guide
- OPAE Installation Guide
- OPAE C API Programming Guide
- OPAE Python Bindings
- OPAE Plugin Developers Guide
- Open Programmable Accelerator Engine (OPAE) Linux Device Driver Architecture
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#12-terminology","title":"1.2 Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#13-opae-software-development-kit-sdk","title":"1.3 OPAE Software Development Kit (SDK)","text":"The OPAE C library is a lightweight user-space library that provides abstraction for FPGA resources in a compute environment. Built on top of the OPAE Intel\u00ae FPGA driver stack that supports Intel\u00ae FPGA platforms, the library abstracts away hardware specific and OS specific details and exposes the underlying FPGA resources as a set of features accessible from within software programs running on the host. The OPAE source code is available on the OPAE SDK repository.
By providing a unified C API, the library supports different FPGA integration and deployment models, ranging from single-node systems with one or a few FPGA devices to large-scale FPGA deployments in a data center. At one end of the spectrum, the API supports a simple application using a PCIe link to reconfigure the FPGA with different accelerator functions. At the other end of the spectrum, resource management and orchestration services in a data center can use this API to discover and select FPGA resources and then allocate them for use by acceleration workloads.
The OPAE provides consistent interfaces to crucial components of the platform. OPAE does not constrain frameworks and applications by making optimizations with limited applicability. When the OPAE does provide convenience functions or optimizations, they are optional. For example, the OPAE provides an interface to allocate physically contiguous buffers in system memory that user-space software and an accelerator can share. This interface enables the most basic feature set of allocating and sharing a large page of memory in one API call. However, it does not provide a malloc()-like interface backed by a memory pool or slab allocator. Higher layers of the software stack can make such domain-specific optimizations.
Most of the information related to OPAE can be found on the official Open FPGA Stack (OFS) Collateral Site and in the OPAE SDK repository. The following is a summary of the information present on this web page:
- Configuration options present in the OPAE SDK build and installation flow
- The steps required to build a sample OPAE application
- An explanation of the basic application flow
- A reference for the C, C++, and Python APIs
- An explanation of the OPAE Linux Device Driver Architecture
- Definitions for the various user-facing OPAE SDK tools
The remaining sections on OPAE in this document are unique and build on basic principles explained in opae.github.io.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#table-1-additional-websites-and-links","title":"Table 1: Additional Websites and Links","text":"Document Link OPAE SDK on github OPAE SDK repository OPAE Documents Open FPGA Stack (OFS) Collateral Site pybind11 https://pybind11.readthedocs.io/en/stable/ CLI11 https://github.com/CLIUtils/CLI11 spdlog https://github.com/gabime/spdlog"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#table-2-ofs-hardware-terminology","title":"Table 2: OFS Hardware Terminology","text":"Term Description FPGA: Field Programmable Gate Array a discrete or integrated device connecting to a host CPU via PCIe or other type of interconnects. Accelerator Function Unit (AFU) The AFU is the supplied implementation of an accelerator, typically in HDL. AFUs implement a function such as compression, encryption, or mathematical operations.The Quartus Prime Pro software synthesizes the RTL logic into a bitstream. Accelerator Function (AF) The AF is the compiled binary for an AFU. An AF is a raw binary file (.rbf) bitstream. A tool (fpgaconf) reconfigures the FPGA using an AF bitstream. Reconfiguration The process of reprogramming the FPGA with a different AF."},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#20-opae-c-api","title":"2.0 OPAE C API","text":"The following OPAE data structures and functions integrate AFUs into the OPAE environment. The OPAE C API models these data structures and functions. For more information on the object models refer to the Object model section.
- Accelerator: An accelerator is an allocable accelerator function implemented in an FPGA. An accelerator tracks the ownership of an AFU (or part of it) for a process that uses it. Multiple processes can share an accelerator.
- Device: The OPAE enumerates and models two device types: the FPGA and the AFU.
- Events: Events are asynchronous notifications. The FPGA driver triggers particular events to indicate error conditions. Accelerator logic can also define its own events. User applications can choose to be notified when particular events occur and respond appropriately.
- Shared memory buffers: Software allocates shared memory buffers in user process memory on the host. Shared memory buffers facilitate data transfers between the user process and the accelerator that it owns.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#21-libopae-c","title":"2.1 libopae-c","text":"Linking with this library is straightforward. Code using the OPAE library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, here is the simplest compile and link command:
gcc myprog.c -I</path/to/fpga.h> -L</path/to/libopae-c.so> -lopae-c -luuid -ljson-c -lpthread
.. note::
The OPAE library uses the third-party `libuuid` and `libjson-c` libraries that are not distributed with \nthe OPAE library. Make sure to install these libraries.\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#sample-code","title":"Sample Code","text":"The library source includes two code samples. Use these samples to learn how to call functions in the library. Build and run these samples to determine if your installation and environment are set up properly.
Refer to the Running the Hello FPGA Example chapter in the Intel\u00ae Acceleration Stack Quick Start Guide for for Intel Programmable Acceleration Card with Intel Arria\u00ae 10 GX FPGA for more information about using the sample code.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#high-level-directory-structure","title":"High-Level Directory Structure","text":"Building and installing the OPAE library results in the following directory structure on the Linux OS. Windows and MacOS have similar directories and files.
Directory & Files Contents include/opae Directory containing all header files include/opae/fpga.h Top-level header for user code to include include/opae/access.h Header file for accelerator acquire/release, MMIO, memory management, event handling, and so on include/opae/bitstream.h Header file for bitstream manipulation functions include/opae/common.h Header file for error reporting functions include/opae/enum.h Header file for AFU enumeration functions include/opae/manage.h Header file for FPGA management functions include/opae/types.h Various type definitions lib Directory containing shared library files lib/libopae-c.so The shared dynamic library for linking with the user application doc Directory containing API documentation doc/html Directory for documentation of HTML format doc/latex Directory for documentation of LaTex format doc/man Directory for documentation of Unix man page format"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#object-models","title":"Object Models","text":" fpga_objtype: An enum type that represents the type of an FPGA resource, either FPGA_DEVICE or FPGA_ACCELERATOR. An FPGA_DEVICE object corresponds to a physical FPGA device. Only FPGA_DEVICE objects can invoke management functions. The FPGA_ACCELERATOR represents an instance of an AFU. fpga_token: An opaque type that represents a resource known to, but not necessarily owned by, the calling process. The calling process must own a resource before it can invoke functions of the resource.fpga_handle: An opaque type that represents a resource owned by the calling process. The API functions fpgaOpen() and fpgaClose() acquire and release ownership of a resource that an fpga_handle represents. (Refer to the Functions section for more information.)fpga_properties: An opaque type for a properties object. Your applications use these properties to query and search for appropriate resources. The FPGA Resource Properties section documents properties visible to your applications.fpga_event_handle: An opaque handle the FPGA driver uses to notify your application about an event. fpga_event_type: An enum type that represents the types of events. The following are valid values: FPGA_EVENT_INTERRUPT, FPGA_EVENT_ERROR, and FPGA_EVENT_POWER_THERMAL. (The Intel Programmable Acceleration Card (PAC) with Intel Arria 10 GX FPGA does not handle thermal and power events.)fpga_result: An enum type to represent the result of an API function. If the function returns successfully the result is FPGA_OK. Otherwise, the result is the appropriate error codes. Function fpgaErrStr() translates an error code into human-readable strings.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#functions","title":"Functions","text":"The table below groups important API calls by their functionality. For more information about each of the functions, refer to the OPAE C API reference manual.
Functionality API Call FPGA Accelerator Description Enumeration fpgaEnumerate() Yes Yes Query FPGA resources that match certain properties Enumeration: Properties fpga[Get, Update, Clear, Clone, Destroy Properties]() Yes Yes Manage fpga_properties life cycle fpgaPropertiesGet[Prop]() Yes Yes Get the specified property Prop, from the FPGA Resource Properties table fpgaPropertiesSet[Prop]() Yes Yes Set the specified property Prop, from the FPGA Resource Properties table Access: Ownership fpga[Open, Close]() Yes Yes Acquire/release ownership Access: Reset fpgaReset() Yes Yes Reset an accelerator Access: Event handling fpga[Register, Unregister]Event() Yes Yes Register/unregister an event to be notified about fpga[Create, Destroy]EventHandle() Yes Yes Manage fpga_event_handle life cycle Access: MMIO fpgaMapMMIO(), fpgaUnMapMMIO() Yes Yes Map/unmap MMIO space fpgaGetMMIOInfo() Yes Yes Get information about the specified MMIO space fpgaReadMMIO[32, 64]() Yes Yes Read a 32-bit or 64-bit value from MMIO space fpgaWriteMMIO[32, 64]() Yes Yes Write a 32-bit or 64-bit value to MMIO space Memory management: Shared memory fpga[Prepare, Release]Buffer() Yes Yes Manage memory buffer shared between the calling process and an accelerator fpgaGetIOAddress() Yes Yes Return the device I/O address of a shared memory buffer fpgaBindSVA() Yes Yes Bind IOMMU shared virtual addressing Management: Reconfiguration fpgaReconfigureSlot() Yes No Replace an existing AFU with a new one Error report fpgaErrStr() Yes Yes Map an error code to a human readable string .. note::
The UMsg APIs are not supported for the Intel(R) PAC cards. They will be deprecated in a future release.\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#fpga-resource-propertie","title":"FPGA Resource Propertie","text":"Applications query resource properties by specifying the property name for Prop in the fpgaPropertiesGet[Prop]() and fpgaPropertiesSet[Prop]() functions. The FPGA and Accelerator columns state whether or not the Property is available for the FPGA or Accelerator objects.
Property FPGA Accelerator Description Parent No Yes fpga_token of the parent object ObjectType Yes Yes The type of the resource: either FPGA_DEVICE or FPGA_ACCELERATOR Bus Yes Yes The bus number Device Yes Yes The PCI device number Function Yes Yes The PCI function number SocketId Yes Yes The socket ID DeviceId Yes Yes The device ID NumSlots Yes No Number of AFU slots available on an FPGA_DEVICE resource BBSID Yes No The FPGA Interface Manager (FIM) ID of an FPGA_DEVICE resource BBSVersion Yes No The FIM version of an FPGA_DEVICE resource VendorId Yes No The vendor ID of an FPGA_DEVICE resource GUID Yes Yes The GUID of an FPGA_DEVICE or FPGA_ACCELERATOR resource NumMMIO No Yes The number of MMIO space of an FPGA_ACCELERATOR resource NumInterrupts No Yes The number of interrupts of an FPGA_ACCELERATOR resource AcceleratorState No Yes The state of an FPGA_ACCELERATOR resource: either FPGA_ACCELERATOR_ASSIGNED or FPGA_ACCELERATOR_UNASSIGNED"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#opae-c-api-return-codes","title":"OPAE C API Return Codes","text":"The OPAE C library returns a code for every exported public API function. FPGA_OK indicates successful completion of the requested operation. Any return code other than FPGA_OK indicates an error or unexpected behavior. When using the OPAE C API, always check the API return codes.
Error Code Description FPGA_OK Operation completed successfully FPGA_INVALID_PARAM Invalid parameter supplied FPGA_BUSY Resource is busy FPGA_EXCEPTION An exception occurred FPGA_NOT_FOUND A required resource was not found FPGA_NO_MEMORY Not enough memory to complete operation FPGA_NOT_SUPPORTED Requested operation is not supported FPGA_NO_DRIVER Driver is not loaded FPGA_NO_DAEMON FPGA Daemon (fpgad) is not running FPGA_NO_ACCESS Insufficient privileges or permissions FPGA_RECONF_ERROR Error while reconfiguring FPGA"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#211-device-abstraction","title":"2.1.1 Device Abstraction","text":"The OPAE C API relies on two base abstractions concerning how the FIM and accelerator are presented to and manipulated by the user. The FIM is concerned with management functionality. Access to the FIM and its interfaces is typically restricted to privileged (root) users. The accelerator contains the user-defined logic in its reconfigurable region. Most OPAE end-user applications are concerned with querying and opening the accelerator device, then interacting with the AFU via MMIO and shared memory.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2111-device-types","title":"2.1.1.1 Device types","text":"The C enum fpga_objtype defines two variants. The FPGA_DEVICE variant corresponds to the FIM portion of the device, and the FPGA_ACCELERATOR refers to the accelerator, also known as the AFU.
An FPGA_DEVICE refers loosely to the sysfs tree rooted at the dfl-fme.X directory, for example /sys/class/fpga_region/region0/dfl-fme.0, and its associated device file /dev/dfl-fme.0.
An FPGA_ACCELERATOR refers loosely to the sysfs tree rooted at the dfl-port.X directory, for example /sys/class/fpga_region/region0/dfl-port.0, and its associated device file /dev/dfl-port.0.
The number X in dfl-fme.X and dfl-port.X refers to a numeric ID that is assigned by the DFL device driver to uniquely identify an instance of the FIM/Accelerator. Systems with multiple FPGA acceleration devices will have multiple dfl-fme.X\u2019s and matching dfl-port.X\u2019s.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2112-tokens-and-handles","title":"2.1.1.2 Tokens and Handles","text":"An fpga_token is an opaque data structure that uniquely represents an FPGA_DEVICE or an FPGA_ACCELERATOR. Tokens convey existence, but not ownership. Tokens are retrieved via the OPAE enumeration process described below using the fpgaEnumerate() call.
An fpga_handle is an opaque data structure that corresponds to an opened device instance, whether FPGA_DEVICE or FPGA_ACCELERATOR. A Handle is obtained from a token via the fpgaOpen() call. A handle conveys that the /dev/dfl-fme.X or /dev/dfl-port.X device file has been opened and is ready for interaction via its IOCTL interface.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#212-enumeration","title":"2.1.2 Enumeration","text":"Enumeration is the process by which an OPAE application becomes aware of the existence of FPGA_DEVICE\u2019s and FPGA_ACCELERATOR\u2019s. Refer to the signature of the fpgaEnumerate() call:
fpga_result fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters,\nfpaa_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n
Figure 1 fpgaEnumerate()
The typical enumeration flow involves an initial call to fpgaEnumerate() to discover the number of available tokens.
uint32_t num_matches = 0;\nfpgaEnumerate(NULL, 0, NULL, 0, &num_matches);\n
Figure 2 Discovering Number of Tokens
Once the number of available tokens is known, the application can allocate the correct amount of space to hold the tokens:
fpga_token *tokens;\nuint32_t num_tokens = num_matches;\ntokens = (fpga_token *)calloc(num_tokens, sizeof(fpga_token));\nfpgaEnumerate(NULL, 0, tokens, num_tokens, &num_matches);\n
Figure 3 Enumerating All Tokens
Note that parameters filters and num_filters were not used in the preceding example, as they were NULL and 0. When no filtering criteria are provided, fpgaEnumerate() returns all tokens that can be enumerated.
The function will assert a reset on the currently loaded AFU for a selected device through an ioctl call on its VFIO_GROUP.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2121-fpga_properties-and-filtering","title":"2.1.2.1 fpga_properties and Filtering","text":"An fpga_properties is an opaque data structure used to retrieve all of the properties concerning an FPGA_DEVICE or FPGA_ACCELERATOR. These properties can be included in the filters parameter to fpgaEnumerate() to select tokens by specific criteria.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#21211-common-properties","title":"2.1.2.1.1 Common Properties","text":"Table 3 lists the set of properties that are common to FPGA_DEVICE and FPGA_ACCELERATOR:
Property Description fpga_guid guid; FPGA_DEVICE: PR Interface ID FPGA_ACCELERATOR: AFU ID fpga_token parent; FPGA_DEVICE: always NULL FPGA_ACCELERATOR: the token of the corresponding FPGA_DEVICE, if any. Otherwise, NULL. fpga_objtype objtype; FPGA_DEVICE or FPGA_ACCELERATOR uint16_t segment; The segment portion of the PCIe address: ssss:bb:dd.f uint8_t bus; The bus portion of the PCIe address:
ssss:bb:dd.f
uint8_t device; The device portion of the PCIe address:
ssss:bb:dd.f
uint8_t function; The function portion of the PCIe address: ssss:bb:dd.f uint64_t object_id; A unique 64-bit value that identifies this token on the system. uint16_t vendor_id; The PCIe Vendor ID uint16_t device_id; The PCIe Device ID uint32_t num_errors; The number of error sysfs nodes available for this token. fpga_interface interface; An identifier for the underlying plugin-based access method. Table 3 Common Properties
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#21212-fpga_device-properties","title":"2.1.2.1.2 FPGA_DEVICE Properties","text":"Table 4 lists the set of properties that are specific to FPGA_DEVICE token types.
Property Description uint64_t bbs_id; FIM-specific Blue Bitstream ID fpga_version bbs_version; BBS version Table 4 FPGA_DEVICE Properties
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#21213-fpga_accelerator-properties","title":"2.1.2.1.3 FPGA_ACCELERATOR Properties","text":"Table 5 lists the set of properties that are specific to FPGA_ACCELERATOR token types.
Property Description fpga_accelerator_state state; Whether the Accelerator is currently open uint32_t num_mmio; The number of MMIO regions available uint32_t num_interrupts; The number of interrupts available Table 5 FPGA_ACCELERATOR Properties
Following is an example of using fpga_properties to enumerate a specific AFU:
#define NLB0_AFU \"D8424DC4-A4A3-C413-F89E-433683F9040B\"\nfpga_properties filter = NULL;\nfpga_guid afu_id;\nfpgaGetProperties(NULL, &filter); // NULL: a new empty properties\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nuuid_parse(NLB0_AFU, afu_id);\nfpgaPropertiesSetGuid(filter, afu_id);\nfpgaEnumerate(&filter, 1, tokens, num_tokens, &num_matches);\n
Relevant Links: - fpga_guid - fpgaGetProperties - fpgaPropertiesSetObjectType - fpgaPropertiesSetGUID Figure 4 Filtering During Enumeration
Note that fpga_properties and fpga_token\u2019s are allocated resources that must be freed by their respective API calls, ie fpgaDestroyProperties() and fpgaDestroyToken().
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#213-access","title":"2.1.3 Access","text":"Once a token is discovered and returned to the caller by fpgaEnumerate(), the token can be converted into a handle by fpgaOpen(). Upon a successful call to fpgaOpen(), the associated /dev/dfl-fme.X (FPGA_DEVICE) or /dev/dfl-port.X (FPGA_ACCELERATOR) is opened and ready for use. Having acquired an fpga_handle, the application can then use the handle with any of the OPAE APIs that require an fpga_handle as an input parameter.
Like tokens and properties, handles are allocated resources. When a handle is no longer needed, it should be closed and released by calling fpgaClose().
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#214-events","title":"2.1.4 Events","text":"Event registration in OPAE is a two-step process. First, the type of event must be identified. The following fpga_event_type variants are defined:
Event Description FPGA_EVENT_INTERRUPT AFU interrupt FPGA_EVENT_ERROR Infrastructure error event (FME/Port Error) Table 6 FPGA Event Types
Once the desired event type is known, an fpga_event_handle is created via fpgaCreateEventHandle(). Once the event handle is available, the event notification is registered using fpgaRegisterEvent(). In the example below, note the use of the flags field for passing the desired IRQ vector when the event type is FPGA_EVENT_INTERRUPT. With the event registered, the application can then use fpgaGetOSObjectFromEventHandle() to obtain a file descriptor for use with the poll() system call. When the interrupt occurs, the file descriptor will be set to the signaled state by the DFL driver.
fpga_event_handle event_handle = NULL;\nint fd = -1;\nfpgaCreateEventHandle(&event_handle);\nfpgaRegisterEvent(fpga_handle, FPGA_EVENT_INTERRUPT,\nevent_handle, irq_vector);\nfpgaGetOSObjectFromEventHandle(event_handle, &fd);\n
Figure 5 Creating and Registering Events
When an event notification is no longer needed, it should be released by calling fpgaUnregisterEvent(). Like device handles, event handles are allocated resources that must be freed when no longer used. To free an event handle, use the fpgaDestroyEventHandle() call.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#215-mmio-and-shared-memory","title":"2.1.5 MMIO and Shared Memory","text":"Communication with the AFU is achieved via reading and writing CSRs and by reading and writing to AFU/host shared memory buffers. An AFU\u2019s CSRs are memory-mapped into the application process address space by way of the fpgaMapMMIO() call.
uint32_t mmio_num = 0;\nfpgaMapMMIO(fpga_handle, mmio_num, NULL);\nfpgaWriteMMIO64(fpga_handle, mmio_num, MY_CSR, 0xa);\n
Figure 6 Mapping and Accessing CSRs
The second parameter, mmio_num, is the zero-based index identifying the desired MMIO region. The maximum number of MMIO regions for a particular handle is found by accessing the num_mmio property. Refer to the fpgaPropertiesGetNumMMIO() call.
Once the AFU CSRs are mapped into the process address space, the application can use the fpgaReadMMIO**XX**() and fpgaWriteMMIO**XX**() family of functions, eg fpgaReadMMIO64() and fpgaWriteMMIO64(). When an MMIO region is no longer needed, it should be unmapped from the process address space using the fpgaUnmapMMIO() call.
Shared memory buffers are allocated by way of the fpgaPrepareBuffer() call.
fpga_result fpgaPrepareBuffer(fpga_handle handle,\nuint64_t len,\nvoid **buf_addr,\nuint64_t *wsid,\nint flags);\n
Figure 7 fpgaPrepareBuffer()
Three buffer lengths are supported by this allocation method:
Length Description 4096 (4KiB) No memory configuration needed. 2097152 (2MiB) Requires 2MiB huge pages to be allocated. 1073741824 (1GiB) Requires 1GiB huge pages to be allocated. Table 7 fpgaPrepareBuffer() Lengths
echo 8 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages\necho 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages\n
Figure 8 Configuring Huge Pages
The buf_addr parameter to fpgaPrepareBuffer() is a pointer to a void * that accepts the user virtual base address of the newly-created buffer. The wsid parameter is a pointer to a uint64_t that receives a unique workspace ID for the buffer allocation. This workspace ID is used in subsequent calls to fpgaReleaseBuffer(), which should be called when the buffer is no longer needed and in calls to fpgaGetIOAddress() which is used to query the IO base address of the buffer. The IO base address can be programmed into the AFU by means of the AFU CSR space. For example, here is a code snippet from the hello_fpga sample that demonstrates programming a shared buffer\u2019s IO base address into an AFU CSR in MMIO region 0:
#define LOG2_CL 6\n#define CACHELINE_ALIGNED_ADDR(p) ((p) >> LOG2_CL)\nfpgaGetIOAddress(accelerator_handle, input_wsid, &iova);\nfpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_SRC_ADDR,\nCACHELINE_ALIGNED_ADDR(iova));\n
Figure 9 Programming Shared Memory
If applications need to map a shared buffer that has been allocated by some other means than fpgaPrepareBuffer(), then the flags parameter can be set to FPGA_BUF_PREALLOCATED. This causes fpgaPrepareBuffer() to skip the allocation portion of the call and to only memory map the given buf_addr into the application process address space.
Buffers can also be allocated and mapped as read-only by specifying FPGA_BUF_READ_ONLY.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#216-management","title":"2.1.6 Management","text":"The management feature in OPAE concerns re-programming the programmable region of the Port. To program the Port bitstream, pass a handle to the FPGA_DEVICE associated with the desired Port. The slot parameter identifies which Port to program in the case of multi-port implementations. Most designs will only pass zero as the slot parameter. The bitstream parameter is a buffer that contains the entire bitstream contents, including the JSON bitstream header information. The bitstream_len field gives the length of bitstream in bytes.
fpgaReconfigureSlot() first checks whether the FPGA_ACCELERATOR corresponding to the FPGA_DEVICE in fme_handle is open. If it is open, then the programming request is aborted with an error code. The application may pass FPGA_RECONF_FORCE in the flags parameter in order to avoid this open check and forcefully program the bitstream.
fpga_result fpgaReconfigureSlot(fpga_handle fme_handle,\nuint32_t slot,\nconst uint8_t *bitstream,\nsize_t bitstream_len,\nint flags);\n
Figure 10 fpgaReconfigureSlot()
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#217-errors","title":"2.1.7 Errors","text":"The OPAE errors API provides a means to query and clear both FPGA_DEVICE and FPGA_ACCELERATOR errors. Each FPGA device exports a collection of error registers via the DFL drivers\u2019 sysfs tree, for both the FME and the Port. Each register is typically an unsigned 64-bit mask of the current errors, where each bit or some collection of bits specifies an error type. An error is signaled if its bit or collection of bits is non-zero. Note that the 32-bit error index may vary from one process execution to the next. Applications should use fpgaGetErrorInfo() and examine the error name returned in the struct fpga_error_info to identify the desired 64-bit error mask.
struct fpga_error_info {\nchar name[FPGA_ERROR_NAME_MAX];\nbool can_clear;\n};\n
Figure 11 struct fpga_error_info
Each 64-bit mask of errors is assigned a unique 32-bit integer index and a unique name. Given an fpga_token and an error index, fpgaGetErrorInfo() retrieves the struct fpga_error_info corresponding to the error.
fpga_result fpgaGetErrorInfo(fpga_token token,\nuint32_t error_num,\nstruct fpga_error_info *error_info);\n
Figure 12 fpgaGetErrorInfo()
fpgaReadError() provides access to the raw 64-bit error mask, given the unique error index. fpgaClearError() clears the errors for a particular index. fpgaClearAllErrors() clears all the errors for the given fpga_token.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#218-metrics","title":"2.1.8 Metrics","text":"The OPAE metrics API refers to a group of functions and data structures that allow querying the various device metrics from the Board Management Controller component of the FPGA device. A metric is described by an instance of struct fpga_metric_info.
typedef struct fpga_metric_info {\nuint64_t metric_num;\nfpga_guid metric_guid;\nchar qualifier_name[FPGA_METRIC_STR_SIZE];\nchar group_name[FPGA_METRIC_STR_SIZE];\nchar metric_name[FPGA_METRIC_STR_SIZE];\nchar metric_units[FPGA_METRIC_STR_SIZE];\nenum fpga_metric_datatype metric_datatype;\nenum fpga_metric_type metric_type;\n} fpga_metric_info;\n
Relevant Links: - fpga_metric_datatype - fpga_metric_type Figure 13 fpga_metric_info
The group_name field holds a string describing the broad categorization of the metric. Some sample values for group_name are \u201cthermal_mgmt\u201d and \u201cpower_mgmt\u201d. The metric_name field contains the metric\u2019s name. The number and names of metrics may vary from one FPGA platform to the next. The qualifier_name field is a concatenation of group_name and metric_name, with a colon character in between. The metric_units field contains the string name of the unit of measurement for the specific metric. Some examples for metric_units are \u201cVolts\u201d, \u201cAmps\u201d, and \u201cCelsius\u201d.
The metric_datatype field uniquely identifies the underlying C data type for the metric\u2019s value:
enum fpga_metric_datatype {\nFPGA_METRIC_DATATYPE_INT,\nFPGA_METRIC_DATATYPE_FLOAT,\nFPGA_METRIC_DATATYPE_DOUBLE,\nFPGA_METRIC_DATATYPE_BOOL,\nFPGA_METRIC_DATATYPE_UNKNOWN\n};\n
Figure 14 enum fpga_metric_datatype
The metric_type field classifies the metric into a broad category. This information is redundant with the group_name field.
enum fpga_metric_type {\nFPGA_METRIC_TYPE_POWER,\nFPGA_METRIC_TYPE_THERMAL,\nFPGA_METRIC_TYPE_PERFORMANCE_CTR,\nFPGA_METRIC_TYPE_AFU,\nFPGA_METRIC_TYPE_UNKNOWN\n};\n
Figure 15 enum fpga_metric_type
In order to enumerate the information for each of the metrics available from the FPGA device, determine the number of metrics using fpgaGetNumMetrics().
uint64_t num_metrics = 0;\nfpgaGetNumMetrics(handle, &num_metrics);\n
Figure 16 Determining Number of Metrics
This call retrieves the number of available metrics for the FPGA_DEVICE that is opened behind the handle parameter to the call. Refer to 2.1.3 Access for information about the fpgaOpen() call. When the number of available metrics is known, allocate a buffer large enough to hold that many fpga_metric_info data structures, and call fpgaGetMetricsInfo() to populate the entries:
fpga_metric_info *metric_info;\nuint64_t metric_infos = num_metrics;\nmetric_info = calloc(num_metrics, sizeof(fpga_metric_info));\nfpgaGetMetricsInfo(handle, metric_info, &metric_infos);\n
Figure 17 Querying Metrics Info
The fpga_metric structure is the representation of a metric\u2019s value:
typedef struct fpga_metric {\nuint64_t metric_num;\nmetric_value value;\nbool isvalid;\n} fpga_metric;\n
Relevant Links: - metric_value Figure 18 struct fpga_metric
The metric_num field matches the metric_num field of the fpga_metric_info structure. value contains the metric value, which is encoded in the C data type identified by the metric_datatype field of fpga_metric_info. Finally, the isvalid field denotes whether the metric value is valid.
There are two methods of obtaining a metric\u2019s value, given the information in the fpga_metric_info structure:
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2181-querying-metric-values-by-index","title":"2.1.8.1 Querying Metric Values by Index","text":"fpgaGetMetricsByIndex() retrieves a metric value using the metric_num field of the metric info:
uint64_t metric_num = metric_info[0]->metric_num;\nfpga_metric metric0;\nfpgaGetMetricsByIndex(handle, &metric_num, 1, &metric0);\n
Figure 19 Retrieve Metric by Index
This call allows retrieving one or more metric values, each identified by their unique metric_num. The second and fourth parameters allow passing arrays so that multiple values can be fetched in a single call.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2182-querying-metric-values-by-name","title":"2.1.8.2 Querying Metric Values by Name","text":"fpgaGetMetricsByName() retrieves a metric value using the metric_name field of the metric info:
char *metric_name = metric_info[1]->metric_name;\nfpga_metric metric1;\nfpgaGetMetricsByName(handle, &metric_name, 1, &metric1);\n
This call also allows retrieving one or more metric values, each identified by their unique metric_name. The second and fourth parameters allow passing arrays so that multiple values can be fetched in a single call.
The fpgaGetMetricsThresholdInfo() call is provided for legacy implementations only. It should be considered deprecated for current and future FPGA designs.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#219-sysobject","title":"2.1.9 SysObject","text":"When the hardware access method in use is the DFL drivers (see 2.3.2 libxfpga Plugin), the sysfs tree rooted at the struct _fpga_token\u2019s sysfspath member is accessible via the OPAE SDK SysObject API. The SysObject API provides an abstraction to search, traverse, read, and write sysfs entities. These sysfs entities may take the form of directories, which are referred to as containers, or files, which are referred to as attributes. Figure 20 enum fpga_sysobject_type shows the API\u2019s means of distinguishing between the two types.
enum fpga_sysobject_type {\nFPGA_OBJECT_CONTAINER = (1u << 0),\nFPGA_OBJECT_ATTRIBUTE = (1u << 1)\n};\n
Figure 20 enum fpga_sysobject_type
The SysObject API introduces another opaque structure type, fpga_object. An fpga_object can be queried from an fpga_token or an fpga_handle by way of the fpgaTokenGetObject() and fpgaHandleGetObject() API\u2019s.
fpga_result fpgaTokenGetObject(fpga_token token, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaHandleGetObject(fpga_handle handle, const char *name,\nfpga_object *object, int flags);\n
Figure 21 fpgaTokenGetObject() / fpgaHandleGetObject()
The remainder of the SysObject API is broken into two categories of calls, depending on the fpga_object\u2019s type. The type of an fpga_object is learned via fpgaObjectGetType().
fpga_result fpgaObjectGetType(fpga_object obj,\nenum fpga_sysobject_type *type);\n
Figure 22 fpgaObjectGetType()
When an fpga_object is no longer needed, it should be freed via fpgaDestroyObject().
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2191-fpga_object_container-apis","title":"2.1.9.1 FPGA_OBJECT_CONTAINER API\u2019s","text":"For directory sysfs entities, passing a value of FPGA_OBJECT_RECURSE_ONE or FPGA_OBJECT_RECURSE_ALL in the flags parameter to fpgaTokenGetObject() or fpgaHandleGetObject() causes these two API\u2019s to treat the target object as either a single-layer or multi-layer directory structure, making its child entities available for query via fpgaObjectGetObject() and fpgaObjectGetObjectAt().
fpga_result fpgaObjectGetObject(fpga_object parent, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaObjectGetObjectAt(fpga_object parent, size_t idx,\nfpga_object *object);\n
Figure 23 fpgaObjectGetObject() / fpgaObjectGetObjectAt()
Any child object resulting from fpgaObjectGetObject() or fpgaObjectGetObjectAt() must be freed via fpgaDestroyObject() when it is no longer needed.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2192-fpga_object_attribute-apis","title":"2.1.9.2 FPGA_OBJECT_ATTRIBUTE API\u2019s","text":"Attribute sysfs entities may be queried for their size and read from or written to. In order to determine the size of an attribute\u2019s data, use fpgaObjectGetSize().
fpga_result fpgaObjectGetSize(fpga_object obj,\nuint32_t *value, int flags);\n
Figure 24 fpgaObjectGetSize()
Attributes containing arbitrary string data can be read with fpgaObjectRead().
fpga_result fpgaObjectRead(fpga_object obj, uint8_t *buffer,\nsize_t offset, size_t len, int flags);\n
Figure 25 fpgaObjectRead()
If an attribute contains an unsigned integer value, its value can be read with fpgaObjectRead64() and written with fpgaObjectWrite64().
fpga_result fpgaObjectRead64(fpga_object obj,\nuint64_t *value, int flags);\nfpga_result fpgaObjectWrite64(fpga_object obj,\nuint64_t value, int flags);\n
Figure 26 fpgaObjectRead64() / fpgaObjectWrite64()
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2110-utilities","title":"2.1.10 Utilities","text":"The fpga_result enumeration defines a set of error codes used throughout OPAE. In order to convert an fpga_result error code into a printable string, the application can use the fpgaErrStr() call.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#22-dfl-driver-ioctl-interfaces","title":"2.2 DFL Driver IOCTL Interfaces","text":"The DFL drivers export an IOCTL interface which the libxfpga.so plugin consumes in order to query and configure aspects of the FME and Port. These interfaces are used only internally by the SDK; they are not customer-facing. The description here is provided for completeness only.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#221-port-reset","title":"2.2.1 Port Reset","text":"The DFL_FPGA_PORT_RESET ioctl is used by the fpgaReset() call in order to perform a Port reset. The fpga_handle passed to fpgaReset() must be a valid open handle to an FPGA_ACCELERATOR. The ioctl requires no input/output parameters.
Note: fpgaReset() will always reset the entire FIM and AFU region of a device, and will not accept VFIO devices as an input argument. If you wish to issue a reset on the currently loaded AFU only, use the fpgaEnmuerate() function instead.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#222-port-information","title":"2.2.2 Port Information","text":"The DFL_FPGA_PORT_GET_INFO ioctl is used to query properties of the Port, notably the number of associated MMIO regions. The ioctl requires a pointer to a struct dfl_fpga_port_info.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#223-mmio-region-information","title":"2.2.3 MMIO Region Information","text":"The DFL_FPGA_PORT_GET_REGION_INFO ioctl is used to query the details of an MMIO region. The ioctl requires a pointer to a struct dfl_fpga_port_region_info. The index field of the struct is populated by the caller, and the padding, size, and offset values are populated by the DFL driver.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#224-shared-memory-mapping-and-unmapping","title":"2.2.4 Shared Memory Mapping and Unmapping","text":"The DFL_FPGA_PORT_DMA_MAP ioctl is used to map a memory buffer into the application\u2019s process address space. The ioctl requires a pointer to a struct dfl_fpga_port_dma_map.
The DFL_FPGA_PORT_DMA_UNMAP ioctl is used to unmap a memory buffer from the application\u2019s process address space. The ioctl requires a pointer to a struct dfl_fpga_port_dma_unmap.
These ioctls provide the underpinnings of the fpgaPrepareBuffer() and fpgaReleaseBuffer() calls.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#225-number-of-port-error-irqs","title":"2.2.5 Number of Port Error IRQs","text":"The DFL_FPGA_PORT_ERR_GET_IRQ_NUM ioctl is used to query the number of Port error interrupt vectors available. The ioctl requires a pointer to a uint32_t that receives the Port error interrupt count.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#226-port-error-interrupt-configuration","title":"2.2.6 Port Error Interrupt Configuration","text":"The DFL_FPGA_PORT_ERR_SET_IRQ ioctl is used to configure one or more file descriptors for the Port Error interrupt. The ioctl requires a pointer to a struct dfl_fpga_irq_set. The values stored in the evtfds field of this struct should be populated with the event file descriptors for the interrupt, as returned by the eventfd() C standard library API.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#227-number-of-afu-interrupts","title":"2.2.7 Number of AFU Interrupts","text":"The DFL_FPGA_PORT_UINT_GET_IRQ_NUM ioctl is used to query the number of AFU interrupt vectors available. The ioctl requires a pointer to a uint32_t that receives the AFU interrupt count.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#228-user-afu-interrupt-configuration","title":"2.2.8 User AFU Interrupt Configuration","text":"The DFL_FPGA_PORT_UINT_SET_IRQ ioctl is used to configure one or more file descriptors for the AFU interrupt. The ioctl requires a pointer to a struct dfl_fpga_irq_set. The values stored in the evtfds field of this struct should be populated with the event file descriptors for the interrupt, as returned by the eventfd() C standard library API.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#229-partial-reconfiguration","title":"2.2.9 Partial Reconfiguration","text":"The DFL_FPGA_FME_PORT_PR ioctl is used to update the logic stored in the Port\u2019s programmable region. This ioctl must be issued on the device file descriptor corresponding to the FPGA_DEVICE (/dev/dfl-fme.X). The ioctl requires a pointer to a struct dfl_fpga_fme_port_pr with each of the fields populated.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2210-number-of-fme-error-irqs","title":"2.2.10 Number of FME Error IRQs","text":"The DFL_FPGA_FME_ERR_GET_IRQ_NUM ioctl is used to query the number of FME error interrupt vectors available. The ioctl requires a pointer to a uint32_t that receives the FME error interrupt count.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2211-fme-error-interrupt-configuration","title":"2.2.11 FME Error Interrupt Configuration","text":"The DFL_FPGA_FME_ERR_SET_IRQ ioctl is used to configure one or more file descriptors for the FME Error interrupt. The ioctl requires a pointer to a struct dfl_fpga_irq_set. The values stored in the evtfds field of this struct should be populated with the event file descriptors for the interrupt, as returned by the eventfd() C standard library API. as returned by the eventfd() C standard library API.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#23-plugin-manager","title":"2.3 Plugin Manager","text":"The OPAE Plugin Manager refers to initialization code in libopae-c that examines an FPGA device\u2019s PCIe Vendor and Device ID and makes an association between a particular FPGA device and its access method. OPAE currently supports three device access methods:
Access Method
Plugin Module
Device Feature List drivers libxfpga.so Virtual Function I/O libopae-v.so AFU Simulation Environment libase.so Table 9 Plugin Device Access Methods
The Plugin Manager allows code that is written to a specific API signature to access FPGA hardware via different mechanisms. In other words, the end user codes to the OPAE API; and the OPAE API, based on configuration data, routes the hardware access to the device via different means.
As an example, consider an API configuration that accesses FPGA device_A via the Device Feature List drivers and that accesses FPGA device_B via VFIO. The application is coded against the OPAE API.
As part of its initialization process, the application enumerates and discovers an fpga_token corresponding to device_A. That fpga_token is opened and its MMIO region 0 is mapped via a call to fpgaMapMMIO().
The API configuration for device_A is such that the fpga_handle corresponding to device_A routes its hardware access calls through libxfpga.so. The call to fpgaMapMMIO() is redirected to libxfpga.so\u2019s implementation of the MMIO mapping function, xfpga_fpgaMapMMIO(). As a result, the call to xfpga_fpgaMapMMIO() uses its AFU file descriptor to communicate with the DFL driver to map the MMIO region.
Subsequently, the application enumerates and discovers an fpga_token corresponding to device_B. That fpga_token is opened and its MMIO region 0 is mapped via a call to fpgaMapMMIO().
The API configuration for device_B is such that the fpga_handle corresponding to device_B routes its hardware access calls through libopae-v.so. The call to fpgaMapMMIO() is redirected to libopae-v.so\u2019s implementation of the MMIO mapping function, vfio_fpgaMapMMIO(). As a result, the call to vfio_fpgaMapMMIO() uses the MMIO mapping performed by libopaevfio.so during initialization of the VFIO session.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#231-plugin-model","title":"2.3.1 Plugin Model","text":"The OPAE SDK plugin model is facilitated by its use of opaque C structures for fpga_token and fpga_handle. These types are both declared as void *; and this allows the parameters to the OPAE SDK functions to take different forms, depending on the layer of the SDK being used.
At the topmost layer, for example when calling fpgaEnumerate(), the output fpga_token parameter array is actually an array of pointers to opae_wrapped_token struct\u2019s.
typedef struct _opae_wrapped_token {\nuint32_t magic;\nfpga_token opae_token;\nuint32_t ref_count;\nstruct _opae_wrapped_token *prev;\nstruct _opae_wrapped_token *next;\nopae_api_adapter_table *adapter_table;\n} opae_wrapped_token;\n
Figure 27 opae_wrapped_token
An opae_wrapped_token, as the name suggests, is a thin wrapper around the lower-layer token which is stored in struct member opae_token. The adapter_table struct member is a pointer to a plugin-specific adapter table. The adapter table provides a mapping between the top-layer opae_wrapped_token and its underlying plugin-specific API entry points, which are called using the opae_token struct member (the lower-level token).
typedef struct _opae_api_adapter_table {\nstruct _opae_api_adapter_table *next;\nopae_plugin plugin;\n...\nfpga_result (*fpgaEnumerate)(const fpga_properties *filters,\nuint32_t num_filters,\nfpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\nint (*initialize)(void);\nint (*finalize)(void);\n} opae_api_adapter_table;\n
Figure 28 opae_api_adapter_table
When libopae-c loads, the plugin manager uses the plugin configuration data to open and configure a session to each of the required plugin libraries. During this configuration process, each plugin is passed an empty adapter table struct. The purpose of the plugin configuration is to populate this adapter table struct with each of the plugin-specific API entry points.
When the top-level fpgaEnumerate() is called, each adapter table\u2019s plugin-specific fpgaEnumerate() struct member is called; and the output fpga_token\u2019s are collected. At this point, these fpga_token\u2019s are the lower-level token structure types. Before the top-level fpgaEnumerate() returns, these plugin-specific tokens are wrapped inside opae_wrapped_token structures, along with a pointer to the respective adapter table.
After enumeration is complete, the application goes on to call other top-level OPAE SDK functions with the wrapped tokens. Each top-level entry point which accepts an fpga_token knows that it is actually being passed an opae_wrapped_token. With this knowledge, the entry point peeks inside the wrapped token and calls through to the plugin-specific API entry point using the adapter table, passing the lower-level opae_token struct member.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#232-libxfpga-plugin","title":"2.3.2 libxfpga Plugin","text":"2.3.1 Plugin Model introduced the concept of an opae_wrapped_token and a corresponding plugin-specific token structure. libxfpga.so is the plugin library that implements the DFL driver hardware access method. Its plugin-specific token data structure is struct _fpga_token.
struct _fpga_token {\nfpga_token_header hdr;\nuint32_t device_instance;\nuint32_t subdev_instance;\nchar sysfspath[SYSFS_PATH_MAX];\nchar devpath[DEV_PATH_MAX];\nstruct error_list *errors;\n};\n
Relevant Links: - fpga_token_header - error_list Figure 29 struct _fpga_token
A struct _fpga_token corresponding to the Port will have sysfspath and devpath members that contain strings like the following example paths:
sysfspath: \u201c/sys/class/fpga_region/region0/dfl-port.0\u201d\ndevpath: \u201c/dev/dfl-port.0\u201d\n
Figure 30 libxfpga Port Token
Likewise, a struct _fpga_token corresponding to the FME will have sysfspath and devpath members that contain strings like the following example paths:
sysfspath: \u201c/sys/class/fpga_region/region0/dfl-fme.0\u201d\ndevpath: \u201c/dev/dfl-fme.0\u201d\n
Figure 31 libxfpga FME Token
When a call to the top-level fpgaOpen() is made, the lower-level token is unwrapped and passed to xfpga_fpgaOpen(). In return, xfpga_fpgaOpen() opens the character device file identified by the devpath member of the struct _fpga_token. It then allocates and initializes an instance of libxfpga.so\u2019s plugin-specific handle data structure, struct _fpga_handle.
struct _fpga_handle {\npthread_mutex_t lock;\nuint64_t magic;\nfpga_token token;\nint fddev;\nint fdfpgad;\nuint32_t num_irqs;\nuint32_t irq_set;\nstruct wsid_tracker *wsid_root;\nstruct wsid_tracker *mmio_root;\nvoid *umsg_virt;\nuint64_t umsg_size;\nuint64_t *umsg_iova;\nbool metric_enum_status;\nfpga_metric_vector fpga_enum_metric_vector;\nvoid *bmc_handle;\nstruct _fpga_bmc_metric *_bmc_metric_cache_value;\nuint64_t num_bmc_metric;\nuint32_t flags;\n};\n
Relevant Links: - wsid_tracker - fpga_metric_vector - _fpga_bmc_metric Figure 32 struct _fpga_handle
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#233-libopae-v-plugin","title":"2.3.3 libopae-v Plugin","text":"libopae-v.so is the plugin library that implements the VFIO hardware access method. Its plugin-specific token data structure is vfio_token.
#define USER_MMIO_MAX 8\ntypedef struct _vfio_token {\nfpga_token_header hdr;\nfpga_guid compat_id;\npci_device_t *device;\nuint32_t region;\nuint32_t offset;\nuint32_t mmio_size;\nuint32_t pr_control;\nuint32_t user_mmio_count;\nuint32_t user_mmio[USER_MMIO_MAX];\nuint64_t bitstream_id;\nuint64_t bitstream_mdata;\nuint8_t num_ports;\nstruct _vfio_token *parent;\nstruct _vfio_token *next;\nvfio_ops ops;\n} vfio_token;\n
Relevant Links: - fpga_token_header - pci_device_t - vfio_ops Figure 33 vfio_token
When a call to the top-level fpgaOpen() is made, the lower-level token is unwrapped and passed to vfio_fpgaOpen(). In return, vfio_fpgaOpen() opens the VFIO device matching the device address found in the input vfio_token. It then allocates and initializes an instance of libopae-v.so\u2019s plugin-specific handle data structure, vfio_handle.
typedef struct _vfio_handle {\nuint32_t magic;\nstruct _vfio_token *token;\nvfio_pair_t *vfio_pair;\nvolatile uint8_t *mmio_base;\nsize_t mmio_size;\npthread_mutex_t lock;\n#define OPAE_FLAG_HAS_AVX512 (1u << 0)\nuint32_t flags;\n} vfio_handle;\n
Relevant Links: - vfio_token - vfio_pair_t Figure 34 vfio_handle
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2331-supporting-libraries","title":"2.3.3.1 Supporting Libraries","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#23311-libopaevfio","title":"2.3.3.1.1 libopaevfio","text":"libopaevfio.so is OPAE\u2019s implementation of the Linux kernel\u2019s Virtual Function I/O interface. This VFIO interface presents a generic means of configuring and accessing PCIe endpoints from a user-space process via a supporting Linux kernel device driver, vfio-pci.
libopaevfio.so provides APIs for opening/closing a VFIO device instance, for mapping/unmapping MMIO spaces, for allocating/freeing DMA buffers, and for configuring interrupts for the device.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#23312-libopaemem","title":"2.3.3.1.2 libopaemem","text":"Each DMA buffer allocation made by libopaevfio.so\u2019s opae_vfio_buffer_allocate() and opae_vfio_buffer_allocate_ex() APIs requires a backing I/O Virtual Address range. These address ranges are discovered at VFIO device open time by way of the VFIO_IOMMU_GET_INFO ioctl.
Each range specifies a large contiguous block of I/O Virtual Address space. The typical DMA buffer allocation size is significantly less than one of these IOVA blocks, so the division of each block into allocatable segments must be managed so that multiple DMA buffer allocations can be made from a single block. In other words, the IOVA blocks must be memory-managed in order to make efficient use of them.
libopaemem.so provides a generic means of managing a large memory space, consisting of individual large memory blocks of contiguous address space. When a DMA buffer allocation is requested, libopaevfio.so uses this generic memory manager to carve out a small chunk of contiguous IOVA address space in order for the DMA mapping to be made. The IOVA space corresponding to the allocation is marked as allocated, and the rest of the large block remains as allocatable space within the memory manager. Subsequent de-allocation returns a chunk of IOVA space to the free state, coalescing contiguous chunks as they are freed. The allocations and de-allocations of the IOVA space can occur in any order with respect to each other. libopaemem.so tracks both the allocated and free block space, carving out small chunks from the large IOVA blocks on allocations, and coalescing small chunks back into larger ones on frees.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2332-configuring-pcie-virtual-functions","title":"2.3.3.2 Configuring PCIe Virtual Functions","text":"Before an AFU can be accessed with VFIO, the FPGA Physical Function must be configured to enable its Virtual Functions. Then, each VF must be bound to the vfio-pci Linux kernel driver.
As of the Arrow Creek program, the FPGA hardware allows multiple AFU\u2019s to co-exist by placing each AFU in its own PCIe Virtual Function. Upon system startup, no PCIe VF\u2019s exist. The pci_device command can be used to enable the VF\u2019s and their AFU\u2019s. First, use the lspci command to examine the current device topology:
# lspci | grep cel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.3 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n
Figure 35 lspci Device Topology
In this example, VF\u2019s are controlled by PF 0, as highlighted in Figure 35 lspci Device Topology. In the figure, each PF is shown as having the Arrow Creek PF PCIe device ID of bcce.
Now, use the pci_device command to enable three VF\u2019s for PF0:
# pci_device 0000:b1:00.0 vf 3\n# lspci | grep cel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\nb1:00.5 Processing accelerators: Intel Corporation Device bccf\nb1:00.6 Processing accelerators: Intel Corporation Device bccf\nb1:00.7 Processing accelerators: Intel Corporation Device bccf\n
Figure 36 Enable Virtual Functions
Figure 20 Enable Virtual Functions shows that three VF\u2019s were created. Each VF is shown as having the Arrow Creek VF PCIe device ID of bccf.
Now, each Virtual Function must be bound to the vfio-pci Linux kernel driver so that it can be accessed via VFIO:
# opaevfio -i -u myuser -g mygroup 0000:b1:00.5\nBinding (0x8086,0xbccf) at 0000:b1:00.5 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.5 is 318\n
Figure 37 Bind VF's to vfio-pci
Here, myuser and mygroup identify the unprivileged user/group that requires access to the device. The opaevfio command will change the ownership of the device per the values given.
Once the VF\u2019s are bound to vfio-pci, the OPAE SDK will find and enumerate them with libopae-v.so:
# fpgainfo port\n//****** PORT ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:B1:00.0\nDevice Id : 0xBCCE\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nDevice Id : 0xBCCF\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nDevice Id : 0xBCCF\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nDevice Id : 0xBCCF\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n
Figure 38 List VF's with fpgainfo
When the VF\u2019s are no longer needed, they can be unbound from the vfio-pci driver:
# opaevfio -r 0000:b1:00.5\nReleasing (0x8086,0xbccf) at 0000:b1:00.5 from vfio-pci\n# opaevfio -r 0000:b1:00.6\nReleasing (0x8086,0xbccf) at 0000:b1:00.6 from vfio-pci\n# opaevfio -r 0000:b1:00.7\nReleasing (0x8086,0xbccf) at 0000:b1:00.7 from vfio-pci\n
Figure 39 Unbind VF's from vfio-pci
Finally, the VF\u2019s can be disabled:
# pci_device 0000:b1:00.0 vf 0\n# lspci | grep cel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n
Figure 40 Disable Virtual Functions
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#24-application-flow","title":"2.4 Application Flow","text":"A typical OPAE application that interacts with an AFU via MMIO and shared memory will have a flow similar to the one described in this section.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#241-create-filter-criteria","title":"2.4.1 Create Filter Criteria","text":"Refer to 2.1.2 Enumeration. When enumerating AFU\u2019s, if no filtering criteria is specified, then fpgaEnumerate() returns fpga_token\u2019s for each AFU that is present in the system. In order to limit the enumeration search to a specific AFU, create an fpga_properties object and set its guid to that of the desired AFU:
#define MY_AFU_GUID \u201c57fa0b03-ab4f-4b02-b4eb-d3fe1ec18518\u201d\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpgaGetProperties(NULL, &filter);\nuuid_parse(MY_AFU_GUID, guid);\nfpgaPropertiesSetGUID(filter, guid);\n
Figure 41 Flow: Create Filter Criteria
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#242-enumerate-the-afu","title":"2.4.2 Enumerate the AFU","text":"With the filtering criteria in place, enumerate to obtain an fpga_token for the AFU:
fpga_token afu_token = NULL;\nuint32_t num_matches = 0;\nfpgaEnumerate(&filter, 1, &afu_token, 1, &num_matches);\n
Figure 42 Flow: Enumerate the AFU
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#243-open-the-afu","title":"2.4.3 Open the AFU","text":"After finding an fpga_token for the AFU using fpgaEnumerate(), the token must be opened with fpgaOpen() to establish a session with the AFU. The process of opening an fpga_token creates an fpga_handle:
fpga_handle afu_handle = NULL;\nfpgaOpen(afu_token, &afu_handle, 0);\n
Figure 43 Flow: Open the AFU
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#244-map-mmio-region","title":"2.4.4 Map MMIO Region","text":"In order to access the MMIO region of the AFU to program its CSR\u2019s, the region must first be mapped into the application\u2019s process address space. This is accomplished using fpgaMapMMIO():
uint32_t region = 0;\nfpgaMapMMIO(afu_handle, region, NULL);\n
Figure 44 Flow: Map MMIO Region
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#245-allocate-dma-buffers","title":"2.4.5 Allocate DMA Buffers","text":"If the AFU is DMA-capable, shared memory buffers can be allocated and mapped into the process address space and the IOMMU with fpgaPrepareBuffer(). Refer to Figure 8 Configuring Huge Pages for instructions on configuring 2MiB and 1GiB huge pages.
#define BUF_SIZE (2 * 1024 * 1024)\nvolatile uint64_t *src_ptr = NULL;\nuint64_t src_wsid = 0;\nvolatile uint64_t *dest_ptr = NULL;\nuint64_t dest_wsid = 0;\nfpgaPrepareBuffer(afu_handle, BUF_SIZE, (void **)&src_ptr,\n&src_wsid, 0);\nfpgaPrepareBuffer(afu_handle, BUF_SIZE, (void **)&dest_ptr,\n&dest_wsid, 0);\nmemset(src_ptr, 0xaf, BUF_SIZE);\nmemset(dest_ptr, 0xbe, BUF_SIZE);\n
Figure 45 Flow: Allocate DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#246-make-afu-aware-of-dma-buffers","title":"2.4.6 Make AFU Aware of DMA Buffers","text":"The process by which locations of shared memory buffers and their sizes are made known to the AFU is entirely AFU-specific. This example shows the method used by the Native Loopback AFU. Each buffer I/O virtual address is cacheline-aligned and programmed into a unique AFU CSR; then the buffer size in lines is programmed into a length CSR:
#define CSR_SRC_ADDR 0x000A // AFU-specific\n#define CSR_DEST_ADDR 0x000B // AFU-specific\n#define CSR_NUM_LINES 0x000C // AFU-specific\nuint64_t src_iova = 0;\nuint64_t dest_iova = 0;\nfpgaGetIOAddress(afu_handle, src_wsid, &src_iova);\nfpgaGetIOAddress(afu_handle, dest_wsid, &dest_iova);\nfpgaWriteMMIO64(afu_handle, 0, CSR_SRC_ADDR, src_iova >> 6);\nfpgaWriteMMIO64(afu_handle, 0, CSR_DEST_ADDR, dest_iova >> 6);\nfpgaWriteMMIO32(afu_handle, 0, CSR_NUM_LINES, BUF_SIZE / 64);\n
Figure 46 Flow: Make AFU Aware of DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#247-initiate-an-acceleration-task","title":"2.4.7 Initiate an Acceleration Task","text":"With the shared buffer configuration complete, the AFU can be told to initiate the acceleration task. This process is AFU-specific. The Native Loopback AFU starts the acceleration task by writing a value to its control CSR:
#define CSR_CTRL 0x000D // AFU-specific\nfpgaWriteMMIO32(afu_handle, 0, CSR_CTRL, 3);\n
Figure 47 Initiate an Acceleration Task
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#248-wait-for-task-completion","title":"2.4.8 Wait for Task Completion","text":"Once the acceleration task is initiated, the application may poll the AFU for a completion status. This process is AFU-specific. The AFU may provide a status CSR for the application to poll; or the AFU may communicate status to the application by means of a result code written to a shared buffer.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#249-free-dma-buffers","title":"2.4.9 Free DMA Buffers","text":"When the acceleration task completes and the AFU is quiesced such that there are no outstanding memory transactions targeted for the shared memory, the DMA buffers can be unmapped and freed using fpgaReleaseBuffer():
fpgaReleaseBuffer(afu_handle, src_wsid);\nfpgaReleaseBuffer(afu_handle, dest_wsid);\n
Figure 48 Flow: Free DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2410-unmap-mmio-region","title":"2.4.10 Unmap MMIO Region","text":"The MMIO regions should also be unmapped using fpgaUnmapMMIO():
fpgaUnmapMMIO(afu_handle, region);\n
Figure 49 Flow: Unmap MMIO Region
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2411-close-the-afu","title":"2.4.11 Close the AFU","text":"The AFU handle should be closed via fpgaClose() to release its resources:
fpgaClose(afu_handle);\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#2412-release-the-tokens-and-properties","title":"2.4.12 Release the Tokens and Properties","text":"The fpga_token\u2019s returned by fpgaEnumerate() should be destroyed using the fpgaDestroyToken() API. The fpga_properties objects should be destroyed using the fpgaDestroyProperties() API:
fpgaDestroyToken(&afu_token);\nfpgaDestroyProperties(&filter);\n
Figure 51 Flow: Release the Tokens and Properties
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#30-opae-c-api","title":"3.0 OPAE C++ API","text":"The OPAE C++ API refers to a C++ layer that sits on top of the OPAE C API, providing object-oriented implementations of the main OPAE C API abstractions: properties, tokens, handles, dma buffers, etc. Like the OPAE C API, the C++ API headers contain Doxygen markup for each of the provided classes.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#31-libopae-cxx-core","title":"3.1 libopae-cxx-core","text":"The implementation files for the C++ API are compiled into libopae-cxx-core.so. A convenience header, core.h, provides a quick means of including each of the C++ API headers. Each of the types comprising the C++ API is located within the opae::fpga::types C++ namespace.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#311-properties","title":"3.1.1 Properties","text":"Class properties provides the C++ implementation of the fpga_properties type and its associated APIs.
properties::ptr_t filter = properties::get();\n
Figure 52 C++ Create New Empty Properties
Class properties provides member variables for each fpga_properties item that can be manipulated with fpgaPropertiesGet\u2026() and fpgaPropertiesSet\u2026(). For example, to set the AFU ID in a properties instance and to set that instance\u2019s type to FPGA_ACCELERATOR:
#define MY_AFU_ID \u201c8ad74241-d13b-48eb-b428-7986dcbcab14\u201d\nfilter->guid.parse(MY_AFU_ID);\nfilter->type = FPGA_ACCELERATOR;\n
Figure 53 C++ Properties Set GUID and Type
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#312-tokens","title":"3.1.2 Tokens","text":"Class token provides the C++ implementation of the fpga_token type and its associated APIs. Class token also provides the enumerate() static member function:
std::vector<token::ptr_t> tokens = token::enumerate({filter});\nif (tokens.size() < 1) {\n// flag error and return\n}\ntoken::ptr_t tok = tokens[0];\n
Figure 54 C++ Enumeration
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#313-handles","title":"3.1.3 Handles","text":"Class handle provides the C++ implementation of the fpga_handle type and its associated APIs. The handle class provides member functions for opening and closing a token, for reading and writing to MMIO space, and for reconfiguring the FPGA\u2019s Programmable Region.
handle::ptr_t accel = handle::open(tok, 0);\n
Figure 55 C++ Opening a Handle
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#314-shared-memory","title":"3.1.4 Shared Memory","text":"The shared_buffer class provides member functions for allocating and releasing DMA buffers, for querying buffer attributes, and for reading and writing buffers.
#define BUF_SIZE (2 * 1024 * 1024)\nshared_buffer::ptr_t input = shared_buffer::allocate(accel, BUF_SIZE);\nshared_buffer::ptr_t output = shared_buffer::allocate(accel, BUF_SIZE);\nstd::fill_n(input->c_type(), BUF_SIZE, 0xaf);\nstd::fill_n(output->c_type(), BUF_SIZE, 0xbe);\n
Figure 56 C++ Allocate and Init Buffers
Once DMA buffers have been allocated, their IO addresses are programmed into AFU-specific CSRs to enable the DMA. Here, the IO address of each buffer is aligned to the nearest cache line before programming it into the AFU CSR space. The number of cache lines is then programmed into the appropriate AFU CSR.
#define CSR_SRC_ADDR 0x000A // AFU-specific\n#define CSR_DEST_ADDR 0x000B // AFU-specific\n#define CSR_NUM_LINES 0x000C // AFU-specific\n#define LOG2_CL 6\naccel->write_csr64(CSR_SRC_ADDR, input->io_address() >> LOG2_CL);\naccel->write_csr64(CSR_DEST_ADDR, output->io_address() >> LOG2_CL);\naccel->write_csr32(CSR_NUM_LINES, BUF_SIZE / 64);\n
Figure 57 C++ Make the AFU Aware of DMA Buffers
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#315-events","title":"3.1.5 Events","text":"The event class provides member functions for event registration. In order to register an event, provide the handle::ptr_t for the desired device, along with the event type and optional flags.
int vect = 2;\nevent::ptr_t evt = event::register_event(accel,\nFPGA_EVENT_INTERRUPT,\nvect);\nint evt_fd = evt.os_object();\n
Figure 58 C++ Event Registration
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#316-errors","title":"3.1.6 Errors","text":"Class error provides a means of querying the device errors given a token::ptr_t. The token and integer ID provided to the error::get() static member function uniquely identify one of the 64-bit error masks associated with the token.
error::ptr_t err = error::get(tok, 0);
Figure 59 C++ Query Device Errors
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#317-sysobject","title":"3.1.7 SysObject","text":"Class sysobject is the C++ implementation of the OPAE SysObject API. sysobject provides a means of creating class instances via its two sysobject::get() static member functions. A third non-static sysobject::get() enables creating a sysobject instance given a parent sysobject instance. The read64() and write64() member functions allow reading and writing the sysobject\u2019s value as a 64-bit unsigned integer. The bytes() member functions allow reading a sysobject\u2019s value as a raw byte stream.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#40-opae-python-api","title":"4.0 OPAE Python API","text":"The OPAE Python API is built on top of the OPAE C++ Core API and its object model. Because of this, developing OPAE applications in Python is very similar to developing OPAE applications in C++ which significantly reduces the learning curve required to adapt to the Python API. While the object model remains the same, some static factory functions in the OPAE C++ Core API have been moved to module level methods in the OPAE Python API with the exception of the properties class. The goal of the OPAE Python API is to enable fast prototyping, test automation, infrastructure managment, and an easy to use framework for FPGA resource interactions that don\u2019t rely on software algorithms with a high runtime complexity.
The major benefits of using pybind11 for developing the OPAE Python API include, but are not limited to, the following features of pybind11:
- Uses C++ 11 standard library although it can use C++ 14 or C++17.
- Automatic conversions of shared_ptr types
- Built-in support for numpy and Eigen numerical libraries
- Interoperable with the Python C API
Currently, the only Python package that is part of OPAE is opae.fpga. Because opae.fpga is built on top of the opae-cxx-core API, it does require that the runtime libraries for both opae-cxx-core and opae-c be installed on the system (as well as any other libraries they depend on). Those libraries can be installed using the opae-libs package (from either RPM or DEB format - depending on your Linux distribution). Installation for the Python OPAE bindings are included in the Getting Started Guide for your platform.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#41-opae","title":"4.1 opae","text":"The Python API is coded as a pybind11 project, which allows C++ code to directly interface with Python internals. Each C++ API concept is encoded into a Python equivalent. The functionality exists as a Python extension module, compiled into _opae.so.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#411-enumeration","title":"4.1.1 Enumeration","text":"Enumeration is somewhat simplified as compared to the OPAE C/C++ APIs. The fpga.enumerate() function accepts keyword arguments for each of the property names that are defined in the C++ API. As an example, to enumerate for an FPGA_ACCELERATOR by its GUID:
from opae import fpga\nMY_ACCEL = \u201cd573b29e-176f-4cb7-b810-efbf7be34cc9\u201d\ntokens = fpga.enumerate(type=fpga.ACCELERATOR, guid=MY_ACCEL)\nassert tokens, \u201cNo accelerator matches {}\u201d.format(MY_ACCEL)\n
Figure 60 Python Enumeration
The return value from the fpga.enumerate() function is a list of all the token objects matching the search criteria.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#412-properties","title":"4.1.2 Properties","text":"Querying properties for a token or handle is also a bit different in the Python API. In order to query properties for one of these objects, pass the object to the fpga.properties() constructor. The return value is a properties object with each of the property names defined as instance attributes.
prop = fpga.properties(tokens[0])\nprint(f'0x{prop.vendor_id:04x} 0x{prop.device_id:04x}')\n
Figure 61 Python Get Token Properties
Properties objects may also be created by invoking the fpga.properties() constructor, passing the same keyword arguments as those to fpga.enumerate(). Properties objects created in this way are also useful for enumeration purposes:
props = fpga.properties(type=fpga.ACCELERATOR)\ntokens = fpga.enumerate([props])\n
Figure 62 Python Properties Constructor
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#413-tokens","title":"4.1.3 Tokens","text":"Tokens overload both the __getitem__ and __getattr__ methods to enable the SysObject API. Both of the following are valid forms of accessing the \u2018errors/first_error\u2019 sysfs attribute, given a token object:
tok = tokens[0]\nferr = tok['errors/first_error']\nprint(f'first error: 0x{ferr.read64():0x}')\nprint('0x{:0x}'.format(tok.errors.first_error.read64()))\n
Figure 63 Python Tokens and SysObject API
Tokens also implement a find() method, which accepts a glob expression in order to search sysfs. The following example finds the \u201cid\u201d sysfs entry in the given token\u2019s sysfs tree.
my_id = tok.find(\u2018i?\u2019)\nprint(f'{my_id.read64()}')\n
Figure 64 Python Token Find
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#414-handles","title":"4.1.4 Handles","text":"Tokens are converted to handles by way of the fpga.open() function. The flags (second) parameter to fpga.open() may be zero or fpga.OPEN_SHARED.
with fpga.open(tok, fpga.OPEN_SHARED) as handle:\nuse_the_handle(handle)\n
Figure 65 Python Open Handle
Like token objects, handle objects overload __getitem__ and __getattr__ methods to enable the SysObject API. handle also provides a find() method similar to token\u2019s find().
err = handle['errors/errors']\nprint(f'errors: 0x{err.read64():0x}')\nprint('first error: 0x{:0x}'.format(\nhandle.errors.first_error.read64()))\nmy_id = handle.find('i?')\nprint(f'{my_id.read64()}')\n
Figure 66 Python Handles and SysObject API
Partial reconfiguration is provided by class handle\u2019s reconfigure() method. The first parameter, slot, will be zero in most designs. The second parameter is an opened file descriptor to the file containing the GBS image. The third parameter, flags, defaults to zero.
with open(\u2018my.gbs\u2019, \u2018rb\u2019) as fd:\nhandle.reconfigure(0, fd)\n
Figure 67 Python Partial Reconfiguration
Device reset is accomplished by means of handle\u2019s reset() method, which takes no parameters.
Finally for handles, CSR space reads are accomplished via read_csr32() and read_csr64(). Both methods accept the register offset as the first parameter and an optional csr_space index, which defaults to zero, as the second parameter. CSR space writes are accomplished via write_csr32() and write_csr64(). Both methods accept the register offset as the first parameter, the value to write as the second, and an optional csr_space index, which defaults to zero, as the third.
print(\u20190x{:0x}\u2019.format(handle.read_csr32(0x000a)))\nprint(\u20180x{:0x}\u2019.format(handle.read_csr64(0x000c)))\nhandle.write_csr32(0x000b, 0xdecafbad, 2)\nhandle.write_csr64(0x000e, 0xc0cac01adecafbad, 2)\n
Figure 68 Python Read/Write CSR
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#415-shared-memory","title":"4.1.5 Shared Memory","text":"The fpga.allocate_shared_buffer() function provides access to the OPAE memory allocator. The allocation sizes and required huge page configurations are the same as those noted in 2.1.5 MMIO and Shared Memory.
The fpga.allocate_shared_buffer() function returns an object instance of type shared_buffer. The shared_buffer class implements methods size(), wsid(), and io_address(), which return the buffer size in bytes, the unique workspace ID, and the IO address respectively:
buf = fpga.allocate_shared_buffer(handle, 4096)\nprint(f\u2019size: {buf.size()}\u2019)\nprint(f\u2019wsid: 0x{buf.wsid():0x}\u2019)\nprint(f\u2019io_address: 0x{buf.io_address():0x}\u2019)\n
Figure 69 Python Allocate Shared Memory
The shared_buffer class implements a fill() method which takes an integer parameter which is applied to each byte of the buffer (similar to C standard library\u2019s memset()). The compare() method compares the contents of the first size bytes of one buffer to another. The value returned from compare() is the same as the C standard library\u2019s memcmp(). The copy() method copies the first size bytes of the calling buffer into the argument buffer.
b0 = fpga.allocate_shared_buffer(handle, 4096)\nb1 = fpga.allocate_shared_buffer(handle, 4096)\nb0.fill(0xa5)\nb1.fill(0xa5)\nprint(f'compare: {b0.compare(b1, 4096)}')\nb1.fill(0xa0)\nb0.copy(b1, 4096)\nprint(f'compare: {b0.compare(b1, 4096)}')\n
Figure 70 Python Buffer Fill, Copy, Compare
shared_buffer\u2019s read32() and read64() methods read a 32- or 64-bit value from the given offset. The write32() and write64() methods write a 32- or 64-bit value to the given offset.
print(f'value at 0: 0x{b0.read32(0):0x}')\nprint(f'value at 4: 0x{b0.read64(4):0x}')\nb0.write32(0xabadbeef, 0)\nb0.write64(0xdecafbadabadbeef, 4)\nprint(f'value at 0: 0x{b0.read32(0):0x}')\nprint(f'value at 4: 0x{b0.read64(4):0x}')\n
Figure 71 Python Buffer Read and Write
The shared_buffer class provides three polling methods: poll(), poll32(), and poll64(). Each method takes an offset as its first parameter. The second parameter is a value and the third is a mask. The value and mask parameters are 8 bits wide for poll(), 32 bits wide for poll32(), and 64 bits wide for poll64(). The fourth and last parameter is a timeout value which defaults to 1000 microseconds.
Each polling method reads the n-bit wide item at offset and applies (logical AND) the mask to that value. The masked value created in the previous step is then compared to the second parameter, value. If the two values are equal, then the method returns true immediately. Otherwise, the method continues to loop, attempting the same comparison over and over without sleeping. Finally, if the elapsed time from the beginning of the call to the current time is greater than or equal to the timeout value, then the method times out and returns false.
if b0.poll32(0, 0xbebebebe, 0xffffffff, 250):\nprint(\u2018Got it!\u2019)\n
Figure 72 Python Buffer Poll
The shared_buffer split() method allows creating two or more buffer objects from one larger buffer object. The return value is a list of shared_buffer instances whose sizes match the arguments given to split().
b1, b2 = b1.split(2048, 2048)\nprint(f'b1 io_address: 0x{b1.io_address():0x}')\nprint(f'b2 io_address: 0x{b2.io_address():0x}')\n
Figure 73 Python Splitting Buffer
Finally, the shared_buffer class implements the Python buffer protocol to support memoryview objects. The Python buffer protocol allows access to an object\u2019s underlying memory without copying that memory. As a brief example:
mv = memoryview(b1)\nassert mv\nassert mv[0] == 0xbe\nb1[15] = int(65536)\nassert struct.unpack('<L', bytearray(b1[15:19]))[0] == 65536\n
Figure 74 Python memoryview
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#416-events","title":"4.1.6 Events","text":"Given a handle and an event type, the fpga.register_event() function returns an object of type event. The event class implements one method, os_object(), which returns the underlying file descriptor that can be used to poll for the event:
import select\nevt = fpga.register_event(handle, fpga.EVENT_ERROR)\nos_object = evt.os_object()\nreceived_event = False\nepoll = select.epoll()\nepoll.register(os_object, select.EPOLLIN)\nfor fileno, ev in epoll.poll(1):\nif fileno == os_object:\nreceived_event = True\nprint(f'received: {received_event}')\n
Figure 75 Python Events
In addition to fpga.EVENT_ERROR, fpga.EVENT_INTERRUPT, and fpga.EVENT_POWER_THERMAL are also supported.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#417-errors","title":"4.1.7 Errors","text":"Given a token, the fpga.errors() function returns a list of objects of type error. Each error instance represents a 64-bit mask of error values. The error bit masks are platform-dependent. Each error instance has two attributes: name and can_clear and one method: read_value() which returns the 64-bit error mask.
for e in fpga.errors(tok):\nprint(f\u2019name: \"{e.name}\"\u2019)\nprint(f\u2019can_clear: {e.can_clear}\u2019)\nprint(f\u2019value: {e.read_value()}\u2019)\n
Figure 76 Python Get Errors
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#418-sysobject","title":"4.1.8 SysObject","text":"The Python API\u2019s SysObject implementation is introduced in 4.1.3 Tokens and 4.1.4 Handles. When the index operator (__getitem__) or attribute reference (__getattr__) is used and the referenced string or attribute name corresponds to a sysfs entry in the sysfs path of either a token or a handle, then an object of type sysobject is returned.
The size() method returns the length of the sysfs entry in bytes. Note that a typical sysfs entry is terminated with a \u2018\\n\u2019 followed by the \u2018\\0\u2019 NULL terminator. The bytes() method returns the sysfs entry\u2019s value as a string.
afu_id = tok['afu_id']\nassert afu_id\nprint(f'size: {afu_id.size()} bytes: {afu_id.bytes().rstrip()}')\n
Figure 77 Python sysobject as Bytes
The sysobject read64() and write64() methods provide a means to read and write a sysfs entry\u2019s value as an unsigned 64-bit integer. The sysobject class itself also implements the __getitem__ and __getattr__ methods so that a sysobject of type FPGA_OBJECT_CONTAINER can retrieve sysobject instances for child sysfs entries.
errs = tok['errors']\nfirst = errs['first_error']\nassert first\nprint(f'first 0x{first.read64():0x}')\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#418-python-example-application","title":"4.1.8 Python Example Application","text":"The following example is an implementation of the sample, hello_fpga.c, which is designed to configure the NLB0 diagnostic accelerator for a simple loopback.
import time\nfrom opae import fpga\nNLB0 = \"d8424dc4-a4a3-c413-f89e-433683f9040b\"\nCTL = 0x138\nCFG = 0x140\nNUM_LINES = 0x130\nSRC_ADDR = 0x0120\nDST_ADDR = 0x0128\nDSM_ADDR = 0x0110\nDSM_STATUS = 0x40\ndef cl_align(addr):\nreturn addr >> 6\ntokens = fpga.enumerate(type=fpga.ACCELERATOR, guid=NLB0)\nassert tokens, \"Could not enumerate accelerator: {}\".format(NlB0)\nwith fpga.open(tokens[0], fpga.OPEN_SHARED) as handle:\nsrc = fpga.allocate_shared_buffer(handle, 4096)\ndst = fpga.allocate_shared_buffer(handle, 4096)\ndsm = fpga.allocate_shared_buffer(handle, 4096)\nhandle.write_csr32(CTL, 0)\nhandle.write_csr32(CTL, 1)\nhandle.write_csr64(DSM_ADDR, dsm.io_address())\nhandle.write_csr64(SRC_ADDR, cl_align(src.io_address())) # cacheline-aligned\nhandle.write_csr64(DST_ADDR, cl_align(dst.io_address())) # cacheline-aligned\nhandle.write_csr32(CFG, 0x42000)\nhandle.write_csr32(NUM_LINES, 4096/64)\nhandle.write_csr32(CTL, 3)\nwhile dsm[DSM_STATUS] & 0x1 == 0:\ntime.sleep(0.001)\nhandle.write_csr32(CTL, 7)\n
This example shows how one might reprogram (Partial Reconfiguration) an accelerator on a given bus, 0x5e, using a bitstream file, m0.gbs.
from opae import fpga\nBUS = 0x5e\nGBS = 'm0.gbs'\ntokens = fpga.enumerate(type=fpga.DEVICE, bus=BUS)\nassert tokens, \"Could not enumerate device on bus: {}\".format(BUS)\nwith open(GBS, 'rb') as fd, fpga.open(tokens[0]) as device:\ndevice.reconfigure(0, fd)\n
Figure 78 Python sysobject Container
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#50-management-interfaces-opaeadmin","title":"5.0 Management Interfaces - opae.admin","text":"While the OPAE SDK C, C++, and Python APIs focus on presenting the AFU and all its related functionality to the end user, there is also a need for a maintenance functionality to aid in configuring the platform and performing secure firmware updates for the FPGA device and its components. opae.admin is a Python framework which provides abstractions for performing these types of maintenance tasks on FPGA devices. opae.admin provides Python classes which model the FPGA and the sysfs interfaces provided by the DFL drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#51-sysfs","title":"5.1 sysfs","text":"opae.admin\u2019s sysfs module provides abstractions for interacting with sysfs nodes, which comprise the base entity abstraction of opae.admin.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#511-sysfs_node","title":"5.1.1 sysfs_node","text":"A sysfs_node is an object that tracks a unique path within a sysfs directory tree. sysfs_node provides methods for finding and constructing other sysfs_node objects, based on the root path of the parent sysfs_node object. sysfs_node also provides a mechanism to read and write sysfs file contents. sysfs_node serves as the base class for many of the sysfs module\u2019s other classes.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#512-pci_node","title":"5.1.2 pci_node","text":"A pci_node is a sysfs_node that is rooted at /sys/bus/pci/devices. Each pci_node has a unique PCIe address corresponding to the PCIe device it represents. Methods for finding the pci_node\u2019s children, for determining the PCIe device tree rooted at the node, for manipulating the node\u2019s PCIe address, for determining the vendor and device ID\u2019s, and for removing, unbinding, and rescanning the device are provided.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#513-sysfs_driver","title":"5.1.3 sysfs_driver","text":"A sysfs_driver is a sysfs_node that provides a method for unbinding a sysfs_device object.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#514-sysfs_device","title":"5.1.4 sysfs_device","text":"A sysfs_device is a sysfs_node that is located under /sys/class or /sys/bus. sysfs_device provides the basis for opae.admin\u2019s FPGA enumeration capability.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#515-pcie_device","title":"5.1.5 pcie_device","text":"A pcie_device is a sysfs_device that is rooted at /sys/bus/pci/devices.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#52-fpga","title":"5.2 fpga","text":"opae.admin\u2019s fpga module provides classes which abstract an FPGA and its components.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#521-region","title":"5.2.1 region","text":"A region is a sysfs_node that has an associated Linux character device, rooted at /dev. Methods for opening the region\u2019s character device file and for interacting with the character device via its IOCTL interface are provided.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#522-fme","title":"5.2.2 fme","text":"An fme is a region that represents an FPGA device\u2019s FME component. An fme provides accessors for the PR interface ID, the various bus paths that may exist under an FME, and the BMC firmware revision information.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#523-port","title":"5.2.3 port","text":"A port is a region that represents an FPGA device\u2019s Port component. A port provides an accessor for the Port AFU ID.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#524-fpga_base","title":"5.2.4 fpga_base","text":"An fpga_base is a sysfs_device that provides accessors for the FPGA device\u2019s FME, for the FPGA device\u2019s Port, and for the secure update sysfs controls. fpga_base provides routines for enabling and disabling AER and for performing device RSU.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#525-fpga","title":"5.2.5 fpga","text":"An fpga (derived from fpga_base) is the basis for representing the FPGA device in opae.admin. Utilities such as fpgasupdate rely on fpga\u2019s enum classmethod to enumerate all of the FPGA devices in the system. In order for a device to enumerate via this mechanism, it must be bound to the dfl-pci driver at the time of enumeration.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#53-opaeadmin-utilities","title":"5.3 opae.admin Utilities","text":"Several utilities are written on top of opae.admin\u2019s class abstractions. The following sections highlight some of the most commonly-used utilities.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#531-fpgasupdate","title":"5.3.1 fpgasupdate","text":"fpgasupdate, or FPGA Secure Update, is used to apply firmware updates to the components of the FPGA. As the name implies, these updates target a secure FPGA device, one that has the ability to implement a secure root of trust. The command-line interface to fpgasupdate was designed to be as simple as possible for the end user. The command simply takes a path to the firmware update file to be applied and the PCIe address of the targeted FPGA device.
# fpgasupdate update-file.bin 0000:b2:00.0\n
Figure 79 fpgasupdate Interface
fpgasupdate can apply a variety of firmware image updates. | Image| Description| | -----| -----| |Programmable Region Image| .gbs or Green BitStream| |SR Root Key Hash| Static Region RKH| |PR Root Key Hash| Programmable Region RKH| |FPGA Firmware Image| Static Region Device Firmware| |PR Authentication Certificate| Programmable Region Auth Cert| |BMC Firmware Image| Board Management Controller Firmware| |SR Thermal Image| Static Region Thermal Sensor Thresholds| |PR Thermal Image| Programmable Region Thermal Sensor Thresholds| |CSK Cancelation| Code Signing Key Cancelation Request| |SDM Image| Secure Device Manager Firmware|
Table 10 fpgasupdate Image Types
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#532-pci_device","title":"5.3.2 pci_device","text":"pci_device is a utility that provides a convenient interface to some of the Linux Kernel\u2019s standard PCIe device capabilities.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5321-pci_device-aer-subcommand","title":"5.3.2.1 pci_device aer subcommand","text":"The aer dump subcommand displays the Correctable, Fatal, and NonFatal device errors.
# pci_device 0000:b2:00.0 aer dump\n
Figure 80 pci_device aer dump
The aer mask subcommand displays, masks, or unmasks errors using the syntax of the setpci command.
# pci_device 0000:b2:00.0 aer mask show\n0x00010000 0x000031c1\n# pci_device 0000:b2:00.0 aer mask all\n# pci_device 0000:b2:00.0 aer mask off\n# pci_device 0000:b2:00.0 aer mask 0x01010101 0x10101010\n
Figure 81 pci_device aer mask
The aer clear subcommand clears the current errors.
# pci_device 0000:b2:00.0 aer clear\naer clear errors: 00000000\n
Figure 82 pci_device aer clear
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5322-pci_device-unbind-subcommand","title":"5.3.2.2 pci_device unbind subcommand","text":"The unbind subcommand unbinds the target device from the currently-bound device driver.
# pci_device 0000:b2:00.0 unbind\n
Figure 83 pci_device unbind
In order to re-bind the device to a driver, eg dfl-pci, use the following commands:
# cd /sys/bus/pci/drivers/dfl-pci\n# echo 0000:b2:00.0 > bind\n
Figure 84 Re-binding a Driver
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5323-pci_device-rescan-subcommand","title":"5.3.2.3 pci_device rescan subcommand","text":"The rescan subcommand triggers a PCIe bus rescan of all PCIe devices.
# pci_device 0000:b2:00.0 rescan\n
Figure 85 pci_device rescan
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5324-pci_device-remove-subcommand","title":"5.3.2.4 pci_device remove subcommand","text":"The remove subcommand removes the target device from Linux kernel management.
# pci_device 0000:b2:00.0 remove\n
Figure 86 pci_device remove
Note: a reboot may be required in order to re-establish the Linux kernel management for the device.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5325-pci_device-topology-subcommand","title":"5.3.2.5 pci_device topology subcommand","text":"The topology subcommand shows a tab-delimited depiction of the target device as it exists in the PCIe device tree in the Linux kernel.
# pci_device 0000:b2:00.0 topology\n[pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)] (pcieport)\n[pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:3c:09.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:b2:00.0), pci_id(0x8086, 0x0b30)] (dfl-pci)\n[pci_address(0000:3c:11.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:43:00.0), pci_id(0x8086, 0x0b32)] (no driver)\n[pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:3d:00.1), pci_id(0x8086, 0x0d58)] (i40e)\n[pci_address(0000:3d:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n[pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)] (pcieport)\n[pci_address(0000:41:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n[pci_address(0000:41:00.1), pci_id(0x8086, 0x0d58)] (i40e)\n
Figure 87 pci_device topology
The green output indicates the target device. The other endpoint devices are shown in blue.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5326-pci_device-vf-subcommand","title":"5.3.2.6 pci_device vf subcommand","text":"The vf subcommand allows setting the value of the sriov_numvfs sysfs node of the target device. This is useful in scenarios where device functionality is presented in the form of one or more PCIe Virtual Functions.
# pci_device 0000:b2:00.0 vf 3\n# pci_device 0000:b2:00.0 vf 0\n
Figure 88 pci_device vf
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#533-rsu","title":"5.3.3 rsu","text":"rsu is a utility that performs Remote System Update. rsu is used subsequent to programming a firmware update or other supported file type with fpgasupdate, in order to reset the targeted FPGA entity so that a newly-loaded firmware image becomes active.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5331-rsu-bmc-subcommand","title":"5.3.3.1 rsu bmc subcommand","text":"The bmc subcommand causes a Board Management Controller reset. This command is used to apply a previous fpgasupdate of a BMC firmware image. The --page argument selects the desired boot image. Valid values for --page are \u2018user\u2019 and \u2018factory\u2019.
# rsu bmc --page user 0000:b2:00.0\n# rsu bmc --page factory 0000:b2:00.0\n
Figure 89 rsu bmc
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5332-rsu-retimer-subcommand","title":"5.3.3.2 rsu retimer subcommand","text":"The retimer subcommand causes a Parkvale reset (specific to Vista Creek). This command is used to apply a previous fpgasupdate of a BMC firmware image (the Parkvale firmware is contained within the BMC firmware image). The retimer subcommand causes only the Parkvale to reset.
# rsu retimer 0000:b2:00.0\n
Figure 90 rsu retimer
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5333-rsu-fpga-subcommand","title":"5.3.3.3 rsu fpga subcommand","text":"The fpga subcommand causes a reconfiguration of the FPGA Static Region. This command is used to apply a previous fpgasupdate of the Static Region image. The --page argument selects the desired boot image. Valid values for --page are \u2018user1\u2019, \u2018user2\u2019, and \u2018factory\u2019.
# rsu fpga --page user1 0000:b2:00.0\n# rsu fpga --page user2 0000:b2:00.0\n# rsu fpga --page factory 0000:b2:00.0\n
Figure 91 rsu fpga
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5334-rsu-sdm-subcommand","title":"5.3.3.4 rsu sdm subcommand","text":"The sdm subcommand causes a reset of the Secure Device Manager. This command is used to apply a previous fpgasupdate of the SDM image.
# rsu sdm 0000:b2:00.0\n
Figure 92 rsu sdm
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#5335-rsu-fpgadefault-subcommand","title":"5.3.3.5 rsu fpgadefault subcommand","text":"The fpgadefault subcommand can be used to display the default FPGA boot sequence; and it can be used to select the image to boot on the next reset of the FPGA. When given without additional parameters, the fpgadefault subcommand displays the default FPGA boot sequence:
# rsu fpgadefault 0000:b2:00.0\n
Figure 93 rsu Displaying FPGA Boot Sequence
The parameters to the fpgadefault subcommand are --page and --fallback. The --page parameter accepts \u2018user1\u2019, \u2018user2\u2019, or \u2018factory\u2019, specifying the desired page to boot the FPGA from on the next reset. Note that this subcommand does not actually cause the reset to occur. Please refer to rsu fpga subcommand for an example of resetting the FPGA using the rsu command.
# rsu fpgadefault --page user1 0000:b2:00.0\n# rsu fpgadefault --page user2 0000:b2:00.0\n# rsu fpgadefault --page factory 0000:b2:00.0\n
Figure 94 rsu Select FPGA Boot Image
The --fallback parameter accepts a comma-separated list of the keywords \u2018user1\u2019, \u2018user2\u2019, and \u2018factory\u2019. These keywords, in conjunction with the --page value are used to determine a fallback boot sequence for the FPGA. The fallback boot sequence is used to determine which FPGA image to load in the case of a boot failure. For example, given the following command, the FPGA would attempt to boot in the order \u2018factory\u2019, \u2018user1\u2019, \u2018user2\u2019. That is to say, if the \u2018factory\u2019 image failed to boot, then the \u2018user1\u2019 image would be tried. Failing to boot \u2018user1\u2019, the \u2018user2\u2019 image would be tried.
# rsu fpgadefault --page factory --fallback user1,user2 0000:b2:00.0\n
Figure 95 rsu Select FPGA Boot Image and Fallback
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#60-sample-applications","title":"6.0 Sample Applications","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#61-afu-test-framework","title":"6.1 afu-test Framework","text":"afu-test refers to a test-writing framework that exists as a set of C++ classes written on top of the OPAE C++ bindings. The first class, afu, serves as the base class for the test application abstraction. Class afu provides integration with CLI11, a C++ \u201911 command-line parsing framework, and with spdlog, a C++ logging library. The second class, command represents a unique test sequence that is called by the afu object. Instances of the command class implement the test-specific workload.
class afu {\npublic:\nafu(const char *name,\nconst char *afu_id = nullptr,\nconst char *log_level = nullptr);\nint open_handle(const char *afu_id);\nint main(int argc, char *argv[]);\nvirtual int run(CLI::App *app, command::ptr_t test);\ntemplate<class T>\nCLI::App *register_command();\n};\n
Figure 96 C++ class afu
The afu class constructor initializes the CLI11 command parser with some general, application-wide parameters.
Subcommand Description -g,--guid Accelerator AFU ID. -p,--pci-address Address of the accelerator device. -l,--log-level Requested spdlog output level. -s,--shared Open the AFU in shared mode? -t,--timeout Application timeout in milliseconds. Figure 97 class afu Application Parameters
The register_command() member function adds a test command instance to the afu object. Each test command that an afu object is capable of executing is registered during the test\u2019s startup code. For instance, here is the hssi application\u2019s use of register_command():
hssi_afu app;\nint main(int argc, char *argv[])\n{\napp.register_command<hssi_10g_cmd>();\napp.register_command<hssi_100g_cmd>();\napp.register_command<hssi_pkt_filt_10g_cmd>();\napp.register_command<hssi_pkt_filt_100g_cmd>();\n\u2026\napp.main(argc, argv);\n}\n
Figure 98 hssi's app.register_command()
Next, the afu instance\u2019s main() member function is called. main() initializes the spdlog instance, searches its database of registered commands to find the command matching the test requested from the command prompt, uses the open_handle() member function to enumerate for the requested AFU ID, and calls its run() member function, passing the CLI::App and the test command variables. The run() member function initializes a test timeout mechanism, then calls the command parameter\u2019s run() to invoke the test-specific logic.
With all the boiler-plate of application startup, configuration, and running handled by the afu class, the test-specific command class is left to implement only a minimum number of member functions:
class command {\npublic:\nvirtual const char *name() const = 0;\nvirtual const char *description() const = 0;\nvirtual int run(afu *afu, CLI::App *app) = 0;\nvirtual void add_options(CLI::App *app) { }\nvirtual const char *afu_id() const { return nullptr; }\n};\n
Figure 99 class command
The name() member function gives the unique command name. Some examples of names from the hssi app are hssi_10g, hssi_100g, pkt_filt_10g, and pkt_filt_100g. The description() member function gives a brief description that is included in the command-specific help output. add_options() adds command-specific command-line options. afu_id() gives the AFU ID for the command, in string form. Finally, run() implements the command-specific test functionality.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#62-afu-test-based-samples","title":"6.2 afu-test Based Samples","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#621-dummy_afu","title":"6.2.1 dummy_afu","text":"The dummy_afu application is a afu-test based application that implements three commands: mmio, ddr, and lpbk.
Target Description # dummy_afu mmio Targets special scratchpad area implemented by the AFU. # dummy_afu ddr Execute dummy_afu-specific DDR test. # dummy_afu lpbk Execute a simple loopback test."},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#622-host_exerciser","title":"6.2.2 host_exerciser","text":"host_exerciser markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#623-hssi","title":"6.2.3 hssi","text":"hssi markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#70-other-utilities","title":"7.0 Other Utilities","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#71-opaeio","title":"7.1 opae.io","text":"opae.io markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#72-bitstreaminfo","title":"7.2 bitstreaminfo","text":"The bitstreaminfo command prints diagnostic information about firmware image files that have been passed through the PACSign utility. PACSign prepends secure block 0 and secure block 1 data headers to the images that it processes. These headers contain signature hashes and other metadata that are consumed by the BMC firmware during a secure update.
To run bitstreaminfo, pass the path to the desired firmware image file:
# bitstreaminfo my_file.bin \n
Figure 100 Running bitstreaminfo
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#73-fpgareg","title":"7.3 fpgareg","text":"The fpgareg command prints the register spaces for the following fpga device components:
Command Description # fpgareg 0000:b1:00.0 pcie Walks and prints the DFL for the device. # fpgareg 0000:b1:00.0 bmc Prints the BMC registers for the device. # fpgareg 0000:b1:00.0 hssi Prints the HSSI registers for the device. # fpgareg 0000:b1:00.0 acc Prints the AFU register spaces. Figure 101 fpgareg Commands
Note that fpgareg is only available as of Arrow Creek ADP and forward. It will not work with prior platforms, eg N3000.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#74-opaevfio","title":"7.4 opaevfio","text":"opaevfio markdown document.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#80-opae-build-configurations","title":"8.0 OPAE Build Configurations","text":"Refer to the software installation guides on this website for detailed instructions and building and installing both the OPAE SDK and Linux DFL driver set for any given release. The following section has been included to provide more information on the build targets for the DFL drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#81-cmake-options","title":"8.1 CMake Options","text":"Option Description Values Default -DCMAKE_BUILD_TYPE Configure debugging info Debug
Release
Coverage
RelWithDebInfo
RelWithDebInfo -DCMAKE_INSTALL_PREFIX Root install path /usr/local -DOPAE_BUILD_SPHINX_DOC Enable/Disable docs ON/OFF OFF -DOPAE_BUILD_TESTS Enable/Disable unit tests ON/OFF OFF -DOPAE_ENABLE_MOCK Enable/Disable mock driver for unit tests ON/OFF OFF -DOPAE_INSTALL_RPATH Enable/Disable rpath for install ON/OFF OFF -DOPAE_VERSION_LOCAL Local version string -DOPAE_PRESERVE_REPOS Preserve local changes to external repos? ON/OFF OFF -D OPAE_BUILD_LIBOPAE_CXX Enable C++ bindings ON/OFF ON -DOPAE_WITH_PYBIND11 Enable pybind11 ON/OFF ON -D OPAE_BUILD_PYTHON_DIST Enable Python bindings ON/OFF OFF -DOPAE_BUILD_LIBOPAEVFIO Build libopaevfio.so ON/OFF ON -D OPAE_BUILD_PLUGIN_VFIO Build libopae-v.so ON/OFF ON -DOPAE_BUILD_LIBOPAEUIO Build libopaeuio.so ON/OFF ON -DOPAE_BUILD_LIBOFS Build OFS Copy Engine ON/OFF ON -DOPAE_BUILD_SAMPLES Build Samples ON/OFF ON -DOPAE_BUILD_LEGACY Build legacy repo ON/OFF OFF -DOPAE_LEGACY_TAG Specify legacy build tag master -DOPAE_WITH_CLI11 Enable apps which use CLI11 ON/OFF ON -DOPAE_WITH_SPDLOG Enable apps which use spdlog ON/OFF ON -DOPAE_WITH_LIBEDIT Enable apps which use libedit ON/OFF ON -DOPAE_WITH_HWLOC Enable apps which use hwloc ON/OFF ON -DOPAE_WITH_TBB Enable apps which use Thread Building Blocks ON/OFF ON -DOPAE_MINIMAL_BUILD Enable/Disable minimal build. When set to ON, disable CLI11, spdlog, libedit, hwloc, tbb ON/OFF OFF Table 12 CMake Options
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#82-building-opae-for-debug","title":"8.2 Building OPAE for Debug:","text":"$ cmake .. -DCMAKE_BUILD_TYPE=Debug\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#83-creating-opae-packages","title":"8.3 Creating OPAE Packages","text":"To ease the RPM creation process, the OPAE SDK provides a simple RPM creation script. The parameter to the RPM create script for either Fedora or RHEL is unrestricted For rhel, the build flag -DOPAE_MINIMAL_BUILD is set to ON, omitting the binaries which have dependencies on external components that RHEL does not include in its base repositories.
In order to create RPMs for Fedora, run the create script on a system loaded with all the Fedora build prerequisites. If prerequisites are missing, the create script will complain until they are resolved.
In order to create RPMs for RHEL, run the create script on a system loaded with all the RHEL build prerequisites. If prerequisites are missing, the create script will complain until they are resolved.
$ cd opae-sdk\n$ ./packaging/opae/rpm/create unrestricted\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#84-updating-the-rpm-version-information","title":"8.4 Updating the RPM Version Information","text":"The RPMs will be versioned according to the information found in the file packaging/opae/version. If you edit this file to update the version information, then re-run the create script to create the RPMs.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#90-debugging-opae","title":"9.0 Debugging OPAE","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#91-enabling-debug-logging","title":"9.1 Enabling Debug Logging","text":"The OPAE SDK has a built-in debug logging facility. To enable it, set the cmake flag -DCMAKE_BUILD_TYPE=Debug and then use the following environment variables: | Variable| Description| | ----- | ----- | |LIBOPAE_LOG=1| Enable debug logging output. When not set, only critical error messages are displayed.| |LIBOPAE_LOGFILE=file.log| Capture debug log output to file.log. When not set, the logging appears on stdout and stderr. The file must appear in a relative path or it can be rooted at /tmp.|
Table 13 Logging Environment Variables
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#92-gdb","title":"9.2 GDB","text":"To enable gdb-based debugging, the cmake configuration step must specify a value for -DCMAKE_BUILD_TYPE of either Debug or RelWithDebInfo so that debug symbols are included in the output binaries. The OPAE SDK makes use of dynamically-loaded library modules. When debugging with gdb, the best practice is to remove all OPAE SDK libraries from the system installation paths to ensure that library modules are only loaded from the local build tree:
$ cd opae-sdk/build\n$ LD_LIBRARY_PATH=$PWD/lib gdb --args <some_opae_executable> <args>\n
Figure 103 Debugging with GDB
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#100-adding-new-device-support","title":"10.0 Adding New Device Support","text":"As of OPAE 2.2.0 the SDK has transitioned to a single configuration file model. The libraries, plugins, and applications obtain their runtime configuration during startup by examining a single JSON configuration file. In doing so, the original configuration file formats for libopae-c and fpgad have been deprecated in favor of the respective sections in the new configuration file.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#101-configuration-file-search-order","title":"10.1 Configuration File Search Order","text":"By default the OPAE SDK will install its configuration file to /etc/opae/opae.cfg.
/etc/opae/opae.cfg
Figure 104 Default Configuration File
The SDK searches for the configuration file during startup by employing the following search algorithm:
First, the environment variable LIBOPAE_CFGFILE is examined. If it is set to a path that represents a valid path to a configuration file, then that configuration file path is used, and the search is complete.
Next, the HOME environment variable is examined. If its value is valid, then it is prepended to the following set of relative paths. If HOME is not set, then the search continues with the value of the current user\u2019s home path as determined by getpwuid(). The home path, if any, determined by getpwuid() is prepended to the following set of relative paths. Searching completes successfully if any of these home-relative search paths is valid.
/.local/opae.cfg /.local/opae/opae.cfg /.config/opae/opae.cfg
Figure 105 HOME Relative Search Paths
Finally, the configuration file search continues with the following system-wide paths. If any of these paths is found to contain a configuration file, then searching completes successfully.
usr/local/etc/opae/opae.cfg /etc/opae/opae.cfg
Figure 106 System Search Paths
If the search exhausts all of the possible configuration file locations without finding a configuration file, then an internal default configuration is used. This internal default configuration matches that of the opae.cfg file shipped with the OPAE SDK.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#102-configuration-file-format","title":"10.2 Configuration File Format","text":"The OPAE SDK configuration file is stored in JSON formatted text. The file has two main sections: \u201cconfigs\u201d and \u201cconfigurations\u201d. The \u201cconfigs\u201d section is an array of strings. Each value in the \u201cconfigs\u201d array is a key into the data stored in the \u201cconfigurations\u201d section. If a key is present in \u201cconfigs\u201d, then that key is searched for and processed in \u201cconfigurations\u201d. If the key is not found in \u201cconfigs\u201d, then that section of \u201cconfigurations\u201d will not be processed, irrespective of whether it exists in \u201cconfigurations\u201d.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... } }, \u201cconfigs\u201d: [ \u201cc6100\u201d ] }
Figure 107 Keyed Configurations
Each keyed section in \u201cconfigurations\u201d has four top-level entries: \u201cenabled\u201d, \u201cplatform\u201d, \u201cdevices\u201d, \u201copae\u201d.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { \u201cenabled\u201d: true, \u201cplatform\u201d: \u201cIntel Acceleration Development Platform C6100\u201d, \u201cdevices\u201d: [ { \u201cname\u201d: \u201cc6100_pf\u201d, \u201cid\u201d: [ ... ] }, { \u201cname\u201d: \u201cc6100_vf\u201d, \u201cid\u201d: [ ... ] } ], \u201copae\u201d: { ... } } }, }
Figure 108 Configurations Format
The \u201cenabled\u201d key holds a Boolean value. If the value is false or if the \u201cenabled\u201d key is omitted, then that configuration is skipped when parsing the file. The \u201cplatform\u201d key holds a string that identifies the current configuration item as a product family. The \u201cdevices\u201d key contains the device descriptions.
\u201cdevices\u201d is an array of objects that contain a \u201cname\u201d and an \u201cid\u201d key. The \u201cname\u201d is a shorthand descriptor for a device PF or VF. The value of \u201cname\u201d appears elsewhere in the current \u201cconfigurations\u201d section in order to uniquely identify the device. \u201cid\u201d is an array of four strings, corresponding to the PCIe Vendor ID, Device ID, Subsystem Vendor ID, and Subsystem Device ID of the device. The entries corresponding to Vendor ID and Device ID must contain valid 16-bit hex integers. The entries corresponding to Subsystem Vendor ID and Subsystem Device ID may be 16-bit hex integers or the special wildcard string \u201c*\u201d, which indicates a don\u2019t care condition.
The remaining sections in this chapter outline the format of the \u201copae\u201d configurations key.
\u201cplugin\u201d: libopae-c and libopae-v
The \u201cplugin\u201d key in the \u201copae\u201d section of a configuration is an array of OPAE SDK plugin configuration data. Each item in the array matches one or more PF or VF devices to a plugin library module.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201cplugin\u201d: [ { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibxfpga.so\u201d, \u201cdevices\u201d: [ \u201cc6100_pf\u201d ], \u201cconfiguration\u201d: {} }, { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibopae-v.so\u201d, \u201cdevices\u201d: [ \u201cc6100_pf\u201d, \u201cc6100_vf\u201d ], \u201cconfiguration\u201d: {} } ], } } }, }
Figure 109 \"opae\" / \"plugin\" key/
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cplugin\u201d array entry is skipped, and parsing continues. The \u201cmodule\u201d key is a string that identifies the desired plugin module library for the entry. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section. The \u201cconfiguration\u201d key of the \u201cplugin\u201d section specifies a unique plugin-specific configuration. Currently, libopae-c and libopae-v use no plugin-specific config, so these keys are left empty.
\u201cfpgainfo\u201d: fpgainfo application
The \u201cfpgainfo\u201d key in the \u201copae\u201d section of a configuration is an array of fpgainfo plugin configuration data. Each item in the array matches one or more PF or VF devices to an fpgainfo plugin library module.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201cfpgainfo\u201d: [ { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibboard_c6100.so\u201d, \u201cdevices\u201d: [ { \u201cdevice\u201d: \u201cc6100_pf\u201d, \u201cfeature_id\u201d: \u201c0x12\u201d }, { \u201cdevice\u201d: \u201cc6100_vf\u201d, \u201cfeature_id\u201d: \u201c0x12\u201d } ] } ], } } }, }
Figure 110 \"opae\" / \"fpgainfo\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cfpgainfo\u201d array entry is skipped, and parsing continues. The \u201cmodule\u201d key is a string that identifies the desired fpgainfo module library for the entry. Each \u201cdevices\u201d array entry gives a PF/VF identifier in its \u201cdevice\u201d key and a DFL feature ID in its \u201cfeature_id\u201d key.
\u201cfpgad\u201d: fpgad daemon process
The \u201cfpgad\u201d key in the \u201copae\u201d section of a configuration is an array of fpgad plugin configuration data. Each item in the array matches one or more PF or VF devices to an fpgad plugin library module.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201cfpgad\u201d: [ { \u201cenabled\u201d: true, \u201cmodule\u201d: \u201clibfpgad-vc.so\u201d, \u201cdevices\u201d: [ \u201cc6100_pf\u201d ], \u201cconfiguration\u201d: { ... } } ], } } }, }
Figure 111 \"opae\" / \"fpgad\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cfpgad\u201d array entry is skipped, and parsing continues. The \u201cmodule\u201d key is a string that identifies the desired fpgad plugin module library for the entry. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section. The \u201cconfiguration\u201d key of the \u201cfpgad\u201d section specifies a unique plugin-specific configuration.
\u201crsu\u201d: rsu script
The \u201crsu\u201d key in the \u201copae\u201d section of a configuration is an array of rsu script configuration data. Each item in the array matches one or more PF devices to an rsu configuration.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201crsu\u201d: [ { \u201cenabled\u201d: true, \u201cdevices\u201d: [ \u201cc6100_pf\u201d ], \u201cfpga_default_sequences\u201d: \u201ccommon_rsu_sequences\u201d } ], } } }, \u201ccommon_rsu_sequences\u201d: [ ... ] }
Figure 112 \"opae\" / \"rsu\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201crsu\u201d array entry is skipped, and parsing continues. When disabled, the device(s) mentioned in that array entry will not be available for the rsu command. The \u201cdevices\u201d array lists one or more PF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section. The \u201cfpga_default_sequences\u201d key of the \u201crsu\u201d section specifies a JSON key. The configuration searches for that JSON key at the global level of the configuration file, and when found applies its value as the valid set of fpga boot sequences that can be used with the rsu fpgadefault subcommand.
C \u201cfpgareg\u201d: fpgareg script
The \u201cfpgareg\u201d key in the \u201copae\u201d section of a configuration is an array of fpgareg script configuration data. Each item in the array matches one or more PF/VF devices to an fpgareg configuration.
```C {
\u201cconfigurations\u201d: {
\u201cc6100\u201d: {\n\n ...\n\n \u201copae\u201d: {\n\n \u201cfpgareg\u201d: [\n\n {\n\n \u201cenabled\u201d: true,\n\n \u201cdevices\u201d: [ \u201cc6100_pf\u201d, \u201cc6100_vf\u201d ]\n\n }\n\n ],\n\n }\n\n}\n
},
} ```
Figure 113 \"opae\" / \"fpgareg\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201cfpgareg\u201d array entry is skipped, and parsing continues. When disabled, the device(s) mentioned in that array entry will not be available for the fpgareg command. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section.
\u201copae.io\u201d: opae.io application
The \u201copae.io\u201d key in the \u201copae\u201d section of a configuration is an array of opae.io configuration data. Each item in the array matches one or more PF/VF devices to an opae.io platform string.
{ \u201cconfigurations\u201d: { \u201cc6100\u201d: { ... \u201copae\u201d: { \u201copae.io\u201d: [ { \u201cenabled\u201d: true, \u201cdevices\u201d: [ \u201cc6100_pf\u201d, \u201cc6100_vf\u201d ] } ], } } }, }
Figure 114 \"opae\" / \"opae.io\" key
If the \u201cenabled\u201d key is false or if it is omitted, then that \u201copae.io\u201d array entry is skipped, and parsing continues. When disabled, the device(s) mentioned in that array entry will continue to be available for the opae.io command. The device(s) platform string will not be shown in the opae.io ls command. The \u201cdevices\u201d array lists one or more PF/VF identifiers. Each array value must be a string, and it must match a device that is described in the \u201cconfigurations\u201d \u201cdevices\u201d section.
Libxfpga \u2013 Updating the Metrics API
Edit libraries/plugins/xfpga/sysfs.c. Find the definition of the opae_id_to_hw_type() function. Update the function to add the new vendor/device ID to hw_type mapping.
This mapping is used by the SDK\u2019s metrics API to determine the method of accessing the board sensor information and is very specific to the underlying BMC implementation. It may be necessary to add a new hw_type value and to update the logic in libraries/plugins/xfpga/metrics.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#110-dfl-linux-kernel-drivers","title":"11.0 DFL Linux Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks such as DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality.
The OFS driver software can be found in the OFS repositories linux-dfl and linux-dfl-backport. These repositoryies have an associated OFS repository - linux-dfl that includes the following information:
- An description of the three available branch archetypes
- Configuration tweaks required while building the kernel
- A functional description of the available DFL framework
- Descriptions for all currently available driver modules that support FPGA DFL board solutions
- Steps to create a new DFL driver
- Steps to port a DFL driver patch
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#111-hardware-architecture","title":"11.1 Hardware Architecture","text":"The Linux Operating System treats the FPGA hardware as a PCIe* device. A predefined data structure, Device Feature List (DFL), allows for dynamic feature discovery in an Intel FPGA solution.
The Linux Device Driver implements PCIe Single Root I/O Virtualization (SR-IOV) for the creation of Virtual Functions (VFs). The device driver can release individual accelerators for assignment to virtual machines (VMs).
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#112-fpga-management-engine-fme","title":"11.2 FPGA Management Engine (FME)","text":"The FPGA Management Engine provides error reporting, reconfiguration, performance reporting, and other infrastructure functions. Each FPGA has one FME which is always accessed through the Physical Function (PF). The Intel Xeon\u00ae Processor with Integrated FPGA also performs power and thermal management. These functions are not available on the Intel Programmable Acceleration Card (PAC).
User-space applications can acquire exclusive access to the FME using open(), and release it using close(). Device access may be managed by standard Linux interfaces and tools.
If an application terminates without freeing the FME or Port resources, Linux closes all file descriptors owned by the terminating process, freeing those resources.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#113-port","title":"11.3 Port","text":"A Port represents the interface between two components: * The FPGA Interface Manager (FIM) which is part of the static FPGA fabric * The Accelerator Function Unit (AFU) which is the partially reconfigurable region
The Port controls the communication from software to the AFU and makes features such as reset and debug available.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#114-accelerator-function-unit-afu","title":"11.4 Accelerator Function Unit (AFU)","text":"An AFU attaches to a Port. The AFU provides a 256 KB memory mapped I/O (MMIO) region for accelerator-specific control registers.
- Use
open() on the Port device to acquire access to an AFU associated with the Port device. - Use
close()on the Port device to release the AFU associated with the Port device. - Use
mmap() on the Port device to map accelerator MMIO regions.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#115-partial-reconfiguration-pr","title":"11.5 Partial Reconfiguration (PR)","text":"Use PR to reconfigure an AFU from a bitstream file. Successful reconfiguration has the following requirement:
- You must generate the reconfiguration AFU for the exact FIM. The AFU and FIM are compatible if their interface IDs match. You can verify this match by comparing the interface ID in the bitstream header against the interface ID that is exported by the driver in sysfs.
In all other cases PR fails and may cause system instability.
Platforms that support 512-bit Partial Reconfiguration require binutils >= version 2.25.
Close any software programs accessing the FPGA, including those running in a virtualized host before initiating PR. For virtualized environments, the recommended sequence is as follows:
- Unload the driver from the guest
- Release the VF from the guest
Releasing the VF from the guest while an application on the guest is still accessing its resources may lead to VM instabilities. We recommend closing all applications accessing the VF in the guest before releasing the VF.
- Disable SR-IOV
- Perform PR
- Enable SR-IOV
- Assign the VF to the guest
- Load the driver in the guest
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#116-fpga-virtualization","title":"11.6 FPGA Virtualization","text":"To enable accelerator access from applications running on a VM, create a VF for the port using the following process:
-
Release the Port from the PF using the associated ioctl on the FME device.
-
Use the following command to enable SR-IOV and VFs. Each VF can own a single Port with an AFU. In the following command, N is the number of Port released from the PF.
echo N > $PCI_DEVICE_PATH/sriov_numvfs\n
The number, 'N', cannot be greater than the number of supported VFs. This can be read from $PCI_DEVICE_PATH/sriov_totalvfs.
-
Pass the VFs through to VMs using hypervisor interfaces.
-
Access the AFU on a VF from applications running on the VM using the same driver inside the VM.
Creating VFs is only supported for port devices. Consequently, PR and other management functions are only available through the PF.
If assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices).
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#117-driver-organization","title":"11.7 Driver Organization","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1171-pcie-module-device-driver","title":"11.7.1 PCIe Module Device Driver","text":"FPGA devices appear as a PCIe devices. Once enumeration detects a PCIe PF or VF, the Linux OS loads the FPGA PCIe device driver. The device driver performs the following functions:
- Walks through the Device Feature List in PCIe device base address register (BAR) memory to discover features and their sub-features and creates necessary platform devices.
- Enables SR-IOV.
- Introduces the feature device infrastructure, which abstracts operations for sub-features and provides common functions to feature device drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1172-pcie-module-device-driver-functions","title":"11.7.2 PCIe Module Device Driver Functions","text":"The PCIe Module Device Driver performs the following functions:
- PCIe discovery, device enumeration, and feature discovery.
- Creates sysfs directories for the device, FME, and Port.
- Creates the platform driver instances, causing the Linux kernel to load their respective drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1173-fme-platform-module-device-driver","title":"11.7.3 FME Platform Module Device Driver","text":"The FME Platform Module Device Driver loads automatically after the PCIe driver creates the FME Platform Module. It provides the following features for FPGA management:
-
Power and thermal management, error reporting, performance reporting, and other infrastructure functions. You can access these functions via sysfs interfaces the FME driver provides.
-
Partial Reconfiguration. During PR sub-feature initialization, the FME driver registers the FPGA Manager framework to support PR. When the FME receives the relevant ioctl request from user-space, it invokes the common interface function from the FPGA Manager to reconfigure the AFU using PR.
-
Port management for virtualization (releasing/assigning port device).
After a port device is released, you can use the PCIe driver SR-IOV interfaces to create/destroy VFs.
For more information, refer to \"FPGA Virtualization\".
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1174-fme-platform-module-device-driver-functions","title":"11.7.4 FME Platform Module Device Driver Functions","text":"The FME Platform Module Device Driver performs the the following functions:
- Creates the FME character device node.
- Creates the FME sysfs files and implements the FME sysfs file accessors.
- Implements the FME private feature sub-drivers.
- FME private feature sub-drivers: * FME Header * Partial Reconfiguration * Global Error * Global Performance
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1175-port-platform-module-device-driver","title":"11.7.5 Port Platform Module Device Driver","text":"After the PCIe Module Device Driver creates the Port Platform Module device, the FPGA Port and AFU driver are loaded. This module provides an interface for user-space applications to access the individual accelerators, including basic reset control on the Port, AFU MMIO region export, DMA buffer mapping service, and remote debug functions.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1176-port-platform-module-device-driver-functions","title":"11.7.6 Port Platform Module Device Driver Functions","text":"The Port Platform Module Device Driver performs the the following functions:
- Creates the Port character device node.
- Creates the Port sysfs files and implements the Port sysfs file accessors.
- Implements the following Port private feature sub-drivers. * Port Header * AFU * Port Error * Signal Tap
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#1177-opae-fpga-driver-interface","title":"11.7.7 OPAE FPGA Driver Interface","text":"The user-space interface consists of a sysfs hierarchy and ioctl requests. Most kernel attributes can be accessed/modified via sysfs nodes in this hierarchy. More complex I/O operations are controlled via ioctl requests. The OPAE API implementation, libopae-c, has been designed to use this interface to interact with the OPAE FPGA kernel drivers.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#120-plugin-development","title":"12.0 Plugin Development","text":""},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#121-overview","title":"12.1 Overview","text":"Beginning with OPAE C library version 1.2.0, OPAE implements a plugin-centric model. This guide serves as a reference to define the makeup of an OPAE C API plugin and to describe a sequence of steps that one may follow when constructing an OPAE C API plugin.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#122-plugin-required-functions","title":"12.2 Plugin Required Functions","text":"An OPAE C API plugin is a runtime-loadable shared object library, also known as a module. On Linux systems, the dl family of APIs from libdl are used to interact with shared objects. Refer to \"man dlopen\" and \"man dlsym\" for examples of using the libdl API.
An OPAE C API plugin implements one required function. This function is required to have C linkage, so that its name is not mangled.
int opae_plugin_configure(opae_api_adapter_table *table, const char *config);\n
During initialization, the OPAE plugin manager component loads each plugin, searching for its opae_plugin_configure function. If none is found, then the plugin manager rejects that plugin. When it is found, opae_plugin_configure is called passing a pointer to a freshly-created opae_api_adapter_table and a buffer consisting of configuration data for the plugin.
The job of the opae_plugin_configure function is to populate the given adapter table with each of the plugin's API entry points and to consume and comprehend the given configuration data in preparation for initialization.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#123-opae-api-adapter-table","title":"12.3 OPAE API Adapter Table","text":"The adapter table is a data structure that contains function pointer entry points for each of the OPAE APIs implemented by a plugin. In this way, it adapts the plugin-specific behavior to the more general case of a flat C API. Note that OPAE applications are only required to link with opae-c. In other words, the name of the plugin library should not appear on the linker command line. In this way, plugins are truly decoupled from the OPAE C API, and they are required to adapt to the strict API specification by populating the adapter table only. No other linkage is required nor recommended.
adapter.h contains the definition of the opae_api_adapter_table. An abbreviated version is depicted below, along with supporting type opae_plugin:
typedef struct _opae_plugin {\nchar *path;\nvoid *dl_handle;\n} opae_plugin;\ntypedef struct _opae_api_adapter_table {\nstruct _opae_api_adapater_table *next;\nopae_plugin plugin;\nfpga_result (*fpgaOpen)(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result (*fpgaClose)(fpga_handle handle);\n...\nfpga_result (*fpgaEnumerate)(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n// configuration functions\nint (*initialize)(void);\nint (*finalize)(void);\n// first-level query\nbool (*supports_device)(const char *device_type);\nbool (*supports_host)(const char *hostname);\n} opae_api_adapter_table;\n
Some points worth noting are that the adapter tables are organized in memory by adding them to a linked list data structure. This is the use of the next structure member. (The list management is handled by the plugin manager.) The plugin structure member contains the handle to the shared object instance, as created by dlopen. This handle is used in the plugin's opae_plugin_configure to load plugin entry points. A plugin need only implement the portion of the OPAE C API that a target application needs. Any API entry points that are not supported should be left as NULL pointers (the default) in the adapter table. When an OPAE API that has no associated entry point in the adapter table is called, the result for objects associated with that plugin will be FPGA_NOT_SUPPORTED.
The following code illustrates a portion of the opae_plugin_configure for a theoretical OPAE C API plugin libfoo.so:
/* foo_plugin.c */\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\nadapter->fpgaOpen = dlsym(adapter->plugin.dl_handle, \"foo_fpgaOpen\");\nadapter->fpgaClose =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaClose\");\n...\nadapter->fpgaEnumerate =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaEnumerate\");\n...\nreturn 0;\n}\n
Notice that the implementations of the API entry points for plugin libfoo.so are prefixed with foo_. This is the recommended practice to avoid name collisions and to enhance the debugability of the application. Upon successful configuration, opae_plugin_configure returns 0 to indicate success. A non-zero return value indicates failure and causes the plugin manager to reject the plugin from futher consideration.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#124-plugin-optional-functions","title":"12.4 Plugin Optional Functions","text":"Once the plugin manager loads and configures each plugin, it uses the adapter table to call back into the plugin so that it can be made ready for runtime. This is the job of the opae_plugin_initialize entry point, whose signature is defined as:
int opae_plugin_initialize(void);\n
The function takes no parameters, as the configuration data was already given to the plugin by opae_plugin_configure. opae_plugin_initialize returns 0 if no errors were encountered during initialization. A non-zero return code indicates that plugin initialization failed. A plugin makes its opae_plugin_initialize available to the plugin manager by populating the adapter table's initialize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_initialize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->initialize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_initialize\");\n...\nreturn 0;\n}\n
If a plugin does not implement an opae_plugin_initialize entry point, then the initialize member of the adapter table should be left uninitialized. During plugin initialization, if a plugin has no opae_plugin_initialize entry in its adapter table, the plugin initialization step will be skipped, and the plugin will be considered to have initialized successfully.
Once plugin initialization is complete for all loaded plugins, the system is considered to be running and fully functional.
During teardown, the plugin manager uses the adapter table to call into each plugin's opae_plugin_finalize entry point, whose signature is defined as:
int opae_plugin_finalize(void);\n
opae_plugin_finalize returns 0 if no errors were encountered during teardown. A non-zero return code indicates that plugin teardown failed. A plugin makes its opae_plugin_finalize available to the plugin manager by populating the adapter table's finalize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_finalize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->finalize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_finalize\");\n...\nreturn 0;\n}\n
If a plugin does not implement an opae_plugin_finalize entry point, then the finalize member of the adapter table should be left uninitialized. During plugin cleanup, if a plugin has no opae_plugin_finalize entry point in its adapter table, the plugin finalize step will be skipped, and the plugin will be considered to have finalized successfully.
In addition to initialize and finalize, an OPAE C API plugin has two further optional entry points that relate to device enumeration. During enumeration, when a plugin is being considered for a type of device, the plugin may provide input on that decision by exporting an opae_plugin_supports_device entry point in the adapter table:
bool opae_plugin_supports_device(const char *device_type);\n
opae_plugin_supports_device returns true if the given device type is supported and false if it is not. A false return value from opae_plugin_supports_device causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_device is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_device(const char *device_type)\n{\nif (/* device_type is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_device =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_device\");\n...\nreturn 0;\n}\n
.. note::\n The `opae_plugin_supports_device` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\n
Similarly to determining whether a plugin supports a type of device, a plugin may also answer questions about network host support by populating an opae_plugin_supports_host entry point in the adapter table:
bool opae_plugin_supports_host(const char *hostname);\n
opae_plugin_supports_host returns true if the given hostname is supported and false if it is not. A false return value from opae_plugin_supports_host causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_host is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_host(const char *hostname)\n{\nif (/* hostname is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_host =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_host\");\n...\nreturn 0;\n}\n
.. note::\n The `opae_plugin_supports_host` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#125-plugin-construction","title":"12.5 Plugin Construction","text":"The steps required to implement an OPAE C API plugin, libfoo.so, are:
- Create foo_plugin.c: implements
opae_plugin_configure, opae_plugin_initialize, opae_plugin_finalize, opae_plugin_supports_device, and opae_plugin_supports_host as described in the previous sections. - Create foo_plugin.h: implements function prototypes for each of the plugin-specific OPAE C APIs.
/* foo_plugin.h */\nfpga_result foo_fpgaOpen(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result foo_fpgaClose(fpga_handle handle);\n...\nfpga_result foo_fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n
- Create foo_types.h: implements plugin-specific types for opaque data structures.
/* foo_types.h */\nstruct _foo_token {\n...\n};\nstruct _foo_handle {\n...\n};\nstruct _foo_event_handle {\n...\n};\nstruct _foo_object {\n...\n};\n
- Create foo_enum.c: implements
foo_fpgaEnumerate, foo_fpgaCloneToken, and foo_fpgaDestroyToken. - Create foo_open.c: implements
foo_fpgaOpen. - Create foo_close.c: implements
foo_fpgaClose. - Create foo_props.c: implements
foo_fpgaGetProperties, foo_fpgaGetPropertiesFromHandle, foo_fpgaUpdateProperties - Create foo_mmio.c: implements
foo_fpgaMapMMIO, foo_fpgaUnmapMMIO foo_fpgaWriteMMIO64, foo_fpgaReadMMIO64, foo_fpgaWriteMMIO32, foo_fpgaReadMMIO32. - Create foo_buff.c: implements
foo_fpgaPrepareBuffer, foo_fpgaReleaseBuffer, foo_fpgaGetIOAddress. - Create foo_error.c: implements
foo_fpgaReadError, foo_fpgaClearError, foo_fpgaClearAllErrors, foo_fpgaGetErrorInfo. - Create foo_event.c: implements
foo_fpgaCreateEventHandle, foo_fpgaDestroyEventHandle, foo_fpgaGetOSObjectFromEventHandle, foo_fpgaRegisterEvent, foo_fpgaUnregisterEvent. - Create foo_reconf.c: implements
foo_fpgaReconfigureSlot. - Create foo_obj.c: implements
foo_fpgaTokenGetObject, foo_fpgaHandleGetObject, foo_fpgaObjectGetObject, foo_fpgaDestroyObject, foo_fpgaObjectGetSize, foo_fpgaObjectRead, foo_fpgaObjectRead64, foo_fpgaObjectWrite64. - Create foo_clk.c: implements
foo_fpgaSetUserClock, foo_fpgaGetUserClock.
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#130-building-a-sample-application","title":"13.0 Building a Sample Application","text":"Building a sample application The library source includes code samples. Use these samples to learn how to call functions in the library. Build and run these samples as quick sanity checks to determine if your installation and environment are set up properly.
In this guide, we will build hello_fpga.c. This is the \"Hello World!\" example of using the library. This code searches for a predefined and known AFU called \"Native Loopback Adapter\" on the FPGA. If found, it acquires ownership and then interacts with the AFU by sending it a 2MB message and waiting for the message to be echoed back. This code exercises all major components of the API except for AFU reconfiguration: AFU search, enumeration, access, MMIO, and memory management.
You can also find the source for hello_fpga in the samples directory of the OPAE SDK repository on GitHub.
int main(int argc, char *argv[])\n{\nfpga_properties filter = NULL;\nfpga_token afu_token;\nfpga_handle afu_handle;\nfpga_guid guid;\nuint32_t num_matches;\nvolatile uint64_t *dsm_ptr = NULL;\nvolatile uint64_t *status_ptr = NULL;\nvolatile uint64_t *input_ptr = NULL;\nvolatile uint64_t *output_ptr = NULL;\nuint64_t dsm_wsid;\nuint64_t input_wsid;\nuint64_t output_wsid;\nfpga_result res = FPGA_OK;\nif (uuid_parse(NLB0_AFUID, guid) < 0) {\nfprintf(stderr, \"Error parsing guid '%s'\\n\", NLB0_AFUID);\ngoto out_exit;\n}\n/* Look for accelerator by its \"afu_id\" */\nres = fpgaGetProperties(NULL, &filter);\nON_ERR_GOTO(res, out_exit, \"creating properties object\");\nres = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nON_ERR_GOTO(res, out_destroy_prop, \"setting object type\");\nres = fpgaPropertiesSetGuid(filter, guid);\nON_ERR_GOTO(res, out_destroy_prop, \"setting GUID\");\n/* TODO: Add selection via BDF / device ID */\nres = fpgaEnumerate(&filter, 1, &afu_token, 1, &num_matches);\nON_ERR_GOTO(res, out_destroy_prop, \"enumerating accelerators\");\nif (num_matches < 1) {\nfprintf(stderr, \"accelerator not found.\\n\");\nres = fpgaDestroyProperties(&filter);\nreturn FPGA_INVALID_PARAM;\n}\n/* Open accelerator and map MMIO */\nres = fpgaOpen(afu_token, &afu_handle, 0);\nON_ERR_GOTO(res, out_destroy_tok, \"opening accelerator\");\nres = fpgaMapMMIO(afu_handle, 0, NULL);\nON_ERR_GOTO(res, out_close, \"mapping MMIO space\");\n/* Allocate buffers */\nres = fpgaPrepareBuffer(afu_handle, LPBK1_DSM_SIZE,\n(void **)&dsm_ptr, &dsm_wsid, 0);\nON_ERR_GOTO(res, out_close, \"allocating DSM buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&input_ptr, &input_wsid, 0);\nON_ERR_GOTO(res, out_free_dsm, \"allocating input buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&output_ptr, &output_wsid, 0);\nON_ERR_GOTO(res, out_free_input, \"allocating output buffer\");\nprintf(\"Running Test\\n\");\n/* Initialize buffers */\nmemset((void *)dsm_ptr, 0, LPBK1_DSM_SIZE);\nmemset((void *)input_ptr, 0xAF, LPBK1_BUFFER_SIZE);\nmemset((void *)output_ptr, 0xBE, LPBK1_BUFFER_SIZE);\ncache_line *cl_ptr = (cache_line *)input_ptr;\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE / CL(1); ++i) {\ncl_ptr[i].uint[15] = i+1; /* set the last uint in every cacheline */\n}\n/* Reset accelerator */\nres = fpgaReset(afu_handle);\nON_ERR_GOTO(res, out_free_output, \"resetting accelerator\");\n/* Program DMA addresses */\nuint64_t iova;\nres = fpgaGetIOAddress(afu_handle, dsm_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting DSM IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_AFU_DSM_BASEL, iova);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_AFU_DSM_BASEL\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 0);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 1);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaGetIOAddress(afu_handle, input_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting input IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_SRC_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_SRC_ADDR\");\nres = fpgaGetIOAddress(afu_handle, output_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting output IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_DST_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_DST_ADDR\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_NUM_LINES, LPBK1_BUFFER_SIZE / CL(1));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_NUM_LINES\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CFG, 0x42000);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nstatus_ptr = dsm_ptr + DSM_STATUS_TEST_COMPLETE/8;\n/* Start the test */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 3);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Wait for test completion */\nwhile (0 == ((*status_ptr) & 0x1)) {\nusleep(100);\n}\n/* Stop the device */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 7);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Check output buffer contents */\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE; i++) {\nif (((uint8_t*)output_ptr)[i] != ((uint8_t*)input_ptr)[i]) {\nfprintf(stderr, \"Output does NOT match input \"\n\"at offset %i!\\n\", i);\nbreak;\n}\n}\nprintf(\"Done Running Test\\n\");\n/* Release buffers */\nout_free_output:\nres = fpgaReleaseBuffer(afu_handle, output_wsid);\nON_ERR_GOTO(res, out_free_input, \"releasing output buffer\");\nout_free_input:\nres = fpgaReleaseBuffer(afu_handle, input_wsid);\nON_ERR_GOTO(res, out_free_dsm, \"releasing input buffer\");\nout_free_dsm:\nres = fpgaReleaseBuffer(afu_handle, dsm_wsid);\nON_ERR_GOTO(res, out_unmap, \"releasing DSM buffer\");\n/* Unmap MMIO space */\nout_unmap:\nres = fpgaUnmapMMIO(afu_handle, 0);\nON_ERR_GOTO(res, out_close, \"unmapping MMIO space\");\n/* Release accelerator */\nout_close:\nres = fpgaClose(afu_handle);\nON_ERR_GOTO(res, out_destroy_tok, \"closing accelerator\");\n/* Destroy token */\nout_destroy_tok:\nres = fpgaDestroyToken(&afu_token);\nON_ERR_GOTO(res, out_destroy_prop, \"destroying token\");\n/* Destroy properties object */\nout_destroy_prop:\nres = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(res, out_exit, \"destroying properties object\");\nout_exit:\nreturn res;\n}\n
Linking with the OPAE library is straightforward. Code using this library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, the minimalist compile and link line should look like:
gcc -std=c99 hello_fpga.c -I/usr/local/include -L/usr/local/lib -lopae-c -luuid -ljson-c -lpthread -o hello_fpga\n
The API uses some features from the C99 language standard. The -std=c99 switch is required if the compiler does not support C99 by default.
Third-party library dependency: The library internally uses libuuid and libjson-c. But they are not distributed as part of the library. Make sure you have these libraries properly installed. The -c flag may not be necessary depending on the platform being used.
sudo ./hello_fpga -c\nRunning Test\nRunning on bus 0x08.\nAFU NLB0 found @ 28000\nDone Running Test\n
To run the hello_fpga application: sudo ./hello_fpga\nRunning Test\nDone\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#appendix-a-integrating-an-n6001-based-custom-platform-with-dfl-and-opae","title":"Appendix A - Integrating an N6001 Based Custom Platform with DFL and OPAE","text":"The process of adding a custom device to the DFL framework (backport or not) and OPAE SDK requires changes to files in several locations. In this section we will walk through the additions necessary to instruct the kernel's probe and match function to associate your new device with OPAE, choose the correct OPAE plugin to associate with your board, and change basic descriptors to properly display the name of your new custom platform when using OPAE's management interfaces.
This section does not require useage of an entirely new platform - we will use the Intel N6001's FIM design as our base, and alter only those settings in the PCIe IP necessary to change the following PCIe IDs for both the PF and VF:
- Vendor ID
- Device ID
- Subsystem Vendor ID
- Subsystem Device ID
This document will not cover the process by which a FIM can be modified to support these new IDs. Refer to the Shell Developer Guide: Agilex\u00ae 7 SoC Attach: Open FPGA Stack or Shell Developer Guide for Open FPGA Stack: Intel\u00ae FPGA SmartNIC N6000-PL / Intel\u00ae FPGA SmartNIC N6001-PL PCIe Attach FIM Developer guide for an overview of this process. The following sections assume you have a FIM that has been configured with new IDs. This FIM can be loaded onto your N6001 board using either the SOF via a USB-BlasterII cable and the Quartus Programmer, or by using the OPAE command fpgasupdate consuming a compatible binary programming file. The following values will be used as our new device IDs.
ID Type PF0 PF0-VF Vendor ID 0xff00 0xff00 Device ID 0x1234 0x5555 Subsystem Vendor ID 0xff00 0xff00 Subsystem Device ID 0x0x5678 0x5678 The only value that differs between PF0 and PF0-VF in this example is the Device ID, and all other values do not currently exist in either the OPAE SDK or linux-dfl drivers. You will need to download the OPAE SDK and linux-dfl sources from GitHub and modify their contents. We will be using a validated Agilex OFS release to pair our OPAE SDK and linux-dfl versions together. Refer to the below table for a list of the exact version we will use for each software component and where you can learn how to build and install. Your versions may not match those in the document.
Software Version Build Instructions OPAE SDK 2.3.0-1 4.0 OPAE Software Development Kit linux-dfl ofs-2022.3-2 3.0 Intel OFS DFL Kernel Drivers The following steps will enable your device to use the OPAE SDK. We will call our new device the \"Intel FPGA Programmable Acceleration Card N6002\". This device is identical to the Intel FPGA Programmable Acceleration Card N6001, and will use the pre-existing plugins and feature ID associated with that device. We will also use the enum value FPGA_HW_DCP_N6002 to describe our new board in the code. These plugins can be customized as you become more familiar with the OPAE SDK software.
- Download the OPAE SDK from GitHub. File paths assume the user is in the directory
opae-sdk. - Open the file
binaries/opae.io/opae/io/config.py. Add a new configuration entry under DEFAULT_OPAE_IO_CONFIG. Save and exit.
Example:
(0xff00, 0x1234, 0xff00, 0x5678) : {\n'platform': 'Intel FPGA Programmable Acceleration Card N6002'\n},\n
- Open the file
libraries/libopae-c/cfg-file.c. Add two new entries (one for PF0 and PF0-VF) under STATIC libopae_config_data default_libopae_config_table[]. Add two new entries under STATIC fpgainfo_config_data default_fpgainfo_config_table[]. Save and exit.
Example:
STATIC fpgainfo_config_data default_fpgainfo_config_table[] = {\n...\n{ 0xff00, 0x1234, 0xff00, 0x5678, 0x12, \"libboard_n6000.so\", NULL,\n\"Intel FPGA Programmable Acceleration Card N6002\" }, // N6002 PF0\n{ 0xff00, 0x5555, 0xff00, 0x5678, 0x12, \"libboard_n6000.so\", NULL,\n\"Intel FPGA Programmable Acceleration Card N6002\" }, // N6002 PF0-VF\n
Example:
STATIC libopae_config_data default_libopae_config_table[] = {\n...\n{ 0xff00, 0x1234, 0xff00, 0x5678, \"libxfpga.so\", \"{}\", 0 } , // N6002 PF0\n{ 0xff00, 0x5555, 0xff00, 0x5678, \"libxfpga.so\", \"{}\", 0 } , // N6002 PF0-VF\n
4. Open the file libraries/plugins/xfpga/metrics/metric_utils.c. Add one entry to the switch case under enum_fpga_metrics(fpga_handle handle). The enum value used should match the enum set in step 6. Add a new condition to the if statement beginning if (((_fpga_enum_metric->hw_type == FPGA_HW_DCP_N3000). Save and exit. Example:
// DCP VC DC\ncase FPGA_HW_DCP_N3000:\ncase FPGA_HW_DCP_D5005:\ncase FPGA_HW_DCP_N6002:\ncase FPGA_HW_DCP_N5010: {\n...\n
Example:
if (((_fpga_enum_metric->hw_type == FPGA_HW_DCP_N3000) ||\n(_fpga_enum_metric->hw_type == FPGA_HW_DCP_D5005) ||\n(_fpga_enum_metric->hw_type == FPGA_HW_DCP_N6002) ||\n(_fpga_enum_metric->hw_type == FPGA_HW_DCP_N5010)) &&\n
5. Open the file libraries/plugins/xfpga/sysfs.c. Add a new set of switch cases under enum fpga_hw_type opae_id_to_hw_type(uint16_t vendor_id, uint16_t device_id). The enum value used should match the enum value set in step 6. Save and exit. Example:
if (vendor_id == 0xff00) { switch (device_id) {\ncase 0x1234:\ncase 0x5555:\nhw_type = FPGA_HW_DCP_N6002;\nbreak;\ndefault:\nOPAE_ERR(\"unknown device id: 0x%04x\", device_id);\n
6. Open the file libraries/plugins/xfpga/types_int.h. Add your new enum value under enum fpga_hw_type. Save and exit. Example:
enum fpga_hw_type {\nFPGA_HW_MCP,\nFPGA_HW_DCP_RC,\nFPGA_HW_DCP_D5005,\nFPGA_HW_DCP_N3000,\nFPGA_HW_DCP_N5010,\nFPGA_HW_DCP_N6002,\nFPGA_HW_UNKNOWN\n};\n
7. Open the file opae.cfg. Create a new entry for device \"n6002\" by copying the entry for \"n6001,\"\" substituting our new values. Add one new entry under \"configs\" for the name \"n6002.\" Save and exit. Example:
\"n6002\": {\n\"enabled\": true,\n\"platform\": \"Intel Acceleration Development Platform N6002\",\n\"devices\": [\n{ \"name\": \"n6002_pf\", \"id\": [ \"0xff00\", \"0x1234\", \"0xff00\", \"0x5678\" ] },\n{ \"name\": \"n6002_vf\", \"id\": [ \"0xff00\", \"0x5555\", \"0xff00\", \"0x5678\" ] }\n],\n\"opae\": {\n\"plugin\": [\n{\n\"enabled\": true,\n\"module\": \"libxfpga.so\",\n\"devices\": [ \"n6002_pf\" ],\n\"configuration\": {}\n},\n{\n\"enabled\": true,\n\"module\": \"libopae-v.so\",\n\"devices\": [ \"n6002_pf\", \"n6002_vf\" ],\n\"configuration\": {}\n}\n],\n\"fpgainfo\": [\n{\n\"enabled\": true,\n\"module\": \"libboard_n6000.so\",\n\"devices\": [\n{ \"device\": \"n6002_pf\", \"feature_id\": \"0x12\" },\n{ \"device\": \"n6002_vf\", \"feature_id\": \"0x12\" }\n]\n}\n],\n\"fpgad\": [\n{\n\"enabled\": true,\n\"module\": \"libfpgad-vc.so\",\n\"devices\": [ \"n6002_pf\" ],\n\"configuration\": {\n\"cool-down\": 30,\n\"get-aer\": [ \"setpci -s %s ECAP_AER+0x08.L\",\n\"setpci -s %s ECAP_AER+0x14.L\" ],\n\"disable-aer\": [ \"setpci -s %s ECAP_AER+0x08.L=0xffffffff\",\n\"setpci -s %s ECAP_AER+0x14.L=0xffffffff\" ],\n\"set-aer\": [ \"setpci -s %s ECAP_AER+0x08.L=0x%08x\",\n\"setpci -s %s ECAP_AER+0x14.L=0x%08x\" ],\n\"sensor-overrides\": [],\n\"monitor-seu\": false\n}\n}\n],\n\"rsu\": [\n{\n\"enabled\": true,\n\"devices\": [ \"n6002_pf\" ],\n\"fpga_default_sequences\": \"common_rsu_sequences\"\n}\n],\n\"fpgareg\": [\n{\n\"enabled\": true,\n\"devices\": [ \"n6002_pf\", \"n6002_vf\" ]\n}\n],\n\"opae.io\": [\n{\n\"enabled\": true,\n\"devices\": [ \"n6002_pf\", \"n6002_vf\" ]\n}\n]\n}\n},\n
Example:
\"configs\": [\n\"mcp\",\n\"a10gx\",\n\"d5005\",\n\"n3000\",\n\"n5010\",\n\"n5013\",\n\"n5014\",\n\"n6000\",\n\"n6001\",\n\"n6002\",\n...\n
These conclude our integration of our new platform with the OPAE SDK. The next step is to download the source for linux-dfl (as shown above) and configure our new kernel's match and probe function to associate the DFL drivers with our new custom platform. The following file path assumes the user is in the directory linux-dfl 1. Open the file drivers/fpga/dfl-pci.c. Define a list of necessary ID values at the top of the file. Use these values to add two new entries under pci_device_id cci_pcie_id_tbl[], one for PF0 and the other for PF0-VF. Save and exit. Example:
/* N6002 IDS */\n#define PCIE_DEVICE_ID_PF_N6002 0x1234\n#define PCIE_VENDOR_ID_PF_N6002 0xff00\n#define PCIE_SUBDEVICE_ID_PF_N6002 0x5678\n#define PCIE_DEVICE_ID_VF_N6002 0x5555\n
Example:
static struct pci_device_id cci_pcie_id_tbl[] = {\n...\n{PCI_DEVICE_SUB(PCIE_VENDOR_ID_PF_N6002, PCIE_DEVICE_ID_PF_N6002,\nPCIE_VENDOR_ID_PF_N6002, PCIE_SUBDEVICE_ID_PF_N6002),}, //N6002 PF0\n{PCI_DEVICE_SUB(PCIE_VENDOR_ID_PF_N6002, PCIE_DEVICE_ID_VF_N6002,\nPCIE_VENDOR_ID_PF_N6002, PCIE_SUBDEVICE_ID_PF_N6002),}, //N6002 PF0-VF\n...\n
This concludes our integration our new platform with the linux-dfl driver set. Build and install the linux-dfl enabled kernel and the OPAE SDK userspace libraries as discussed in their relevant sections in the user guide linked above. If the above conditions have been met, and your N6001 board has been configured with this new \"N6002\" FIM, you should see the following output when running the command \"fpgainfo fme\" (your Bitstream ID, PR Interface ID, and Image Info entries may differ). Check that the board's display name at the top and values for Vendor/Device/SubVendor/Subdevice IDs are correct.
Intel Acceleration Development Platform N6002\nBoard Management Controller NIOS FW version: 3.11.0\nBoard Management Controller Build version: 3.11.0\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0xFF00\nDevice Id : 0x1234\nSubVendor Id : 0xFF00\nSubDevice Id : 0x5678\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010202C8AD72D7\nBitstream Version : 5.0.1\nPr Interface Id : 8df219e3-cf25-5e77-8689-f57102d54435\nBoot Page : user1\nFactory Image Info : a2b5fd0e7afca4ee6d7048f926e75ac2\nUser1 Image Info : 9804075d2e8a71a192ec89602f2f5544\nUser2 Image Info : 9804075d2e8a71a192ec89602f2f5544\n
"},{"location":"hw/common/reference_manual/ofs_sw/mnl_sw_ofs/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/","title":"oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#10-overview","title":"1.0 Overview","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a reference manual for platform designers wanting to enable oneAPI support on their Open FPGA Stack(OFS) platforms. The document describes essential hardware and software components required for enabling this design flow using OFS.
Please use table below as a quick reference for the contents in this document.
Table 1-1: Document Overview
Concept Description Sections Overview, Prerequisites Covers the flow of the document, important terms and prerequisite knowledge for this manual * 1.0 Overview * 1.1 About this Document * 1.2 Terminology * 1.3 Prerequisites Introduction to the oneAPI FPGA Acceleration Flow using Open FPGA Stack Covers the introduction to accelerating oneAPI applications on an FPGA platform design using Open FPGA Stack. This gives an overview of the complete stack consisting of the oneAPI application and explains the high level tool . it also covers an introduction to what the oneAPI Accelerator Support Package 1.4 Introduction to oneAPI on Open FPGA Stack(OFS) * 1.5 Introduction to oneAPI Accelerator Support Package(ASP) oneAPI Accelerator Support Package Design Fundamentals This covers the information on design elements required to ensure the FPGA hardware design is functional with the oneAPI base toolkit. It covers both the components required in hardware design as well as in the software layer that oneAPI runtime requires * 2.0 XML Files in oneAPI ASP * 2.1 board_spec.xml File * 2.2 board_env.xml File * 3.0 oneAPI ASP Hardware * 3.1 Host to External Memory Interface(EMIF) * 3.2 Host to Kernel Interface * 3.3 Kernel to External Memory Interface * 4.0 oneAPI ASP Software * 4.1 Memory Mapped Device(MMD) Layer * 4.2 Board Utilities oneAPI ASP Implementation on OFS Reference Platforms Provides detailed explanation of the oneAPI ASP design for OFS reference platforms. These sections cover information about the harware as well as software design files in the oneapi-asp repository * 5.0 oneapi-asp Implementation Details * 5.1 oneapi-asp Directory Structure * 5.2 oneapi-asp Build Flow * 5.3 oneapi-asp Hardware Implementation * 5.4 oneapi-asp Memory Mapped Device(MMD) Layer Implementation * 5.5 oneapi-asp Utilities Implementation Appendix This currently has information about additional features provided in the oneapi-asp repository. Appendix A covers debug capabilities in the MMD. Appendix B has information about a new functionality added in the oneapi-asp tag ofs-2024.2-1 called oneAPI ASP Editor, enabling a GUI based approach to design oneAPI ASPs * Appendix A: Memory Mapped Device(MMD) Layer Debug Variables * Appendix B: oneAPI Accelerator Support Package(ASP) Editor Note: Table 1-1 in oneAPI Accelerator Support Package (ASP): Getting Started User Guide lists OFS reference platforms.
For more information about developing application kernels for FPGA using oneAPI Base Toolkit (Base Kit) refer to Intel\u00ae FPGA Add-on for oneAPI Base Toolkit webpage.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#12-terminology","title":"1.2 Terminology","text":"This table defines some of the common terms used when discussing OFS.
Table 1-2: Terminology
Term Abbreviation Description Open FPGA Stack OFS A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. Accelerator Functional Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. FPGA Interface Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. High Level Design HLD For the purpose of this guide, this term refers to designing with High Level Design tools like oneAPI Base Toolkit (Base Kit). oneAPI Accelerator Support Package oneAPI ASP A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and other OFS hardware, software components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Intel\u00ae FPGA Basic Building Blocks BBB Basic Building Blocks (BBB) for Altera\u00ae FPGAs is a suite of application building blocks and shims like Memory Properties Factory (MPF). BBB Memory Properties Factory BBB MPF Intel\u00ae FPGA BBB MPF block provides features like virtual to physical address (VTP), ordering read responses, read/write hazard detection, and masked (partial) writes. oneapi-asp uses MPF VTP feature. Open Programmable Acceleration Engine Software Development Kit OPAE SDK A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. Platform Interface Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Device Feature List DFL A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. Best Known Configuration BKC The exact hardware configuration that the solution has been optimized and validated against. SYCL - SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL\u2122) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Installable Client Driver ICD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports the OpenCL ICD extension from the Khronos Group\u2122. The OpenCL ICD extension allows you to have multiple OpenCL implementations on your system. With the OpenCL ICD Loader Library, you may choose from a list of installed platforms and execute OpenCL API calls that are specific to your OpenCL implementation of choice. FPGA Client Driver FCD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports FPGA Client Driver(FCD) extension. FCD allows the runtime to automatically find and load the oneAPI ASP libraries at host run time Note: oneapi-asp was referred to as ofs-hld-shim in OFS (for Agilex\u00ae 7 FPGA & Stratix\u00ae 10 FPGA) and OpenCL AFU Shim (ofs-opencl-afu-shim) in OFS early access(EA) release (for Stratix\u00ae 10 FPGA with Intel\u00ae FPGA PAC D5005 as reference platform).
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#13-prerequisites","title":"1.3 Prerequisites","text":"The content in this manual requires readers to be familiar with:
- Hardware and software components of Open FPGA Stack, especially the following:
- FPGA Interface Manager(FIM)
- Stratix\u00ae 10 FPGA:
- Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs
- Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs
- Agilex\u00ae 7 FPGA:
- Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs
- Accelerator Functional Unit(AFU)
- Stratix\u00ae 10 FPGA: Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs
- Agilex\u00ae 7 FPGA: Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs
- Software Reference Manual: Open FPGA Stack (OPAE SDK and Linux-DFL section)
- ofs-platform-afu-bbb
In addition to above, developers must be familiar with the following tools & concepts:
- Quartus\u00ae Prime Design Software (Quartus\u00ae software revisions, Platform Designer, compilation Flows, timing analysis, compilation reports, understanding FPGA resource utilization, programming Altera\u00ae FPGAs)
- Partial Reconfiguration (PR)
- FPGA Peripheral IPs (PCIe, External Memory IP, Ethernet IP)
- Avalon\u00ae Interface
- Scripting (TCL, Python, Shell scripts)
- Verilog, SystemVerilog
- C++
- Familiarity with SYCL
- Familiarity with oneAPI compilation process for FPGAs & oneAPI code samples
- Familiarity with oneAPI Accelerator Support Package (ASP): Getting Started User Guide
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#14-introduction-to-oneapi-on-open-fpga-stackofs","title":"1.4 Introduction to oneAPI on Open FPGA Stack(OFS)","text":"The Intel\u00ae oneAPI Base Toolkit (Base Kit) is a core set of tools and libraries for developing high-performance, data-centric applications across diverse architectures (CPUs, GPUs and FPGAs). It features an industry-leading C++ compiler that implements SYCL, an evolution of C++ for heterogeneous computing.
Figure 1-1 shows the high-level representation of oneAPI application developers using FPGAs for acceleration. The runtime flow consists of a host code running on a processor and an application kernel code running on an FPGA. Open FPGA Stack enables vendors to enable support for this flow on their platforms.
Figure 1-1: oneAPI Application on OFS Platforms
oneAPI Base Toolkit (Base Kit) consists of a compiler and runtime environment. The compiler converts a SYCL kernel (FPGA application code) into a hardware circuit. This hardware circuit requires additional logic to communicate with the runtime and FPGA board peripherals. This additional logic is provided by oneAPI Accelerator Support Package(oneAPI ASP). oneAPI ASP consists of hardware components that enable this generated hardware circuit to communicate with the host processor as well as software components that enable the runtime to identify and communicate with the kernel.
Figure 1-2 shows the workload design steps and steps in which the oneAPI Base Toolkit (Base Kit) requires oneAPI ASP as input. For more information about workload development and how workload developers target a specific platform during compilation, refer to Intel oneAPI Programming Guide and Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs. The next section introduces oneAPI ASP.
Figure 1-2: High Level Design Flow for FPGAs with oneAPI Base Toolkit (Base Kit)
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#15-introduction-to-oneapi-accelerator-support-packageasp","title":"1.5 Introduction to oneAPI Accelerator Support Package(ASP)","text":"As mentioned in previous section, oneAPI ASP is a collection of hardware and software components that interface with the hardware circuit generated by the oneAPI compiler. The hardware circuit generated by the oneAPI compiler from a oneAPI kernel is referred to as the kernel system. While the kernel system consists of logic controlled by the workload developer's specifications, the kernel system interfaces are generated by the oneAPI compiler based on specifications provided by the oneAPI ASP designer. These specifications are input to the compiler using XML files (discussed in section 2.0).
Note: All the interfaces generated by the oneAPI compiler are Avalon\u00ae Interfaces.
Figure 1-3: Kernel System Interfaces
Figure 1-3 shows a high-level representation of an OFS hardware design and interfaces to/from kernel_system. The numbered arrows depict the following:
- Path 1 represents host-to-External Memory Interface (EMIF)
- Path 2 represents the host to kernel interface
- Path 3 represents kernel to EMIF
- Path 4 represents kernel to Unified Shared Memory (USM) Interface
- Path 5 represents kernel to HSSI interface
oneAPI ASP hardware components can be divided into 3 categories:
- RTL components: constituting various interface logic, for example, host to External Memory Interface (EMIF), kernel to EMIF interface, host to kernel interface, kernel to host memory interface as well as additional components to handle kernel control signals and perform Direct Memory Access (DMA)
- XML files: for describing hardware interfaces and compilation environment to oneAPI Base Toolkit (Base Kit)
- Scripts: to control compile flow
In addition to the hardware components, a software layer is required for handling I/O operations between oneAPI runtime and the board. The oneAPI ASP software layer can be divided into 2 categories:
- Memory Mapped Device (MMD) Layer: required by the host & runtime to communicate with the oneAPI kernel & other oneAPI ASP hardware registers
- oneAPI ASP utilities: required to setup and diagnose the board
The MMD uses API provided by OPAE SDK to communicate with the device. The FPGA driver is provided by the linux-DFL kernel driver.
Figure 1-4 shows how the above oneAPI ASP components tie into Open FPGA Stack.
Figure 1-4: Open FPGA Stack (OFS) components
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#20-xml-files-in-oneapi-asp","title":"2.0 XML Files in oneAPI ASP","text":"The kernel system interfaces generated by the oneAPI compiler are based on specifications provided by oneAPI ASP developer. An XML file, called board_spec.xml is used to pass the specifications to the oneAPI compiler. oneAPI ASP developers must create this XML file for their boards.
In addition to board_spec.xml, the oneAPI Base Toolkit (Base Kit) relies on another XML file called board_env.xml to get information about the board environment. The board_env.xml file helps the runtime setup board installation.
The next section explains the contents of board_spec.xml. Section 2.2 covers contents of board_env.xml file.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#21-board_specxml-file","title":"2.1 board_spec.xml File","text":"A typical board_spec.xml structure is shown in Fig 2-1. In addition to kernel system interfaces, the board_spec.xml file is also used to specify other compilation details like Quartus\u00ae Prime Pro Edition Software version used in platform design, compilation scripts to help control Quartus\u00ae software compile flows, FPGA resource utilization details.
Elements of board_spec.xml file are summarized in table 2-1. Each element has additional attributes and parameters. Details are covered in respective sections for each element.
Figure 2-1: board_spec.xml File Structure
Table 2-1: Elements of board_spec.xml
Element Use of Element Attributes board Used to specify a name for the board and the version of Quartus\u00ae Prime Pro Edition Software used to develop the platform design. This board name is used to identify the board design to be compiled and must match the name of the directory in which board_spec.xml resides. version, name compile Used to describe different compile flows project, revision, qsys_file, generic_kernel, generate_cmd, synthesize_cmd, auto_migrate device Used to specify the FPGA device model file for the FPGA part on the board. device_model, used_resources global_mem Different attributes in this element are used to provide details about the external memory used as the global memory for the FPGA oneAPI kernel/application. name, max_bandwidth, interleaved_bytes, config_addr, default, interface host Used to specify the offset at which the kernel resides. kernel_config interfaces Used to specify control signals to oneAPI kernels interface, kernel_clk_reset channels Used to describe interface to stream data directly between kernels and I/O interface The compiler expects a separate board_spec.xml file for every board variant a platform supports. Board variants are different hardware design implementations for the same platform, a oneAPI ASP can have multiple board variants. A oneAPI kernel developer can select the board variant suitable for their application at compile time.
A board_spec.xml file must be located at the top most level of each board variant's hardware directory (the hardware directory is specified by board_env.xml, please refer to section 2.2 for details on hardware element). For example, a separate board_spec.xml file for each board variant for OFS reference platforms is located in oneapi-asp/Platform-Name/hardware/Board-Variant/ directory, where Platform-Name is n6001, fseries-dk, iseries-dk for OFS targeting Agilex\u00ae 7 FPGA and d5005 for OFS targeting Stratix\u00ae 10 FPGA.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#211-board-element","title":"2.1.1 board Element","text":"The board element of the board_spec.xml file provides the Quartus\u00ae Prime Pro Edition Software version and the name of the board.
Table 2-2: Attributes for the board Element
Attribute Description version The version of the board. The board version should match the version of the Quartus\u00ae Prime Pro Edition Software you use to develop the platform design. The oneAPI compiler uses this value to perform environment checks for supported version during application compile name The name of the accelerator board, which must match the name of the directory in which the board_spec.xml file resides. The name must contain a combination of only letters, numbers, underscores (_), hyphens (-), or periods (.) (for example, ofs_n6000). Example below shows the board element populated for a board designed with Quartus\u00ae Prime Pro Edition Software version 22.3 and board variant named \"Agilex_brd1\".
Note: A board variant name is different from a platform directory name. Please see Note in section 2.2 for more information on board variants.
Figure 2-2: board Element
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#212-compile-element","title":"2.1.2 compile Element","text":"Depending on the application requirements, the design may have different compilation flows and different design settings for each flow (for example, there can be a flat flow without partial reconfiguration support or a flow with partitions in the design to enable partial reconfiguration). Designers can control the flow and its settings using scripts. To allow selection of compile flow during application compile & to describe control of Quartus\u00ae software compilation as well as registration, automigration, the compile element of the board_spec.xml file and its associated attributes and parameters are used.
Table 2-3: Attributes for compile Element
Attribute Description name Name of the compilation flow. This name can be used to differentiate between flows at oneAPI kernel compilation time. oneAPI compiler allows selecting a compile flow using -Xsbsp-flow option. project Name of the Quartus\u00ae software project file (.qpf) that the Quartus\u00ae Prime Pro Edition Software intends to compile. revision Name of the revision within the Quartus\u00ae software project that the Quartus\u00ae Prime Pro Edition Software compiles to generate the final bitstream. qsys_file Name of the Platform Designer file into which the oneAPI kernel is embedded. You have the option to assign a value of \"none\" to qsys_file if you do not require the Quartus\u00ae Prime Pro Edition Software to create a top-level .qsys file for your design. In this scenario, oneAPI compiler adds a .qip file into the Quartus\u00ae software project. In this case, the custom oneAPI ASP must manually instantiate the generated HDL entity (generated entity is in the kernel_system.v file). generic_kernel Set this value to 1 if you want the offline compiler to generate a common Verilog interface for all compilations. This setting is necessary in situations where you must set up design partitions around the kernel, such as in the Configuration via Protocol (CvP) flow. generate_cmd Command required to prepare for full compilation, such as to generate the Verilog files for the Platform Designer system into which the oneAPI kernel is embedded. synthesize_cmd Command required to generate the fpga.bin file from the Custom Platform. Usually, this command instructs the Quartus\u00ae Prime Pro Edition Software to perform a full compilation. auto_migrate *platform_type\u2014Choose this value based on the value referenced in the Reference Platform from which you derive your Custom Platform. Valid values are a10_ref, s10_ref, and none. *include fixes\u2014Comma-separated list of named fixes that you want to apply to the Custom Platform. *exclude fixes\u2014Comma-separated list of named fixes that you do not want to apply to the Custom Platform. Example below shows a populated compile element for a sample Quartus\u00ae software Project called ofs.qpf, the Quartus\u00ae software revision to be compiled is called asp (asp.qsf). In this example, the compiler generates the kernel system (entity is called kernel_system) and this entity is instantiated manually in the Quartus\u00ae software project (e.g. in a file called kernel_wrapper.v), hence qsys_file is set to \"none\". The synthesize_cmd points to a script \"compile.tcl\" located in the same directory as the board_spec.xml, compile script performs all necessary system generation and compile steps for generation of final bitstream. The project directory snippet below is for demonstration only. The compile flow is named \"demo_flow\".
There can be multiple compile elements for the different compilation flows that a platform designer wishes to enable in their platform (e.g. different revisions with different Quartus\u00ae software settings or a PR revision).
Figure 2-3: compile Element
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#213-device-element","title":"2.1.3 device Element","text":"A device model(DM) file is an XML file that has the total resources on the device (i.e. ALMs, FFs, DSPs, RAMs). This is required for any FPGA part used in a oneAPI design. Most device model files are provided as part of the oneAPI Base Toolkit (Base Kit) installation ($INTELFPGAOCLSDKROOT/share/models/dm, where INTELFPGAOCLSDKROOT is set by the setvars.sh environment setup script provided by oneAPI toolkit). If the device model file for your part number is not included in $INTELFPGAOCLSDKROOT/share/models/dm, it must be created and placed in the same folder as board_spec.xml. A new device model file can be created using existing files as reference.
The device model file name must be specified in the device_model attribute of device element. The used_resources attribute is used to specify the resources being utilized by the oneAPI ASP and peripheral IPs. The utilization by non-kernel logic is calculated during platform design. The compiler utilizes the total resources from device model file and utilized resources in used_resources section to estimate the available resources for application kernel.
Table 2-4: Attributes for device Element
Attribute Description device_model The file name of the device model file that describes the available FPGA resources on the accelerator board. used_resources Reports the number of adaptive logic modules (ALMs), flip-flops, digital signal processor (DSP) blocks and RAM blocks that the board design consumes in the absence of any kernel. If you create a defined partition around all the board logic, you can obtain the used resources data from the Partition Statistics section of the Fitter report. Extract the information from the following parameters: * alms num \u2014 The number of logic ALMs used, excluding the number of ALMs with only their registers used. The value should correspond to [a]+[b]+[d] from part [A] of the Fitter Partition Statistics. * ffs num \u2014 The number of flip flops. * dsps num \u2014 The number of DSP blocks. * rams num \u2014 The number of RAM blocks. Example below shows the device element added for a Agilex\u00ae 7 FPGA based platform with device model file named \"agfb014r24a2e2vr0_dm.xml\". The number of used_resources are for demonstration purposes and are not to be used by oneAPI ASP developers.
Figure 2-4: device Element
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#214-interface-attribute","title":"2.1.4 interface Attribute","text":"Note: This is different from the interfaces element discussed in upcoming sections. In the board_spec.xml file, each global memory, channel or kernel connection is comprised of individual interfaces. For the global_mem, channels, and interfaces XML elements, an interface attribute must be included to specify the corresponding parameters for each connection.
Table 2-5: Parameters for interface attribute
Parameter Description Applicable Interface name * For global_mem: instance name of the Platform Designer component.* For channels: instance name of the Platform Designer component that has the channel interface.* For interfaces: name of the entity in which the kernel interface resides (for example, board). All port port parameter can be defined either inline in interface attribute or as a separate element in interface attribute. See section 2.1.4.1 for more information. All type * For global_mem: set to agent. * For channels: - Set to streamsource for a stream source that provides data to the kernel.- Set to streamsink for a stream sink interface that consumes data from the kernel.* For interfaces: set to either host, irq, or streamsource. All width * For global_mem: width of the memory interface in bits.* For channels: number of bits in the channel interface.* For interfaces: width of the kernel interface in bits. All waitrequest_allowance * For global_mem: [Optional] Amount of Avalon\u00ae-MM waitrequest allowance supported on the agent interface (that is, kernel-facing interface) of the clock-crossing bridge that spans between the memory and the kernel clock domains.* For kernel_cra: [Optional] Amount of Avalon\u00ae-MM waitrequest allowance that the kernel_cra agent interface must support.This parameter defaults to 0 if you do not specify it in the board_spec.xml file. A value of 0 indicates that this waitrequest allowance feature is disabled. All maxburst Maximum burst size for the agent interface.Attention: The value of width \u00f7 8 x maxburst must be less than the value of interleaved_bytes. global_mem address Starting address of the memory interface that corresponds to the host interface-side address.For example, address 0 should correspond to the bank1 memory host from the Memory Bank Divider. In addition, any non-zero starting address must abut the end address of the previous memory. global_mem size Size of the memory interface in bytes.* For global_mem: The sizes of all memory interfaces should be equal.* For interfaces: interface can have variable sizes. global_meminterfaces > Note: Support for size parameter for interface attribute is available in oneAPI Base Toolkit (Base Kit) version 2024.0 and beyond. latency_type If the memory interface has variable latency, set this parameter to average to signify that the specified latency is considered the average case. If the complete kernel-to-memory path has a guaranteed fixed latency, set this parameter to fixed. global_mem chan_id A string used to identify the channel interface. The string may have up to 128 characters. channels clock For the streamsource kernel interface type, the parameter specifies the name of the clock that the snoop stream uses. Usually, this clock is the kernel clock. interfaces Example for how the interface attribute is used in global_mem and interfaces elements is covered in section for these elements respectively.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#2141-port-parameter","title":"2.1.4.1 port Parameter","text":"As mentioned in Table 2-5, port parameter can be defined either inline in interface attribute or as a separate element in interface attribute. The definition method to use depends on the direction of the port.
- If the direction of the port is either
read or write, it must be a separate element in interface attribute. - If the direction is
readwrite, port must be inline with the port name in interface attribute. No direction specification is required.
Table below shows the port element attributes.
Table 2-6: port parameter
Parameter Description name * For global_mem: name of the Avalon\u00ae-MM interface in the Platform Designer component that corresponds to the interface attribute.* For channels: name of the streaming interface in the Platform Designer component.* For interfaces: name of the interface to the Kernel Interface Platform Designer component. For example, kernel_cra is the Avalon\u00ae-MM interface, and kernel_irq is an interrupt. direction Direction of the port. Valid values are: * \"r\" : Indicates read * \"w\" : Indicates write Snippet below shows the inline and separate element definitions of port parameter.
Note: Direction specification for port is available in oneAPI Base Toolkit (Base Kit) versions 2024.0 and beyond. For versions prior to oneAPI Base Toolkit (Base Kit) version 2024.0, only the default inline definition of port parameter is supported.
Examples for global_mem and interfaces elements in sections below use the inline definition of port.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#215-global_mem-element","title":"2.1.5 global_mem Element","text":"The global_mem element of the board_spec.xml file is used to provide information on the memory interfaces that connect to the kernel.
Note: For each global memory that the kernel accesses, you must include one interface element that describes its characteristics. The different attributes for global_mem element are discussed in table below.
Table 2-7: Attributes for global_mem Element
Attribute Description name The name FPGA application/kernel developer should use to identify the memory type. Each name must be unique and must comprise of less than 32 characters. max_bandwidth The maximum bandwidth, in megabytes per second (MB/s), of all global memory interfaces combined in their current configuration. The oneAPI compiler uses max_bandwidth to choose an architecture suitable for the application and the board. Compute this bandwidth value from datasheets of memories on your board. Example max_bandwidth calculation for a 64-bit DDR4 interface running at 1200 MHz: max_bandwidth = 1200 MHz x 2 x 64 bits \u00f7 8-bits = 19200 MB/s The max_bandwidth value will change based on global memory configuration, for example, if the memory configuration comprises of 4 banks of DDR4 configured as a single homogenous memory, the max_bandwidth will be 19200 x 4 (i.e. number of memory interfaces from kernel). Please see section 2.1.5.1 for more information on global memory configurations. Designers have the option to use block RAM instead of or in conjunction with external memory as global memory. The formula for calculating max_bandwidth for block RAM is max_bandwidth = block RAM speed x (block RAM interface size \u00f7 8 bits). Example max_bandwidth calculation for a 512-bit block RAM running at 100 MHz: max_bandwidth = 100 MHz x 512 bits \u00f7 8 bits = 6400 MB/s interleaved_bytes Include the interleaved_bytes attribute in the board_spec.xml file when you instantiate multiple interfaces(i.e. memory banks) for a given global memory system. This attribute controls the size of data that the offline compiler distributes across the interfaces. The offline compiler currently can interleave data across banks no finer than the size of one full burst. This attribute specifies this size in bytes and following are the recommended values: For two or fewer global memory banks: maxburst x width_bytes For four or more global memory banks: maxburst x width_bytes x 4 The interleaved_bytes value must be the same for the host interface and the kernels. Therefore, the configuration of the Memory Bank Divider must match the exported kernel agent interfaces in this respect (refer to section 3.1.1 for information about Memory Bank Divider) For block RAM, interleaved_bytes equals the width of the interface in bytes. config_addr The address of the ACL Mem Organization Control Platform Designer component (mem_org_mode) that the host software uses to configure memory. You may omit this attribute if your board has homogeneous memory; the software uses the default address (0x18) for this component. If your board has heterogeneous memory, there is a mem_org_mode component in the board system for each memory type. Enter the config_addr attribute and set it to the value of the base address of the mem_org_mode component(s). default Include this optional attribute and assign a value of 1 to set the global memory as the default memory interface. The default memory must start at address 0x0. If you do not implement this attribute, the first memory type defined in the board_spec.xml file becomes the default memory interface. interface See the interface section above for the parameters you must specify for each interface. allocation_type A list that specifies which USM allocator is used to allocate from the global_mem element. Values allowed in this list are host, shared, and device. The following conditions apply: If there are multiple global_mem elements with the same allocation_type attribute, the first allocation_type attribute in the board_spec.xml is assumed to be the one used by the specified allocator. If there is a single global_mem element with multiple allocation_type attributes, this indicates that allocations of the specified types use this global_mem interface. [Legacy support] If you have not specified the allocation_type attribute, it is assumed that all global memory interfaces have the device allocation_type. Example below shows a global_mem element configuration for a kernel system connected to four 4GB DDR4 memory banks. The DDR4 interface is 64 bit operating at 1200MHz. Note that the name of the platform designer system name is board.qsys. As mentioned in description for interleaved_bytes in table above, the Memory Bank Divider configuration ensures that the host interface matches the interleaved_bytes setting (i.e. 512 bits x 64 burst size = 4096 bytes). For information on waitrequest_allowance, refer to section 2.1.4 on interface attribute.
Note: More details on the Memory Bank Divider and the Clock Crossing Bridges is covered in section 3.0
Figure 2-6: Memory Connection Example Block Diagram and Corresponding global_mem Element in board_spec.xml
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#2151-global-memory-configurations","title":"2.1.5.1 Global Memory Configurations","text":"A board can have a single memory bank, multiple memory banks of the same type (e.g. 4 banks of DDR4) or different banks of different types.
The partitioning of memory for oneAPI kernel developers is explained in the Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs. The global memory configuration required by an application kernel must match the configuration in board_spec.xml as the compiler uses this information to generate a suitable architecture for the application. The different memory configurations are
- A single global memory region (possible with same type of memory banks)
- Different global memories (heterogeneous memory)
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#21511-contiguous-global-memory","title":"2.1.5.1.1 Contiguous Global Memory","text":"For boards with multiple memory banks of the same type, designers can configure these as a single contiguous global memory region. This is done by specifying each memory interface within a single global_mem element. Figure 2-6 showed 4 DDR4 memory banks configured as a single global memory region.
With this configuration, FPGA application developers have the option to use contiguous memory region in an interleaved or a non-interleaved fashion. Even with contiguous memory regions, kernel developers can partition data buffers across the banks/memory channels. Please refer to Global Memory Access Optimization section in Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs for more details on these partitioning techniques.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#21512-heterogeneous-memory","title":"2.1.5.1.2 Heterogeneous Memory","text":"For boards with different memory technologies, designers must specify each type of memory that the kernel needs to access as a separate global memory.
Figure 2-7 shows heterogeneous configurations and the global_mem element structure for two different types of memories (QDR, DDR4). The global_mem element in example below also demonstrates use of the default attribute. It is set to \"1\" for the DDR4 memory banks, indicating to the oneAPI compiler that the default global memory for the kernel is DDR4.
Figure 2-7: Heterogeneous Memory Example Block Diagram and Corresponding global_mem Element in board_spec.xml
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#21513-unified-shared-memory","title":"2.1.5.1.3 Unified Shared Memory","text":"For applications that require USM support, the board_spec.xml must specify host and device memories in a heterogeneous manner. The allocation_type must be host for global memory region on the host processor. The allocation_type must be set to device for global memory on the FPGA board. Example below extends the board_spec.xml snippet in figure 2-6 to add a global_mem element for the kernel system to host processor memory interface.
Figure 2-8: global_mem Element Example for Unified Shared Memory(USM)
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#216-host-element","title":"2.1.6 host Element","text":"The host element of the board_spec.xml file provides information on the interface from the host to the kernel. Figure below shows an example of host element.
Figure 2-9: host Element Example
Table 2-8: Attributes for the host Element
Attribute Description kernel_config This attribute informs the oneAPI compiler at what offset the kernel resides, from the perspective of the kernel_cra host on the kernel_interface module.* start: the starting address of the kernel. Normally, this attribute has a value of 0 because the kernel_cra host should not host anything except kernels.* size: keep this parameter at the default value of 0x0100000."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#217-interfaces-element","title":"2.1.7 interfaces Element","text":"The interfaces element of the board_spec.xml file describes the kernel interfaces that connect to application kernels and control kernel behavior. For this element, include one of each interface of types host, irq and streamsource. Refer to the interface section for the parameters you must specify for each interface. In addition to the host, irq, and streamsource interfaces, if your design includes a separate Platform Designer subsystem containing the board logic, the kernel clock and reset interfaces exported from it are also part of the interfaces element. Specify these interfaces with the kernel_clk_reset attribute and its corresponding parameters.
Figure below shows example of interfaces element.
Figure 2-10: interfaces Element Example
Table 2-9: Parameters for the kernel_clk_reset Attribute
Attribute Description clk The Platform Designer name for the kernel clock interface clk2x The Platform Designer name for the 2xkernel clock interface reset The Platform Designer connection for the kernel reset Note: Name the kernel clock and reset interfaces in the Platform Designer connection format (that is, .). For example: board.kernel_clk"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#218-channels-element","title":"2.1.8 channels Element","text":"
The channels element provides channels for streaming data directly between kernel and I/O. Each channel (implemented using Avalon-ST specification) must be connected to the kernel via the interface attribute. The channel interface only supports data, and valid and ready Avalon-ST signals. The I/O channel defaults to 8-bit symbols and big-endian ordering at the interface level.
Figure below shows an example of channels element for a single channel with a width of 64 bits. The chan_id attribute identified helps identify the port in the generated kernel_system. Refer to section 2.1.4 for more information about the interface attribute parameters. Additional interface attributes can be added for additional channels.
Figure 2-11: channels Element Example
For more information about kernel development using channels, refer to I/O Pipes section in Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#22-board_envxml-file","title":"2.2 board_env.xml File","text":"The board_env.xml file is used by the oneAPI toolkit to set up the board installation that enables the compiler to target a specific accelerator platform. The board_env.xml file must be located in the top most level of the oneAPI ASP for each platform. For example, the board_env.xml for oneAPI ASP for OFS reference platforms is located in the oneapi-asp/Platform-Name folder, where Platform-Name is n6001, fseries-dk, iseries-dk for OFS targeting Agilex\u00ae 7 FPGA and d5005 for OFS targeting Stratix\u00ae 10 FPGA.
A sample board_env.xml file is shown below. Table 2-10 explains the elements of this file.
Figure 2-12: board_env.xml File Structure
Table 2-10: Specifications of XML Elements and Attributes in the board_env.xml File
Element Attribute Description board_env * version: The oneAPI compiler version used to create oneAPI ASP* name: The runtime uses this as the name of the FPGA Client Driver(FCD) file name hardware * dir: Name of the subdirectory, within the oneAPI ASP directory, that contains the board variant directories for a platform * default: The default board variant that the compiler targets when a platform has multiple board variants and user does not specify an explicit argument using -Xstarget option platform name: Name of the operating system. A separate platform element must be specified for each supported OS for the oneAPI ASP platform mmdlib A string that specifies the path to the MMD library of your oneAPI ASP. To load multiple libraries, specify them in an ordered, comma-separated list. The host application will load the libraries in the order that they appear in the list> Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if MMD library is located inside linux64/lib folder in oneAPI ASP, the path would be %b/linux64/lib/libintel_opae_mmd.so linkflags A string that specifies the linker flags necessary for linking with the MMD layer available with the board> Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if MMD library is located inside linux64/lib folder in oneAPI ASP, the path would be %b/linux64/lib. linklibs A string that specifies the libraries the oneAPI runtime must link against to use the MMD layer available with the board utilbindir Directory in which the runtime expects to locate board utility executables (i.e. install, uninstall, program, diagnose, flash) > Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if the utilities are located in linux64/libexec folder in oneAPI ASP, the path would be %b/linux64/libexec"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#30-oneapi-asp-hardware","title":"3.0 oneAPI ASP Hardware","text":"The oneAPI compiler generates the kernel system interfaces based on specifications provided by the oneAPI ASP developer in the board_spec.xml file. The kernel system interfaces with the rest of the oneAPI ASP RTL as shown in figure 1-3.
Figure 1-3 shows 5 different paths, summarized below:
- Host to EMIF: Consisting of RTL to handle data transfer between host and on-board memory (e.g. DDR4)
- Host to Kernel: Consisting of RTL to handle control signals & interrupts between host and kernel
- Kernel to EMIF: Consisting of RTL to handle data transfer between kernel and on-board memory
- Kernel to Host memory: Required to support Unified Shared Memory. This requires some additional RTL to handle data transfer between kernel and host memory
- Kernel to HSSI: Consisting of RTL to handle data streaming between kernel and I/O
Please note that the kernel system generated by oneAPI compiler has Avalon\u00ae interfaces. OFS FIM has AXI interfaces. Additional logic blocks from Platform Interface Manager are used to handle protocol conversions. Please refer to section 5.3.1 for more details on PIM. The next few sections cover some of the important IP components required to enable kernel communications with host and board peripherals. More design implementation details are covered in section 5.0.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#31-host-to-external-memory-interfaceemif","title":"3.1 Host to External Memory Interface(EMIF)","text":"The host to EMIF datapath consists of a PCIe Subsytem(SS), EMIF Subsystem located in the FIM and a Direct Memory Access(DMA) engine in the oneAPI ASP.
PCIe Subsystem(SS) has the PCIe IP and additional logic to handle PCIe packet format and routing. FIM handles routing signals received from host to the user design located in a region referred to as Accelerator Functional Unit(AFU) (the Kernel system resides in the AFU).
Note: For more information about the PCIe SS, please refer to Intel FPGA IP Subsystem for PCI Express IP User Guide
The External Memory Interface Subsystem (EMIF SS) consists of EMIF IP and additional logic for handling transfers between AFU and on-board memories.
Note: For more information about the EMIF SS, please refer to Memory Subsystem Intel FPGA IP User Guide
Large buffers of data are usually transferred between host and on-board memory in oneAPI applications. This necessitates a Direct Memory Access(DMA) Engine between host and on-board memory. In oneAPI ASP designs for OFS reference platform, this DMA engine is placed in the AFU region.
As described in section 2.1.5.1, there are different configurations for memories on board. In addition to above, figure 1-3 also shows an additional IP in the host to memory datapath, called Memory Bank Divider. This IP is used for handling one of the most commonly used configurations, i.e. configuring multiple memory banks of same type as a contiguous memory region. In this case, the kernel has a contiguous view of the memory and data can be interleaved across the different memory channels. The host must also have the same view of the memory in order to ensure read and write transactions from correct addresses.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#311-memory-bank-divider","title":"3.1.1 Memory Bank Divider","text":"The Memory Bank Divider is a oneAPI ASP IP component that takes an incoming request from the host interface on the Avalon\u00ae-MM agent port and routes it to the appropriate bank host port. This component must reside on the path between the host and the global memory interfaces. In addition, it must reside outside of the path between the kernel and the global memory interfaces.
The following image shows the IP interfaces. Table 3-2 provides more information on the interface signals.
Figure 3-1: Memory Bank Divider IP
This IP is currently made available as part of the oneapi-asp located in oneapi-asp/common/hardware/common/build/ip folder (memory_bank_divider_hw.tcl). The IP can be instantiated in a top level system by passing values to the parameters as shown in snippet below. Table 3-1 provides more information on the parameters. The oneAPI ASP for OFS reference platforms instantiates this IP in a similar way. Refer to section 5.0 for more information on implementation details for oneAPI ASP for OFS reference platforms.
\n add_instance memory_bank_divider memory_bank_divider <version>\n set_instance_parameter_value memory_bank_divider {NUM_BANKS} {number-of-memory-banks}\n set_instance_parameter_value memory_bank_divider {SEPARATE_RW_PORTS} {true/false-value}\n set_instance_parameter_value memory_bank_divider {PIPELINE_OUTPUTS} {true/false-value}\n set_instance_parameter_value memory_bank_divider {SPLIT_ON_BURSTBOUNDARY} {true/false-value}\n set_instance_parameter_value memory_bank_divider {DATA_WIDTH} {data-width}\n set_instance_parameter_value memory_bank_divider {ADDRESS_WIDTH} {total-addressable-width}\n set_instance_parameter_value memory_bank_divider {BURST_SIZE} {burst-size}\n set_instance_parameter_value memory_bank_divider {MAX_PENDING_READS} {max-pending-reads}\n set_instance_parameter_value memory_bank_divider {ASYNC_RESET} {value}\n set_instance_parameter_value memory_bank_divider {SYNCHRONIZE_RESET} {value}\n Table 3-1: Parameter Settings for the Memory Bank Divider Component
Parameter Name in IP TCL Parameter Description NUM_BANKS Number of banks Number of memory banks for each of the global memory types included in your board system. A single memory bank divider has the following allowed values: 1,2,4,8 for this parameter. SEPARATE_RW_PORTS Separate read/write ports Enable this parameter so that each bank has one port for read operation and one for write operation. PIPELINE_OUTPUTS Add pipeline stage to output Enable this parameter to allow for potential timing improvements. DATA_WIDTH Data Width Width of the data bus to the memory in bits. ADDRESS_WIDTH Address Width (total addressable) Total number of address bits necessary to address all global memory. BURST_SIZE Burst size (maximum) Set to a value equal to interleaved_bytes/(width/8), where interleaved_bytes and width are defined in the interface attribute of the global_mem element in the board_spec.xml file. MAX_PENDING_READS Maximum Pending Reads Maximum number of pending read transfers the component can process without asserting a waitrequest signal. Recommended value is 64 if ASP has two global memory banks or fewer and 128 if ASP has four or more global memory banks. CAUTION: A high Maximum Pending Reads value causes Platform Designer to insert a deep response FIFO buffer, between the component's host and agent, that consumes a lot of device resources. It also increases the achievable bandwidth between host and memory interfaces. SPLIT_ON_BURSTBOUNDARY Split read/write bursts on burst word boundary Enable splitting of read and write bursts on burst word boundary. Enable this parameter if the Number of banks parameter value is greater than 1, and the burst reads and writes that the host controller sends to the agent port crosses burst word boundary. Table 3-2: Signals and Ports for the Memory Bank Divider Component
Signal or Port Description clk The bank divider logic uses this clock input. If the IP of your host and memory interfaces have different clocks, ensure that clk clock rate is not slower than the slowest of the two IP clocks. reset The reset input that connects to the board power-on reset. s The agent port that connects to the host interface controller. kernel_clk The kernel_clk drives this clock input kernel_reset The kernel_reset output from the Kernel Interface IP drives this reset input. acl_asp_snoop Export this Avalon\u00ae Streaming (Avalon\u00ae-ST) source. In the board_spec.xml file, under interfaces, describe only the snoop interface for the default memory (acl_internal_snoop). If you have a heterogeneous memory design, perform these tasks only for the Memory Bank Divider component associated with the default memory. Important: The memory system you build in HW tcl (e.g. ddr_board_hw.tcl in oneapi-asp for OFS reference platforms. Refer to section 5 for more details on ddr_board_hw.tcl) alters the width of acl_asp_snoop. You must update the width of the streamsource interface within the channels element in the board_spec.xml file to match the width of acl_asp_snoop. In the board_spec.xml file, update the width of the snoop interface (acl_internal_snoop) specified with the streamsource kernel interface within the interfaces element. Updating the width ensures that the global_mem interface entries in board_spec.xml match the characteristics of the bankN Avalon\u00ae-MM hosts from corresponding Memory Bank Divider component for the default memory. acl_asp_memorg_host This conduit connects to the acl_asp_memorg_host interface of the Kernel Interface IP.> Note: Signal present if Number of banks > 1. bank1, bank2, ..., bank8 The number of memory hosts available in the Memory Bank Divider depends on the number of memory banks that were included when the unit was instantiated. Connect each bank with each memory interface in the same order as the starting address for the corresponding kernel memory interface specified in the board_spec.xml file. For example, global_mem interface that begins at address 0 must correspond to the memory host in bank1 from the Memory Bank Divider."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#32-host-to-kernel-interface","title":"3.2 Host to Kernel Interface","text":"The host exchanges control signals with kernel with the help of an additional oneAPI ASP IP . The control signals coming from the host are on a different clock domain (PCIe clock) while the kernel runs on different clock frequency . The Kernel Interface IP handles the clock domain crossing for these control signals as well as handles communication with kernel CSR, interrupts and generates the reset for oneAPI kernel. All oneAPI ASP designs must instantiate Kernel Interface IPs to ensure the kernel functions correctly.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#321-kernel-interface","title":"3.2.1 Kernel Interface","text":"The Kernel Interface is an oneAPI ASP component that allows the host interface to access and control the oneAPI kernel.
The following image shows the IP interfaces. Table 3-4 provides more information on the interface signals.
Figure 3-2: Kernel Interface IP
This IP is currently made available as part of the oneapi-asp located in oneapi-asp/common/hardware/common/build/ip folder (kernel_interface_hw.tcl). The IP can be instantiated in a top level system by passing values to the parameters as shown in snippet below. Table 3-3 provides more information on the parameter. The oneAPI ASP for OFS reference platforms instantiates this IP in a similar way. Refer to section 5.0 for more information on implementation details for oneAPI ASP for OFS reference platforms.
\n add_instance kernel_interface kernel_interface <version>\n set_instance_parameter_value kernel_interface {NUM_GLOBAL_MEMS} {value}\n Table 3-3: Parameter Settings for the Kernel Interface Component
Parameter Name in IP TCL Parameter Description NUM_GLOBAL_MEMS Number of global memory systems Number of global memory types in your board design. Table 3-4: Signals and Ports for the Kernel Interface Component
Signal or Port Description clk The clock input used for the host control interface. The clock rate of clk can be slow. reset This reset input resets the control interface. It also triggers the kernel_reset signal, which resets all kernel logic. ctrl Use this agent port to connect to the host interface. This interface is a low-speed interface with which you set kernel arguments and start the kernel's execution. kernel_clk kernel clock drives this clock input. kernel_cra This Avalon\u00ae-MM host interface communicates directly with the kernels generated by the oneAPI compiler. Export the Avalon\u00ae-MM interface to the Kernel Interface and name it in the board_spec.xml file. sw_reset_in When necessary, the host interface resets the kernel via the ctrl interface. If the board design requires a kernel reset, it can do so via this reset input. Otherwise, connect the interface to a global power-on reset. kernel_reset Use this reset output to reset the kernel and any other hardware that communicates with the kernel. Warning: This reset occurs between the MMD open and close calls. Therefore, it must not reset anything necessary for the operation of your MMD. sw_reset_export This reset output is the same as kernel_reset, but it is synchronized to the clk interface. Use this output to reset logic that is not in the kernel_clk clock domain but should be reset whenever the kernel resets. acl_asp_memorg_host The memory interfaces use these signals. Based on the number of global memory systems you specify in the Kernel Interface component parameter editor, the Quartus\u00ae Prime Pro Edition Software creates the corresponding number of copies of this signal, each with a different hexadecimal suffix. Connect each signal to the Memory Bank Divider component associated with each global memory system (for example, DDR). Then, list the hexadecimal suffix in the config_addr attribute of the global_mem element in the board_spec.xml file. kernel_irq_from_kernel An interrupt input from the kernel. This signal is exported and named in the board_spec.xml file. kernel_irq_to_host An interrupt output from the kernel. This signal connects to the host interface."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#33-kernel-to-external-memory-interface","title":"3.3 Kernel to External Memory Interface","text":"The kernel system masters the interface from kernel to external memory. oneAPI compiler generates kernel system memory interface logic (e.g. Load-Store Unit) according to the global memory configuration and interface specifications in board_spec.xml file. The kernel system operates at kernel clock, hence, oneAPI ASP developers must handle clock domain crossing from kernel to EMIF clock domain.
For implementation details for all datapaths discussed above, please refer to section 5.3.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#40-oneapi-asp-software","title":"4.0 oneAPI ASP Software","text":"The software components of oneAPI ASP consist of the Memory Mapped Device(MMD) layer and the board utility routine required by runtime.
Section 4.1 introduces MMD layer and section 4.2 explains board utilities.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#41-memory-mapped-devicemmd-layer","title":"4.1 Memory Mapped Device(MMD) Layer","text":"The oneAPI ASP Memory Mapped Device (MMD) layer sits in between the oneAPI runtime and OPAE SDK and provides a set of API for device communication and control. The runtime calls into the MMD API for various operations like opening a handle to the device, allocating memory etc.
Note: For more information about the FPGA runtime, please refer to FPGA Runtime documentation here.
A header file, called aocl_mmd.h, has the list of MMD API calls that must be implemented by oneAPI ASPs. From the perspective of the caller, below is typical MMD API lifecycle:
- Open device to provide handle for further operations
- Set interrupt and status handlers
- Program device with kernel bitstream
- Allocate memory if required
- Perform Read, Write operations (DMA or MMIO)
- Free memory if allocation done in step 4
- Close device. No further operations permitted until subsequent open device call
Table below summarizes all APIs listed in aocl_mmd.h.
Table 4-1: Summary of MMD API from aocl_mmd.h
API Purpose aocl_mmd_get_offline_info Obtain offline information about the board. This function is offline because it is device-independent and does not require a handle from the aocl_mmd_open() call aocl_mmd_get_info Obtain information about the board specified in the requested_info_id argument (refer to section 4.1.2 for more information) aocl_mmd_open Open and initialize the specified device aocl_mmd_close Close an opened device via its handle aocl_mmd_set_interrupt_handler Set the interrupt handler for the opened device aocl_mmd_set_device_interrupt_handler Sets the device interrupt handler for opened device, interrupt handler is called to notify runtime of any exceptions aocl_mmd_set_status_handler Set the operation status handler for the opened device aocl_mmd_yield The aocl_mmd_yield function is called when the host interface is idle. The host interface might be idle because it is waiting for the device to process certain events aocl_mmd_read Read operation on a single interface aocl_mmd_write Write operation on a single interface aocl_mmd_copy Copy operation on a single interface aocl_mmd_hostchannel_create Creates a channel interface aocl_mmd_hostchannel_destroy Destroys channel interface aocl_mmd_hostchannel_get_buffer Provides host with pointer used to read/write from channel interface aocl_mmd_hostchannel_ack_buffer Acknowledges read/write from channel aocl_mmd_program Reprogram operation for the specified device aocl_mmd_host_alloc Provide memory that is allocated on the host. Host allocations are accessible by the host and one or more devices aocl_mmd_free Free memory that has been allocated by MMD aocl_mmd_device_alloc Allocate memory that is owned by the device aocl_mmd_shared_alloc Allocate shared memory between the host and the FPGA aocl_mmd_shared_migrate Handle migration of non-concurrent shared allocations any time the accessor of the allocation changes Sections below cover more details for each API (expected arguments, return values). Section 5.4 discusses more about the implementation of the MMD layer APIs in oneAPI ASPs for OFS reference platforms.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#411-aocl_mmd_get_offline_info","title":"4.1.1 aocl_mmd_get_offline_info","text":"The aocl_mmd_get_offline_info function obtains offline information about the board specified in the requested_info_id argument. This function is offline because it is device-independent and does not require a handle from the aocl_mmd_open() call.
Syntax
\n\n int aocl_mmd_get_offline_info (\n aocl_mmd_offline_info_t requested_info_id,\n size_t param_value_size,\n void* param_value,\n size_t* param_size_ret )\n\n
Function Arguments
requested_info_id: An enum value of type aocl_mmd_offline_info_t that indicates the offline device information returning to the caller.
Table 4-2: Possible Enum Values for the requested_info_id Argument
Name Description Type AOCL_MMD_VERSION Version of MMD layer char* AOCL_MMD_NUM_BOARDS Number of candidate boards int AOCL_MMD_BOARD_NAMES Names of available boards > Note: Separate each board name by a semicolon (;) delimiter. char* AOCL_MMD_VENDOR_NAME Name of board vendor char* AOCL_MMD_VENDOR_ID An integer board vendor ID int AOCL_MMD_USES_YIELD A value of 0 instructs the runtime to suspend user's processes. The runtime resumes these processes after it receives an event update (for example, an interrupt) from the MMD layer.A value of 1 instructs the runtime to continuously call the aocl_mmd_yield function while it waits for events to complete.CAUTION: Setting AOCL_MMD_USES_YIELD to 1 might cause high CPU utilization if the aocl_mmd_yield function does not suspend the current thread. int -
param_value_size: Size of the param_value field in bytes. This size_t value should match the size of the expected return type that the enum definition indicates. For example, if AOCL_MMD_NUM_BOARDS returns a value of type int, set the param_value_size to sizeof (int). You should see the same number of bytes returned in the param_size_ret argument.
-
param_value: A void* pointer to the variable that receives the returned information.
-
param_size_ret: A pointer argument of type size_t* that receives the number of bytes of returned data.
Return Value
A negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#412-aocl_mmd_get_info","title":"4.1.2 aocl_mmd_get_info","text":"The aocl_mmd_get_info function obtains information about the board specified in the requested_info_id argument.
Syntax
\n\n int aocl_mmd_get_info (\n int handle,\n aocl_mmd_info_t requested_info_id,\n size_t param_value_size,\n void* param_value,\n size_t* param_size_ret )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
requested_info_id: An enum value of type aocl_mmd_info_t that indicates the device information returning to the caller.
Table 4-3: Possible Enum Values for the requested_info_id Argument
Name Description Type AOCL_MMD_NUM_KERNEL_INTERFACES Number of kernel interfaces int AOCL_MMD_KERNEL_INTERFACES Kernel interfaces int* AOCL_MMD_PLL_INTERFACES Kernel clock handles int* AOCL_MMD_MEMORY_INTERFACE Global memory handle int AOCL_MMD_TERMPERATURE Temperature measurement float AOCL_MMD_PCIE_INFO PCIe\u00ae information char* AOCL_MMD_BOARD_NAME Board name char* AOCL_MMD_BOARD_UNIQUE_ID Unique board ID char* AOCL_MMD_CONCURRENT_READS Number of parallel readsA value of 1 indicates serial reads. int AOCL_MMD_CONCURRENT_WRITES Number of parallel writesA value of 1 indicates serial writes. int AOCL_MMD_CONCURRENT_READS_OR_WRITES Total number of concurrent read and write operations int AOCL_MMD_MIN_HOST_MEMORY_ALIGNMENT Minimum alignment that the oneAPI ASP supports for host allocations size_t AOCL_MMD_HOST_MEM_CAPABILITIES Capabilities of aocl_mmd_host_alloc() function unsigned int AOCL_MMD_SHARED_MEM_CAPABILITIES Capabilities of aocl_mmd_shared_alloc() function unsigned int AOCL_MMD_DEVICE_MEM_CAPABILITIES Capabilities of aocl_mmd_device_alloc() function unsigned int AOCL_MMD_HOST_MEM_CONCURRENT_GRANULARITY Granularity of concurrent host accesses size_t AOCL_MMD_SHARED_MEM_CONCURRENT_GRANULARITY Granularity of concurrent shared accesses size_t AOCL_MMD_DEVICE_MEM_CONCURRENT_GRANULARITY Granularity of concurrent device accesses size_t -
param_value_size: Size of the param_value field in bytes. This size_t value should match the size of the expected return type that the enum definition indicates. For example, if AOCL_MMD_TEMPERATURE returns a value of type float, set the param_value_size to sizeof (float). You should see the same number of bytes returned in the param_size_ret argument.
-
param_value: A void* pointer to the variable that receives the returned information.
-
param_size_ret: A pointer argument of type size_t* that receives the number of bytes of returned data.
Capability Values
Table 4-4: Capability Values for aocl_mmd_get_info Function
Value Description AOCL_MMD_MEM_CAPABILITY_SUPPORTED If you do not set this value, allocation function is not supported even if other capabilities are set. AOCL_MMD_MEM_CAPABILITY_ATOMIC Supports atomic access to the memory by either the host or the device. AOCL_MMD_MEM_CAPABILITY_CONCURRENT Supports concurrent access to the memory either by the host or the device if the accesses are not on the same block. Block granularity is defined by AOCL_MMD_*_MEM_CONCURRENT_GRANULARITY. Blocks are aligned to this granularity. AOCL_MMD_MEM_CAPABILITY_P2P Memory can be accessed by multiple devices at the same time. Return Value
A negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#413-aocl_mmd_open","title":"4.1.3 aocl_mmd_open","text":"The aocl_mmd_open function opens and initializes the specified device.
Syntax
\n\n int aocl_mmd_open (const char *name)\n\n
Function Arguments
name: The function opens the board with a name that matches this const char* string. The name typically matches the one specified by the AOCL_MMD_BOARD_NAMES offline information.
The runtime first queries the AOCL_MMD_BOARD_NAMES offline information to identify the boards that it might be able to open. Then it attempts to open all possible devices by calling aocl_mmd_open and using each of the board names as argument.
Note: The name must be a C-style NULL-terminated ASCII string.
Return Value
If aocl_mmd_open() executes successfully, the return value is a positive integer that acts as a handle to the board.
If aocl_mmd_open() fails to execute, a negative return value indicates an error. In the event of an error, the runtime proceeds to open other known devices. Therefore, it is imperative that the MMD layer does not exit the application if an open call fails.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#414-aocl_mmd_close","title":"4.1.4 aocl_mmd_close","text":"The aocl_mmd_close function closes an opened device via its handle.
Syntax
\n\n int aocl_mmd_close (int handle)\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
Return Value
If the aocl_mmd_close() executes successfully, the return value is 0.
If aocl_mmd_close() fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#415-aocl_mmd_set_interrupt_handler","title":"4.1.5 aocl_mmd_set_interrupt_handler","text":"The aocl_mmd_set_interrupt_handler function sets the interrupt handler for the opened device. When the device internals identify an asynchronous kernel event (for example, a kernel completion), the interrupt handler is called to notify the runtime of the event.
Note: Ignore the interrupts from the kernel until this handler is set.
Syntax
\n\n int aocl_mmd_set_interrupt_handler (\n int handle, \n aocl_mmd_interrupt_handler_fn fn,\n void* user_data )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
fn: The callback function to invoke when a kernel interrupt occurs. The fn argument is of type aocl_mmd_interrupt_handler_fn, which is defined as follows:
\n typedef void (*aocl_mmd_interrupt_handler_fn)( int handle, void* user_data );\n
user_data: The void* type user-provided data that passes to fn when it is called.
Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#416-aocl_mmd_set_device_interrupt_handler","title":"4.1.6 aocl_mmd_set_device_interrupt_handler","text":"The aocl_mmd_set_device_interrupt_handler function sets the device interrupt handler for the opened device. When the device internals identify an asynchronous exception event (for example, a bit correction event), the device interrupt handler is called to notify the runtime of the event.
Note: Ignore the interrupts from the device until this handler is set.
Syntax
\n\n int aocl_mmd_set_device_interrupt_handler (\n int handle, \n aocl_mmd_device_interrupt_handler_fn fn,\n void* user_data )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
fn: The callback function to invoke when a kernel interrupt occurs. The fn argument is of type aocl_mmd_device_interrupt_handler_fn, which is defined as follows:
\n typedef void (*aocl_mmd_device_interrupt_handler_fn)( int handle, aocl_mmd_interrupt_info* data_in, void* user_data );\n
aocl_mmd_interrupt_info is defined as:
\n\n typedef struct {\n unsigned long long int exception_type;\n void *user_private_info;\n size_t user_cb;\n } aocl_mmd_interrupt_info;\n\n Where:
exception_type acts as a bitfield that contains exactly one bit, corresponding to an exception number.user_private_info and user_cb represent pointers to binary data that the OpenCL implementation return. These pointers log additional information that is helpful for debugging the error.
user_data: The void* type user-provided data that passes to fn when it is called.
Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#417-aocl_mmd_set_status_handler","title":"4.1.7 aocl_mmd_set_status_handler","text":"The aocl_mmd_set_status_handler function sets the operation status handler for the opened device. The operation status handler is called under the following circumstances:
- When the operation completes successfully and status is 0.
- When the operation completes with errors and status is a negative value.
Syntax
\n\n int aocl_mmd_set_status_handler (\n int handle,\n aocl_mmd_status_handler_fn fn,\n void* user_data )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
fn: The callback function to invoke when a status update occurs. The fn argument is of type aocl_mmd_status_handler_fn, which is defined as follows:
\n typedef void (*aocl_mmd_status_handler_fn)( int handle, void* user_data, aocl_mmd_op_t op, int status );\n
user_data: The void* type user-provided data that passes to fn when it is called.
Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#418-aocl_mmd_yield","title":"4.1.8 aocl_mmd_yield","text":"The aocl_mmd_yield function is called when the host interface is idle. The host interface might be idle because it is waiting for the device to process certain events.
Syntax
\n\n int aocl_mmd_yield (int handle)\n\n
Function Arguments
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
Return Value
A nonzero return value indicates that the yield function performed work necessary for proper device functioning such as processing direct memory access (DMA) transactions.
A return value of 0 indicates that the yield function did not perform work necessary for proper device functioning.
Note: The yield function might be called continuously if it reports that it has necessary work to perform.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#419-aocl_mmd_read","title":"4.1.9 aocl_mmd_read","text":"The aocl_mmd_read function is the read operation on a single interface.
Syntax
\n\n int aocl_mmd_read (\n int handle,\n aocl_mmd_op_t op,\n size_t len,\n void* dst,\n int mmd_interface, size_t offset )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.
Note: aocl_mmd_op_t is defined as follows:
\n typedef void* aocl_mmd_op_t;\n
-
len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
-
dst: The host buffer, of type void*, to which data is written.
-
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
-
offset: The size_t byte offset within the interface at which the data transfer begins.
Return Value
If the read operation is successful, the return value is 0.
If the read operation fails, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4110-aocl_mmd_write","title":"4.1.10 aocl_mmd_write","text":"The aocl_mmd_write function is the write operation on a single interface.
Syntax
\n\n int aocl_mmd_write (\n int handle,\n aocl_mmd_op_t op,\n size_t len,\n const void* src,\n int mmd_interface, size_t offset )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.
Note: aocl_mmd_op_t is defined as follows:
\n typedef void* aocl_mmd_op_t;\n
-
len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
-
src: The host buffer, of type const void*, from which data is read.
-
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
-
offset: The size_t byte offset within the interface at which the data transfer begins.
Return Value
If the write operation is successful, the return value is 0.
If the write operation fails, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4111-aocl_mmd_copy","title":"4.1.11 aocl_mmd_copy","text":"The aocl_mmd_copy function is the copy operation on a single interface.
Syntax
\n\n int aocl_mmd_copy (\n int handle,\n aocl_mmd_op_t op,\n size_t len,\n int mmd_interface, size_t src_offset, size_t dst_offset )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.
Note: aocl_mmd_op_t is defined as follows:
\n typedef void* aocl_mmd_op_t;\n
-
len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
-
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
-
src_offset: The size_t byte offset within the source interface at which the data transfer begins.
-
dst_offset: The size_t byte offset within the destination interface at which the data transfer begins
Return Value
If the copy operation is successful, the return value is 0.
If the copy operation fails, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4112-aocl_mmd_hostchannel_create","title":"4.1.12 aocl_mmd_hostchannel_create","text":"The aocl_mmd_hostchannel_create function creates a channel interface.
Syntax
\n\n int aocl_mmd_hostchannel_create (\n int handle,\n char *channel_name,\n size_t queue_depth,\n int direction )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
channel_name: Name of the channel to be initialized. The channel name is same as that used in the board_spec.xml file.
-
queue_depth: The size of pinned internal buffer in bytes. Pointer to the internal buffer is provided when the user calls the aocl_mmd_hostchannel_get_buffer() function.
-
direction: The direction of the channel.
Return Value
If the function executes successfully, the return value is positive and is handle to the channel.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4113-aocl_mmd_hostchannel_destroy","title":"4.1.13 aocl_mmd_hostchannel_destroy","text":"The aocl_mmd_hostchannel_destroy function destroys the channel interface.
Syntax
\n\n int aocl_mmd_hostchannel_destroy (\n int handle,\n int channel )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
Return Value
If the function executes successfully, the return value is 0.
If the function fails to execute, a negative return value indicates an error.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4114-aocl_mmd_hostchannel_get_buffer","title":"4.1.14 aocl_mmd_hostchannel_get_buffer","text":"The aocl_mmd_hostchannel_get_buffer function provides a host with a pointer to the buffer they can access to write or read from the channel interface, along with the space or data available in the buffer, in bytes.
Syntax
\n\n void *aocl_mmd_hostchannel_get_buffer (\n int handle,\n int channel,\n size_t *buffer_size,\n int *status )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
-
buffer_size: A pointer to size_t that the function writes available buffer space or size to.
-
status: A pointer to int that the function writes result of the call to.
Return Value
If the function executes successfully, int pointed to by the status pointer is 0. Returned void* may still be NULL, in which case size_t pointed by the buffer_size is 0.
If the function fails to execute, int pointed by the status pointer is a negative value.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4115-aocl_mmd_hostchannel_ack_buffer","title":"4.1.15 aocl_mmd_hostchannel_ack_buffer","text":"You can acknowledge write or read from the channel by calling aocl_mmd_hostchannel_ack_buffer.
Syntax
\n\n size_t aocl_mmd_hostchannel_ack_buffer (\n int handle,\n int channel,\n size_t send_size,\n int *status )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
-
send_size: The size in bytes that the user is acknowledging.
-
status: A pointer to int that the function writes result of the call to.
Return Value
If the function executes successfully, int pointed to by status pointer is 0. Also, there is no guarantee that the user's send_size is the actual size that gets acknowledged. The returned size_t is the amount of bytes that was actually acknowledged.
If the function fails to execute, int pointed by status pointer is a negative value.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4116-aocl_mmd_program","title":"4.1.16 aocl_mmd_program","text":"The aocl_mmd_program function is the program operation for the specified device. The host must guarantee that no other operations are executing on the device during the program operation.
During aocl_mmd_program execution, the kernels are idle and no read, write, or copy operation can occur.
Disable interrupts and program the FPGA with the data from user_data, which has a size specified by the size argument. The host then calls aocl_mmd_set_status_handler and aocl_mmd_set_interrupt_handler again, which enable the interrupts. If events such as interrupts occur during aocl_mmd_program execution, race conditions or data corruption might occur.
Syntax
\n\n int aocl_mmd_program (\n int handle,\n void * user_data,\n size_t size,\n aocl_mmd_program_mode_t program_mode )\n\n
Function Arguments
-
handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
-
user_data: The void* type binary contents of the fpga.bin file that is created during kernel compilation.
-
size: The size of user_data in bytes. The size argument is of size_t.
-
program_mode: The bit-field that specifies the mode of device programming.
Table 4-5: Possible Values for the program_mode Argument
program_mode Argument Value Description AOCL_MMD_PROGRAM_PRESERVE_GLOBAL_MEMORY This flag specifies that during programming the global memory on the devices are preserved. Return Value
If aocl_mmd_program executes successfully, the return value is the pointer value that the host uses to access shared memory.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4117-aocl_mmd_host_alloc","title":"4.1.17 aocl_mmd_host_alloc","text":"Host allocations provide memory that is allocated on the host. This memory must be deallocated with the aocl_mmd_free function. Host allocations are accessible by the host and one or more devices. The same pointer to a host allocation may be used on the host and all supported devices. They have address equivalence.
Syntax
Once the device has signaled completion through the aocl_mmd_interrupt_handler_fn function, the host can assume it has access to the latest contents of the memory, allocated by the aocl_mmd_host_alloc function call.
\n\n void* aocl_mmd_host_alloc (\n int* handles,\n size_t num_devices,\n size_t size,\n size_t alignment,\n aocl_mmd_mem_properties_t *properties,\n int* error )\n\n
Function Arguments
-
handles: Handles for devices that needs access to this memory.
-
num_devices: Number of devices in the handles.
-
size: The size of the memory region.
-
alignment: The alignment (in bytes) of the allocation.
-
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
-
error: The error code defined by AOCL_MMD_ERROR*:
- AOCL_MMD_ERROR_SUCCESS: No error occurred.
- AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
- AOCL_MMD_ERROR_OUT_OF_MEMORY: Ran out of memory.
- AOCL_MMD_ERROR_UNSUPPORTED_ALIGNMENT: The device does not support the provided alignment.
- AOCL_MMD_ERROR_UNSUPPORTED_PROPERTY: The device does not support the provided property.
Return Value
If the aocl_mmd_host_alloc function executes successfully, the return value is a valid pointer value. Otherwise, the return value is NULL.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4118-aocl_mmd_free","title":"4.1.18 aocl_mmd_free","text":"Releases memory that was allocated by MMD.
Syntax
\n\n int aocl_mmd_free (void* mem)\n\n
Function Arguments
mem: The pointer to the memory region. Must be a pointer that is allocated by the MMD.
Return Value
Returns one of the following error code:
- AOCL_MMD_ERROR_SUCCESS: No error occurred
- AOCL_MMD_ERROR_INVALID_POINTER: Invalid pointer provided
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4119-aocl_mmd_device_alloc","title":"4.1.19 aocl_mmd_device_alloc","text":"Allocate memory that is owned by the device. This pointer can only be accessed by the kernel. It cannot be accessed by the host. The host is able to manipulate the pointer (for example, increment it) and not just access the underlying data. This memory must be deallocated by the aocl_mmd_free() function.
Syntax
\n\n void * aocl_mmd_device_alloc (\n int handle, \n size_t size, \n size_t alignment, \n aocl_mmd_mem_properties_t *properties, \n int* error )\n\n
Function Arguments
-
handle: Device that has access to this memory.
-
size: The size of the memory region.
-
alignment: The alignment (in bytes) of the memory region.
-
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
Return Value
Returns one of the following error code:
- AOCL_MMD_ERROR_SUCCESS: No error occurred
- AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
- AOCL_MMD_ERROR_OUT_OF_MEMORY: Ran out of memory.
- AOCL_MMD_ERROR_UNSUPPORTED_ALIGNMENT: The device does not support the provided alignment.
- AOCL_MMD_ERROR_UNSUPPORTED_PROPERTY: The device does not support the provided property.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4120-aocl_mmd_shared_alloc","title":"4.1.20 aocl_mmd_shared_alloc","text":"Shared allocations can migrate between the host and one or more associated device. The same pointer to a shared allocation can be used on the host and the supported device. They have address equivalence.
Syntax
\n\n void * aocl_mmd_shared_alloc (\n int handle, \n size_t size, \n size_t alignment, \n aocl_mmd_mem_properties_t* properties, \n int* error )\n\n
Function Arguments
-
handle: Device that needs access to this memory.
-
size: The size of the memory region.
-
alignment: The alignment (in bytes) of the allocation.
-
properties: Specifies additional information about the allocated memory described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
-
error: The error code defined by AOCL_MMD_ERROR*.
- AOCL_MMD_ERROR_SUCCESS: No error occurred.
- AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
- AOCL_MMD_ERROR_OUT_OF_MEMORY: Ran out of memory.
- AOCL_MMD_ERROR_UNSUPPORTED_ALIGNMENT: The device does not support the provided alignment.
- AOCL_MMD_ERROR_UNSUPPORTED_PROPERTY: The device does not support the provided property.
Return Value
If the aocl_mmd_shared_alloc function executes successfully, the return value is a valid pointer value. Otherwise, the return value is NULL.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#4121-aocl_mmd_shared_migrate","title":"4.1.21 aocl_mmd_shared_migrate","text":"A call to the aocl_mmd_shared_migrate() function must be made for non-concurrent shared allocations any time the accessor of the allocation changes. For example, the aocl_mmd_shared_migrate() function should be called indicating that the allocation should be migrated to the device before a kernel accessing the allocation is launched on the device. Similarly, the aocl_mmd_shared_migrate() function should be called indicating that the allocation is migrated to the host before the host accesses the memory after kernel completion. For concurrent allocations, this call may be used as a performance hint, but it is not strictly required for functionality.
Syntax
\n\n int aocl_mmd_shared_migrate (\n int handle, \n void* shared_ptr, \n size_t size, \n aocl_mmd_migrate_t destination )\n\n
Function Arguments
-
handle: Device that has access to this memory.
-
shared_ptr: Pointer allocated by the aocl_mmd_shared_alloc() function.
-
size: Size (in bytes) of the migration. Must be a multiple of a page boundary that the oneAPI ASP supports.
-
destination: The destination of the migration.
Return Value
Returns one of the following error code:
- AOCL_MMD_ERROR_SUCCESS: No error occurred.
- AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
- AOCL_MMD_ERROR_INVALID_MIGRATION_SIZE: The migration size is not supported by the device.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#42-board-utilities","title":"4.2 Board Utilities","text":"oneAPI runtime provides a set of options for the aocl utility.
Note: aocl is an utility available in the oneAPI runtime environment, please use aocl help command for more information on this.
Table 4-6 shows the subcommands that aocl utility provides for FPGA platforms.
Table 4-6: aocl Board Utilities
Subcommand Description Executable Call install Install board into the host system. This installs the FPGA Client Driver (FCD) for your FPGA platform. FCD allows runtime to find and load the FPGA platform libraries at runtime aocl install path-to-FPGA-platform-oneapi-asp uninstall Uninstall board from the host system. Removes FCD. aocl uninstall path-to-FPGA-platform-oneapi-asp initialize Configure a default FPGA image onto the board. For more information about initialization refer to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs Two methods are available to initalize the board: 1. aocl initialize device-name board-variant 2. aocl initialize device-name oneAPI fat binary > Note: The second option requires oneAPI Base Toolkit (Base Kit) version 2024.0 & above as well as 2023.3 OFS Release for oneAPI Accelerator Support Package and above program Configure a new FPGA image onto the board aocl program device-name aocx file diagnose Runs ICD and FCD diagnostics followed by querying devices in installed platforms. If a device-name is specified in the call, it run board vendor's test program for the FPGA platform * aocl diagnose : This queries the devices in FPGA platform and supplies a list of valid strings assigned to the list of devices * aocl diagnose device-name : This runs full diagnostic test on the FPGA platform The runtime expects the routine for each of this utilities to be defined in the oneAPI ASP. It looks for the routine executables in the location specified by the utilbinder element in the board_env.xml file.
install
oneAPI runtime uses the information in board_env.xml file to create a FCD file. The FCD file name matches name attribute of board_env element and the FCD contents are the platform libraries specified in mmdlib element. Refer to section 2.2 for more information about board_env.xml file. The runtime adds the installed platform to the list of installed packages(file used by runtime to track installed platforms) and then invokes the install routine from oneAPI ASP.
uninstall
oneAPI runtime removes the FCD file and removes the platform from list of installed packages. It then invokes the uninstall routine for oneAPI ASP.
initialize
oneAPI runtime invokes initialize routine provided by oneAPI ASPs for installed platforms.
program
oneAPI runtime loads the programming file on the FPGA by invoking program routine provided by the oneAPI ASPs for the installed platform.
diagnose
oneAPI runtime runs ICD and FCD diagnostics to check the ICD and FCD files installed on the system. It then queries for available boards in the installed platform and lists boards matching every installed platform. If a device-name is specified in the call, runtime invokes the diagnostic routine provided in oneAPI ASP.
For more information about the implementation of these routines in oneAPI ASPs for OFS reference platforms, please refer to section 5.5.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#50-oneapi-asp-implementation-details","title":"5.0 oneapi-asp Implementation Details","text":"oneapi-asp in the OFS has four reference platform releases, one is based on Stratix\u00ae 10 FPGA and the other three are based on Agilex\u00ae 7 FPGA. This chapter aims to explain the architecture and current implementation details of oneapi-asp for these platforms. The oneapi-asp repository is located here.
The next section explains the oneapi-asp directory structure, followed by sections on hardware and MMD layers.
In addition to the information covered in the sections 5.1 to 5.5, there is a new functionality added to the oneapi-asp repository called the oneAPI ASP Editor. The editor enables easy parameterization and generation of the oneAPI ASP using IP Parameter Editor GUI in Quartus Prime.
If you want to customize oneAPI ASP implementation, you can use one of the two approaches:
-
Follow the design implementation in section 5.1 to 5.5 as a reference to manually make updates to your oneAPI ASP design or
-
Use the oneAPI ASP Editor to parameterize your ASP design and let the editor handle ASP directory & design files creation, information about the ASP editor is covered in Appendix B.
Using approach (1) allows fine grained control over design, allowing you to tweak all parameters, interfaces, code blocks in the ASP. The ASP editor (approach 2) abrstracts away some of this details, allowing easy parameter and interface settings, hence reducing design time. There are also limits on some parameters in the editor. Please refer to Appendix B for more information about the parameters in the editor.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#51-oneapi-asp-directory-structure","title":"5.1 oneapi-asp Directory Structure","text":"As described in section 2.0, oneAPI compiler & runtime use the board_env.xml and board_spec.xml files to get information about the FPGA platform. The compiler expects the board_env.xml file to be at the topmost level in the platform directory. The board_env.xml file describes the location of the hardware files & platform libraries.
\n\n <?xml version=\"1.0\"?>\n <board_env version=\"Compiler-version\" name=\"ofs_Platform-name/FCD-name\">\n <hardware dir=\"hardware\" default=\"Board-Variant-1\"></hardware>\n <platform name=\"linux64\">\n <mmdlib>libopae-c.so,%b/linux64/lib/libMPF.so,%b/linux64/lib/libintel_opae_mmd.so</mmdlib>\n <linkflags>-L%a/linux64/lib -L%b/linux64/lib</linkflags>\n <linklibs>-lintel_opae_mmd -lrt -lMP</linklibs>\n <utilbindir>%b/linux64/libexec</utilbindir>\n </platform>\n\n </board_env>\n\n
Snippet above shows a sample board_env.xml file, the corresponding oneAPI ASP directory structure must match the following format. Table 5-1 provides details on each folder.
In addition to below folders, oneAPI ASP for OFS reference platforms have another folder called commmon (oneapi-asp/common) containing the hardware files common across all reference platforms (oneapi-asp/common/hardware) and source code for the software layer, i.e. MMD & board utilities ( oneapi-asp/common/source). This is because the software layer is compatible with both Stratix\u00ae 10 FPGA and Agilex\u00ae 7 FPGA PCIe attach reference platform ASPs.
\n oneapi-asp/Platform-Name/\n |--hardware/\n |--|--Board-Variant-1/\n |--|--Board-Variant-2/\n |--linux64/\n |--board_env.xml\n
Note: In addition to above folders, oneAPI ASPs for OFS reference platforms have additional directories called scripts (in oneapi-asp/common and oneapi-asp/Platform-Name) which contains helper scripts for platform generation and oneapi-asp/common/bringup folder which contains a sample for board bring up. Please refer to the README for each reference platform in the oneASP-asp repository for more information on these additional folders. * README for oneapi-asp targeting Intel\u00ae FPGA PAC D5005 reference platform: README * README for oneapi-asp targeting Intel\u00ae FPGA SmartNIC N6001-PL reference platform: README * README for oneapi-asp targeting Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):README * README for oneapi-asp targeting Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile): README
The Platform-Name is used for identifying the platform and can be alphanumeric value decided by the platform developer. For example, Platform-Name d5005 is used for oneapi-asp for Stratix\u00ae 10 FPGA as the reference platform is Intel\u00ae FPGA PAC D5005.
Table 5-1: Directory Contents
Files/Folder (path relative to oneapi-asp/Platform-Name/) Description hardware Contains hardware files (RTL, platform designer files, SDCs, compilation scripts, floorplan settings) and the board_spec.xml files for each board variants. See table 5-2 for more details ../common/source Source code for MMD layer as well as oneapi-asp board utilities linux64 Location for FPGA platform libraries and executables for oneapi-asp board utilities > Note: The linux64 folder does not exist in the Platform-Name directory, it is added after the oneapi-asp build flow is complete board_env.xml Contains platform installation information. Please refer to section 2.2 for more details on board_env.xml elements Tables 5-2 to 5-4 give more details on each of these folders for oneAPI ASPs for OFS reference platforms (before oneapi-asp build flow unless noted otherwise, for more information about oneapi-asp build flow refer to section 5.2).
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#511-hardware-folder","title":"5.1.1 hardware Folder","text":"\noneapi-asp/\n|--common\n|--|--hardware/\n|--Platform-Name\n|--|--hardware/\n|--|--|--common/\n|--|--|--|--build\n|--|--|--board-variant-1/\n|--|--|--|--build\n|--|--|--|--board_spec.xml\n|--|--|--|--part-number_dm.xml (Please see note for this file in table 5-2)\n
Table 5-2: hardware Folder Contents
Files/Folder (path relative to oneapi-asp/) Description common/hardware Contains hardware design files common to all reference platforms including Quartus\u00ae software Settings Files (*.qsf), IP files (*.v, *_hw.tcl), common RTL code (*.v, *.sv) as well as scripts to control compile flow Platform-Name/hardware/common/build Contains platform specific hardware design files (common to each board variant for a platform) including *.sv, *.sdc, Quartus\u00ae software ini settings file as well as scripts to control compile flow Platform-Name/hardware/board-variant-1/build Contains board variant specific hardware design files like variant specific Verilog header file (*.vh), oneapi_afu.json, *.tcl Platform-Name/hardware/board-variant-1/board_spec.xml Defines compile flow, global memory, kernel interfaces. Please see section 2.1 for more information about board_spec.xml file Platform-Name/hardware/board-variant-1/part-number_dm.xml Device Model file for Altera\u00ae FPGA part on the target platform. The name must be of the format part-number_dm.xml. This file has the total resources available on the device. > Note: The device model files provided as part of the oneAPI Base Toolkit (Base Kit) installation are located in $INTELFPGAOCLSDKROOT/share/models/dm, where INTELFPGAOCLSDKROOT is set by the setvars.sh environment setup script provided by oneAPI toolkit. If the device model file for your part number is not included in $INTELFPGAOCLSDKROOT/share/models/dm, it must be created and placed in the same folder as board_spec.xml. In oneapi-asp for reference platforms, this file only exists for platforms whose target device model files are not provided by oneAPI Base Toolkit (Base Kit) installation."},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#512-commonsource-folder","title":"5.1.2 common/source Folder","text":"\noneapi-asp/common/source/\n|--cmake/\n|--|--modules\n|--extra/\n|--|--intel-fpga-bbb\n|--host\n|--include\n|--util\n|--CMakeLists.txt\n
Table 5-3: oneapi-asp/common/source Folder Contents
Files/Folder Description cmake/modules Contains Find***.cmake files to find packages required for building MMD library extra/intel-fpga-bbb oneAPI ASP for OFS reference platforms uses a library provided as part of Intel\u00ae FPGA Basic Building Blocks (BBB) repository. The oneapi-asp build scripts clone this repository in extra directory by default host Source code for MMD API include Contains MMD header files util Contains source code for oneapi-asp reference platform routines for diagnose and program utilities CMakeLists.txt Top-level CMakeLists.txt file for building MMD and other libraries required by oneapi-asp as well as the executables for diagnose and program utilities"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#513-linux64-folder","title":"5.1.3 linux64 Folder","text":"\noneapi-asp/Platform-Name/linux64/\n|--include (see note below)\n|--lib\n|--libexec\n
Note: The linux64 folder does not exist in the Platform-Name directory, it is present in the oneapi-asp/common folder as the utilities are common for all reference platforms. The include and lib folders also do not exist in the oneapi-asp repository. These along with the utilities are added to the oneapi-asp/Platform-Name folder after the oneapi-asp build flow is complete.
Table 5-4: linux64 Folder Contents
Files/Folder Description include Contains header files from Intel\u00ae FPGA BBB required by MMD (see section 5.4 for more information on use of MPF from Intel\u00ae FPGA BBB in MMD) lib Contains MMD and Intel\u00ae FPGA BBB libraries libexec Contains executables/scripts for oneapi-asp board utilities"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#52-oneapi-asp-build-flow","title":"5.2 oneapi-asp Build Flow","text":"Figure 1-3 shows a high level overview of the hardware design and Figure 1-4 shows oneAPI ASP as part of the Open FPGA Stack. As shown in these images, all oneapi-asp hardware components reside in the AFU region in the PR slot.
Note: The architecture of the FIM with PR slot is explained in the FIM technical reference manuals, please refer to section 1.3 for links to the manuals.
The oneapi-asp repository contains source files for components that reside in the AFU region for each reference platform. oneapi-asp expects a compiled FIM netlist and a corresponding PR tree. The FIM database is copied to the oneapi-asp during ASP build flow (oneapi-asp/Platform-Name/scripts/build-asp.sh) and ASP compile scripts import the FIM database during oneAPI kernel compilation.
Notes: 1. FIM developer guide outlines steps to compile a FIM and generate PR tree, please refer to section 1.3 for links to FIM developer guides 2. The steps to build oneapi-asp using PR tree and build-asp.sh script are covered in the oneAPI Accelerator Support Package (ASP): Getting Started User Guide
The following figure shows the oneapi-asp build process.
Figure 5-1: oneapi-asp Build Flow
Table 5-5 Environment Variables used in Build Flow
Variable Number Environment Variable Description 1 OFS_PLATFORM_AFU_BBB Should point to location where ofs-platform-afu-bbb repository is cloned, if this variable is not set, build-asp.sh script clones the repository 2 OPAE_PLATFORM_ROOT Must point to the PR tree generated during FIM build, this is a required variable and build flow fails without this 3 LIBOPAE_C_ROOT Should point to the installation location for OPAE libraries (please see oneAPI Accelerator Support Package (ASP): Getting Started User Guide for more information on this variable setting), build-opae.sh script is used to clone & build OPAE library if this variable is not set. 4 OPAE_SDK_REPO_BRANCH If LIBOPAE_C_ROOT is not set, it is recommended to set this variable to indicate the OPAE SDK branch to be used for building OPAE libraries. > Note: build-opae.sh always clones the master branch if OPAE_SDK_REPO_BRANCH environment variable is not set. If you do not have the OPAE SDK installed, please ensure the OPAE_SDK_REPO_BRANCH is set to point to the OPAE release tag matching the release notes for the OFS release you are using. Please refer to oneAPI Accelerator Support Package (ASP): Getting Started User Guide for more information on environment variable and build steps The scripts required for oneapi-asp build that are common to all OFS reference platforms are located in oneapi-asp/common/scripts and build scripts specific to individual platforms are located in oneapi-asp/Platform-Name/scripts folders in the oneapi-asp repository, where Platform-Name is:
n6001 for Intel\u00ae FPGA SmartNIC N6001-PL reference platformd5005 for Intel\u00ae FPGA PAC D5005 reference platformfseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)iseries-dkfor Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
The build flow copies the scripts from oneapi-asp/common/scripts into the oneapi-asp/Platform-Name/scripts directory and generates the complete hardware directories for all board variants in oneapi-asp.
Note: For more information about the build scripts, please refer to README in oneapi-asp/common/scripts directory.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#53-oneapi-asp-hardware-implementation","title":"5.3 oneapi-asp Hardware Implementation","text":"This section goes deeper into the current hardware architecture of the oneapi-asp.
Figure 1-3 shows a high level overview of the hardware design. OFS reference platforms have different board variants enabling the different paths shown in Figure 1-3. Table below summarizes the board variants and paths enabled in the oneAPI ASP reference design for each. The Path numbers in the table match the ones in Figure 1-3. Figure 5-2 to 5-5 show the detailed diagram for oneapi-asp components in each of these board variants.
Note: The design files have the option to enable multiple global memories (oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems), however, for the purpose of readability, the figures 5-2 to 5-5 show the design with one global memory. For more information about multiple global memories, refer to section 5.3.3.
Table 5-6: OFS Reference Platform Board Variants
# Device Board Variant Host to EMIF with DMA Engine (Path 1) Host to Kernel Interface (Path 2) Kernel to EMIF (Path 3) Kernel to Unified Shared Memory (Path 4) Kernel to HSSI (Path 5) Figure # 1 Agilex\u00ae 7 FPGA Stratix\u00ae 10 FPGA oneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001 * ofs_fseries-dk * ofs_iseries-dk oneapi-asp for Stratix\u00ae 10 FPGA : * ofs_d5005 Yes Yes Yes No No 5-2 2 Agilex\u00ae 7 FPGA Stratix\u00ae 10 FPGA oneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001_usm * ofs_fseries-dk_usm * ofs_iseries-dk_usm oneapi-asp for Stratix\u00ae 10 FPGA : * ofs_d5005_usm Yes Yes Yes Yes No 5-3 3 Agilex\u00ae 7 FPGA oneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001_iopies Yes Yes Yes No Yes 5-4 4 Agilex\u00ae 7 FPGA oneapi-asp for Agilex\u00ae 7 FPGA : * ofs_n6001_usm_iopipes Yes Yes Yes Yes Yes 5-5 Note: Please see oneapi-asp/Platform-Name/hardware folder for the board variants for each OFS reference platform.
Figure 5-2: oneapi-asp Reference Platform Hardware Design - Board Variant #1
Figure 5-3: oneapi-asp Reference Platform with USM Hardware Design - Board Variant #2
Figure 5-4: oneapi-asp Reference Platform with IO Pipes Hardware Design - Board Variant #3
Figure 5-5: oneapi-asp Reference Platform with IO Pipes and USM Hardware Design - Board Variant #4
The ofs_plat_afu.sv is the top level entity for the oneapi-asp.
Hardware design files that are common for all OFS reference platforms (for the components inside ofs_plat_afu module) are located in oneapi-asp/common/hardware/common/build and hardware design file specific to the individual platforms are located in the oneapi-asp/Platform-Name/hardware/Board-Variant/build directory. The common files are copied to the oneapi-asp/Platform-Name/hardware/Board-Variant folder during oneapi-asp build flow. The FIM database files imported during oneapi-asp build (section 5.2) are located in oneapi-asp/Platform-Name/hardware/Board-Variant/fim_platform directory.
Tables 5-7 give a brief description of the important design files in all board variants, the location for these files shown in table 5-7 is post oneapi-asp build flow.
Table 5-7: Hardware Design Files in oneapi-asp/Platform-Name/hardware/Board-Variant/build
Files/Folder Description afu_ip.qsf Adds platform specific settings & design files, this file is sourced in the oneapi-asp revision being compiled (e.g.afu_flat.qsf) mpf_vtp.qsf Adds IP components from Intel\u00ae FPGA BBB (see section 5.3.2 below for information about use of MPF blocks from Intel\u00ae FPGA BBB) repository used in the design asp_design_files.tcl Adds design files to the project, this file is source in afu_ip.qsf oneapi_afu.json Accelerator Description File that describes the metadata associated with a oneAPI ASP. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration > Note: For more information about use of a JSON file for storing meadata for an AFU, refer to Accelerator Description File section in PIM Based AFU Developer Guide ip Contains the HW TCL and Verilog source code files for IP components used in the hardware design scripts Contains scripts to control oneAPI kernel and asp Quartus\u00ae software compile flow (see section 5.3.5 for more information on compile flow) rtl Contains the SystemVerilog and Verilog design files shown in figures 5-2 to 5-5 above rtl/ofs_plat_afu.sv Top level wrapper file for oneapi-asp. Contains the Platform Interface Manager (PIM) blocks for protocol translation & connecting FIM ports to oneapi-asp ports. For more information about PIM, see the PIM subtopic below. rtl/afu.sv Contains the PIM blocks for VTP address translation and instantiates the kernel_wrapper entity. For more information about PIM, see the PIM subtopic below. rtl/kernel_wrapper.v Contains the kernel_system instantiation rtl/asp_logic.sv Contains the DMA engine, board system instantiation, please see information about board_hw.tcl, ddr_board_hw.tcl and ddr_channel_hw.tcl below rtl/ofs_asp.vh Contains the macros to control instantiation of logic on host to memory path, logic required to enable Unified Shared Memory(USM) support and IO pipes support rtl/ofs_asp_pkg.sv Contains Verilog parameters used in controlling values of number of channels (memory/IO pipes), address widths, pipeline stages etc. for logic in the oneapi-asp board_hw.tcl Contains the Kernel Interface IP instantiation, register to store board AFU ID and ddr_board system instantiation. Please refer to ddr_board_hw.tcl and ddr_channel_hw.tcl below. > Note: The board AFU ID is used by the MMD layer during board discovery ddr_board_hw.tcl Instantiated in the board_hw.tcl, contains IP components in the host to EMIF and kernel to EMIF datapath. Memory Bank Divider is instantiated in this file. Please refer to section 3.1.1 for more details on Memory Bank Divider ddr_channel_hw.tcl Instantiated in ddr_board_hw.tcl, contains bridges required for clock domain crossings between PCIe and EMIF clocks as well as kernel and EMIF clock ofs_asp.sdc Contains clock group constraints for all clocks in oneAPI ASP ../fim_platform Contains Platform Interface Manager(PIM) and FIM database used in the design. See PIM subtopic below The hardware implementation diagrams show a PF/VF Mux/De-mux module in the AFU region. The PF/VF mux routes packets to AFU component based on pf_num and vf_num information in PCIe TLP header. For more information about the PF/VF mux, please refer to the PF/VF Mapping details in FPGA Interface Manager Technical Reference Manual for your target device (links are available in section 1.3).
The oneapi-asp resides inside the AFU region and, depending on the FIM configuration, connects to the lowest PF or lowest VF number that routes into the PR slot. Table below summazires the PF/VF mapping in oneapi-asp for some of the different FIM configurations.
Table 5-8: PF/VF Mapping in oneapi-asp
Target Device for OFS FIM Configuration* PF/VF Mapping in oneapi-asp Stratix\u00ae 10 FPGA Default PF0-VF1 Agilex\u00ae 7 FPGA Default PF0-VF0 Agilex\u00ae 7 FPGA Compiled using <n6001/iseries-dk>_1pf_1vf.ofss, i.e. PCIe Subsystem configuration is set to PF0 with 1 VF PF0-VF0 Agilex\u00ae 7 FPGA Compiled using n6001_2pf.ofss, i.e. PCIe Subsystem configuration is set to two physical functions PF0 and PF1 PF1 *Note: For more information on different FIM configurations & how to compile these, please refer to FPGA Interface Manager(FIM) Developer Guides for Open FPGA Stack for your target device (links are available in section 1.3).
Sections below provide some more information on some blocks in the hardware design block diagram shown above. Refer to section 3.1 for more information about Memory Bank Divider and to section 3.2 for information about Kernel Interface.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#531-platform-interface-managerpim","title":"5.3.1 Platform Interface Manager(PIM)","text":"In addition to above files/folders, an important part of the design are the ofs_plat_* modules provided by Platform Interface Manager (PIM). oneAPI kernel system and oneapi-asp have Avalon\u00ae interfaces. FIM components have AXI interfaces. The oneAPI compiler generated kernel_system with Avalon interfaces. The PIM modules are used for protocol translation.
In addition to this, PIM blocks are also used for Virtual to Physical (VTP) address translation. The Memory Mapped Device(MMD) layer provides virtual addresses which are converted to physical addresses by the VTP blocks to allow Unified Shared Memory(USM) and DMA transfers to/from host memory.
Note: For more information about PIM, please refer to PIM README here.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#532-direct-memory-accessdma-module","title":"5.3.2 Direct Memory Access(DMA) Module","text":"The Direct Memory Access(DMA) module is located in the host to EMIF datapath in the oneapi-asp and provides a controller to execute transfers from host to DDR on the board and vice versa. The source files for the DMA module used in oneapi-asp for OFS reference platforms are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl/dma directory (location post oneapi-asp build flow). Figure below shows the DMA module interface.
Figure 5-7: DMA Controller Block Diagram
Figure 5-2 shows the instantiation of this dma_top module as dma_controller_inst. Table 5-9 shows the signal names and descriptions. Section 5-5 covers the complete compilation flow.
Table 5-9: DMA Module Signal Descriptions
Signal or Port Description mmio64_if Control signal from host, connects to DMA Control Status Registers(CSR) host_mem_rd_avmm_if Avalon\u00ae memory-mapped interface to read from host memory host_mem_wr_avmm_if Avalon\u00ae memory-mapped interface to write to host memory local_mem_rd_avmm_if Avalon\u00ae memory-mapped interface to read from on-board DDR4 memory local_mem_wr_avmm_if Avalon\u00ae memory-mapped interface to write to on-board DDR4 memory dma_irq_host2fpga Interrupt to indicate completion of host to FPGA DMA transaction (read from host, write to DDR4) dma_irq_fpga2host Interrupt to indicate completion of FPGA DDR4 to host DMA transaction (read from DDR4, write to host)> Note: The oneapi-asp for OFS reference platforms uses a host memory write to indicate completion of FPGA to host transaction The data transfer module manages data movement from source to destination addresses.
To start a data transfer, the data transfer module requires following information, this is set by the dma_dispatcher:
- Source address
- Destination address
- Number of bytes to transfer
oneapi-asp for OFS reference platforms uses virtual addresses in the DMA controller for data transfer. The Memory Properties Factory(MPF) Virtual to Physical(VTP) blocks (i.e.mpf_vtp_* modules) shown in figure 5-2 translate virtual addresses to appropriate host addresses.
Note: Memory Properties Factory(MPF) is a part of Intel\u00ae FPGA Basic Building Blocks. Please refer to Intel\u00ae FPGA BBB repository for more information.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#533-board_hwtcl-file","title":"5.3.3 board_hw.tcl File","text":"As described in table 5-7, the board_hw.tcl contains the Kernel Interface IP instantiation, register to store board AFU ID and ddr_board system instantiation. The board_hw.tcl component has a few parameters that can be used to control the internal composition. Snippet below shows the portion of the TCL script adding these parameters. These are described in table below.
board_hw.tcl sources another file called parameters.tcl, this file is located in oneapi-asp/Platform-Name/hardware/Board-Variant/build directory and contains different values for the parameters based on the board variant. All the parameters in the board_hw.tcl derive their value from respective values in parameters.tcl(all variables starting with p_<parameter-name>) for each board variant when the oneapi-asp is built.
\n\n +-----------------------------------\n | parameters\n |\nsource parameters.tcl\n\nadd_parameter AFU_ID_H STD_LOGIC_VECTOR $p_AFU_ID_H\nset_parameter_property AFU_ID_H DEFAULT_VALUE $p_AFU_ID_H\nset_parameter_property AFU_ID_H DISPLAY_NAME \"AFU ID H\"\nset_parameter_property AFU_ID_H AFFECTS_ELABORATION true\n\nadd_parameter AFU_ID_L STD_LOGIC_VECTOR $p_AFU_ID_L\nset_parameter_property AFU_ID_L DEFAULT_VALUE $p_AFU_ID_L\nset_parameter_property AFU_ID_L DISPLAY_NAME \"AFU ID L\"\nset_parameter_property AFU_ID_L AFFECTS_ELABORATION true\n\nadd_parameter IOPIPE_SUPPORT BOOLEAN $p_IOPIPE_SUPPORT\nset_parameter_property IOPIPE_SUPPORT DEFAULT_VALUE $p_IOPIPE_SUPPORT\nset_parameter_property IOPIPE_SUPPORT DISPLAY_NAME \"IO Pipe Support\"\nset_parameter_property IOPIPE_SUPPORT AFFECTS_ELABORATION true\n\nadd_parameter NUMBER_OF_DMA_CHANNELS INTEGER $p_NUMBER_OF_DMA_CHANNELS\nset_parameter_property NUMBER_OF_DMA_CHANNELS DEFAULT_VALUE $p_NUMBER_OF_DMA_CHANNELS\nset_parameter_property NUMBER_OF_DMA_CHANNELS DISPLAY_NAME \"Number of DMA Channels\"\nset_parameter_property NUMBER_OF_DMA_CHANNELS AFFECTS_ELABORATION true\n\nadd_parameter SNOOP_PORT_ENABLE BOOLEAN $p_SNOOP_PORT_ENABLE\nset_parameter_property SNOOP_PORT_ENABLE DEFAULT_VALUE $p_SNOOP_PORT_ENABLE\nset_parameter_property SNOOP_PORT_ENABLE DISPLAY_NAME \"Enable Snoop Port\"\nset_parameter_property SNOOP_PORT_ENABLE AFFECTS_ELABORATION true\n\nadd_parameter NUMBER_OF_GLOBAL_MEMORY_SYSTEMS INTEGER $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS\nset_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS DEFAULT_VALUE $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS\nset_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS DISPLAY_NAME \"Number of Global Memory Systems\"\nset_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS AFFECTS_ELABORATION true\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 0 } {\n add_parameter MEM_0_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_0_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_0_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 0 Banks\"\n set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_0_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_0_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_0_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 0 Memory Bank Address Width\"\n set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_0_DATA_WIDTH INTEGER $p_MEM_0_DATA_WIDTH\n set_parameter_property MEM_0_DATA_WIDTH DEFAULT_VALUE $p_MEM_0_DATA_WIDTH\n set_parameter_property MEM_0_DATA_WIDTH DISPLAY_NAME \"Global Memory 0 Data Width\"\n set_parameter_property MEM_0_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_0_MAX_BURST_SIZE INTEGER $p_MEM_0_MAX_BURST_SIZE\n set_parameter_property MEM_0_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_0_MAX_BURST_SIZE\n set_parameter_property MEM_0_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 0 Maximum Burst Size\"\n set_parameter_property MEM_0_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 0 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_0_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_0_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_0_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 0 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 1 } {\n add_parameter MEM_1_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_1_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_1_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 1 Banks\"\n set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_1_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_1_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_1_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 1 Memory Bank Address Width\"\n set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_1_DATA_WIDTH INTEGER $p_MEM_1_DATA_WIDTH\n set_parameter_property MEM_1_DATA_WIDTH DEFAULT_VALUE $p_MEM_1_DATA_WIDTH\n set_parameter_property MEM_1_DATA_WIDTH DISPLAY_NAME \"Global Memory 1 Data Width\"\n set_parameter_property MEM_1_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_1_MAX_BURST_SIZE INTEGER $p_MEM_1_MAX_BURST_SIZE\n set_parameter_property MEM_1_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_1_MAX_BURST_SIZE\n set_parameter_property MEM_1_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 1 Maximum Burst Size\"\n set_parameter_property MEM_1_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 1 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_1_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_1_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_1_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 1 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 2 } {\n add_parameter MEM_2_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_2_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_2_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 2 Banks\"\n set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_2_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_2_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_2_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 2 Memory Bank Address Width\"\n set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_2_DATA_WIDTH INTEGER $p_MEM_2_DATA_WIDTH\n set_parameter_property MEM_2_DATA_WIDTH DEFAULT_VALUE $p_MEM_2_DATA_WIDTH\n set_parameter_property MEM_2_DATA_WIDTH DISPLAY_NAME \"Global Memory 2 Data Width\"\n set_parameter_property MEM_2_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_2_MAX_BURST_SIZE INTEGER $p_MEM_2_MAX_BURST_SIZE\n set_parameter_property MEM_2_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_2_MAX_BURST_SIZE\n set_parameter_property MEM_2_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 2 Maximum Burst Size\"\n set_parameter_property MEM_2_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 2 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_2_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_2_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_2_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 2 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\nif { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 3 } {\n add_parameter MEM_3_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_3_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_3_NUMBER_OF_MEMORY_BANKS\n set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME \"Number of Global Memory 3 Banks\"\n set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true\n\n add_parameter MEM_3_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_3_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_3_MEMORY_BANK_ADDRESS_WIDTH\n set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME \"Global Memory 3 Memory Bank Address Width\"\n set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_3_DATA_WIDTH INTEGER $p_MEM_3_DATA_WIDTH\n set_parameter_property MEM_3_DATA_WIDTH DEFAULT_VALUE $p_MEM_3_DATA_WIDTH\n set_parameter_property MEM_3_DATA_WIDTH DISPLAY_NAME \"Global Memory 3 Data Width\"\n set_parameter_property MEM_3_DATA_WIDTH AFFECTS_ELABORATION true\n\n add_parameter MEM_3_MAX_BURST_SIZE INTEGER $p_MEM_3_MAX_BURST_SIZE\n set_parameter_property MEM_3_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_3_MAX_BURST_SIZE\n set_parameter_property MEM_3_MAX_BURST_SIZE DISPLAY_NAME \"Global Memory 3 Maximum Burst Size\"\n set_parameter_property MEM_3_MAX_BURST_SIZE AFFECTS_ELABORATION true\n\n add_parameter MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE\n set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME \"Global Memory 3 Kernel to global memory waitrequest allowance\"\n set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true\n\n add_parameter MEM_3_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_3_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_3_MBD_TO_MEMORY_PIPE_STAGES\n set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME \"Global Memory 3 MBD to Memory Pipeline Stages\"\n set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true\n}\n\n Table 5-10 board_hw.tcl Parameters
Parameter Name in IP TCL Parameter Description AFU_ID_H AFU ID H Upper 32 bits of the board AFU ID. > Note: The board AFU ID is used by the MMD layer during board discovery AFU_ID_L AFU ID L Lower 32 bits of the board AFU ID. IOPIPE_SUPPORT IO Pipe Support Boolean value used to determine if the oneapi-asp has IO pipes support. This must be set to true for oneapi-asp board variant that has the channels element defined in the board_spec.xml. Table 5-6 shows the OFS reference platform board variants that have IO pipes enabled. To ensure correct functionality, the HSSI susbsystem in the FIM must support the settings in the oneapi-asp. For more information about channels element refer to section 2.1.8 NUMBER_OF_DMA_CHANNELS Number of DMA Channels Reserved for future use > Note: oneapi-asp tag ofs-2024.2-1 supports only 1 DMA channel SNOOP_PORT_ENABLE Enable Snoop Port Used to enable the acl_asp_snoop port for Memory Bank Divider, for more information about acl_asp_snoop port, refer to Table 3-2 NUMBER_OF_GLOBAL_MEMORY_SYSTEMS* Please see notes below the table for additional design details Number of Global Memory Systems Number of global memories connected to the oneAPI kernel > Note: oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems MEM_<id>_NUMBER_OF_MEMORY_BANKS Number of Global Memory <id> Banks The number of homogenous memory banks in <id>th global memory system, where id can be 0, 1, 2 or 3 (oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems). Used for parameterizing the ddr_board system. The ddr_board system in oneapi-asp for OFS reference platforms has a single Memory Bank Divider instantiated for each global memory. The number of memory banks in a single global memory can be 1, 2, 4 or 8 (Memory Bank Divider IP requirement). To expand number of banks in a global memory beyond this range, additional Memory Bank Didvider IP will need to be instantiated and connected in ddr_board_hw.tcl. For more information about Memory Bank Divider, please refer to section 3.1.1 MEM_<id>_MEMORY_BANK_ADDRESS_WIDTH Global Memory <id> Memory Bank Address Width This is the total addressable width for a single memory bank/interface in <id>th global memory. For example, if global memory 0 (id) consists of 32 GB on-board DDR4 memory with four 8 GB memory banks, then this memory bank address width parameter must be set to 33 (single 8 GB channel is considered) for MEM_0_MEMORY_BANK_ADDRESS_WIDTH. This value is used for parameterizing the ddr_board system. MEM_<id>_DATA_WIDTH Global Memory <id> Data Width This is the width of the data bus for a single single memory bank/interface in the <id>th global memory to the oneAPI kernel MEM_<id>_MAX_BURST_SIZE Global Memory <id> Maximum Burst Size Maximum burst size for the pipeline bridges in the host to memory as well as kernel to memory data path for <id>th global memory MEM_<id>_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE Global Memory <id> Kernel to global memory waitrequest allowance This is the waitrequest_allowance for the clock crossing bridge in the kernel to memory data path for <id>th global memory > Note: For more information about waitrequest_allowance refer to section 2.1.4 on interface attribute MEM_<id>_MBD_TO_MEMORY_PIPE_STAGES Global Memory <id> MBD to Memory Pipeline Stages Value is used to set the number of Avalon-MM pipeline bridges between Memory Bank Divider and the PCIe clock domain to EMIF clock domain clock crossing bridge for <id>th global memory * > Notes (about Number of Global Memory): 1. If the number of global memory systems in more than 1, a pipeline bridge is added in the host to memory data path ensuring the host sees a continuous address space for the total addressable memory. The kernel gets added as a single memory space using pipeline bridge, the kernel connected to each global memory system. 2. If you are using the oneAPI ASP Editor (described in Appendix B), then the RTL design parameterization and board_spec.xml file generaration is handled by the editor. However if you are manually making changes to the ASP implementation, then the ofs_asp_pkg.sv file must be updated to ensure it has all the ASP_GLOBAL_MEM_<id>_* parameters for all global memories in your system. The default file (in oneapi-asp/common/hardware/common/build/rtl if oneapi-asp build flow is not run, else update the file in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl) contains the parameters for Global memory 0 only. The board_spec.xml file must also be edited to add global memory interfaces. For more information about board_spec.xml and different global memory configurations, refer to secton 2.1.5.
The internal component instantiations and connections are done in the composition callback (compose procedure) in board_hw.tcl.
For more information about the different component commands and callback used in board_hw.tcl, ddr_board_hw.tcl and ddr_channel.tcl, please refer to the Quartus\u00ae Prime Pro Edition User Guide: Platform Designer(Component Interface Tcl Reference section)
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#534-user-datagram-protocoludp-engine","title":"5.3.4 User Datagram Protocol(UDP) Engine","text":"I/O pipes allow kernel to stream data directly using HSSI. To demonstrate this functionality, reference design in oneapi-asp repository (refer to Figure 5-4 and 5-5) has a UDP protocol engine to allow transmitting UDP/IP packets over HSSI..
Figure below shows a simple block diagram of the UDP engine.
Figure 5-8: UDP Offload Engine
The UDP engine consists of a separate receive (rx) and trasmit (tx) path. The following functionalilty is performed by this reference design engine:
- Implements an Address Resolution Protocol (ARP) functionality to respond to be able to send & respond to ARP requests. This is needed for routing between different subnets using a gateway.
- Packetizes data from kernel to add the required header information (for MAC, IP & UDP layers)
- Extracts data from packets received by removing header information
- Handles clock crossing between kernel clock and Ethernet MAC clock domains
The source files for UDP engine used in oneapi-asp for OFS reference platform are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl/udp_offload_engine directory.
Note: The same engine is used in the board variant with USM shown in Figure 5-5.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#535-hardware-compile-flow","title":"5.3.5 Hardware Compile Flow","text":"Figure below shows the compile flow overview; the oneAPI compiler generated hardware circuit is compiled by Quartus\u00ae software along with design files for oneapi-asp.
Figure 5-9: oneAPI Compile Flow Overview
The oneAPI compiler uses the board_spec.xml to get more information about the oneapi-asp configuration. board_spec.xml file has a compile element to allow control of the Quartus\u00ae software compilation flow. The attributes of this element are discussed in section 2.1.2. The oneapi-asp uses tcl scripts to control the Quartus\u00ae software compilation flow. Figure 5-10 shows the flow and scripts used. All compilation scripts are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/scripts folder (location post oneapi-asp build flow).
Figure 5-10: Compilation Scripts in oneapi-asp
Table 5-11 summarizes notes for reference numbers 1-2 marked in figure above.
Table 5-11: Notes for Reference Numbers in Figure 5-10
Reference Number Note 1 revision_name is afu_flat for oneapi-asps for OFS reference platforms 2 gen-asp-quartus-report.tcl script generates a report (acl_quartus_report.txt) containing resource utilization and kernel clock frequency summary. See Note below about the acl_quartus_report.txt file Note: The acl_quartus_report.txt is required by the oneAPI compiler to generate the Quartus (Static) summary in FPGA optimization reports successfully. The oneAPI compiler expects the format to be as follows (order of metrics can be different, but the names of these FPGA metrics should match the format given here):
\nALUTs: <value>\nRegisters: <value>\nLogic utilization: <value>\nI/O pins: <value>\nDSP blocks: <value>\nMemory bits: <value>\nRAM blocks: <value>\nActual clock freq: <value>\nKernel fmax: <value>\n1x clock fmax: <value>\n2x clock fmax: <value>\nHighest non-global fanout: <value>\n
If acl_quartus_report.txt is missing, the oneAPI compiler will fail to generate the Quartus(Static) summary in FPGA optimization reports successfully. If you are creating a custom platform, please ensure the acl_quartus_report.txt report is generated. The gen-asp-quartus-report.tcl can be used as a reference to implement the generation of this report in your custom platform
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#54-oneapi-asp-memory-mapped-devicemmd-layer-implementation","title":"5.4 oneapi-asp Memory Mapped Device(MMD) Layer Implementation","text":"As discussed in section 4.1, the MMD provides a set of API that allow the runtime to control the device and communicate with it.
The source code for MMD layer is located in oneapi-asp/common/source/host folder. aocl_mmd.h is the header file for the implemented API and is located in oneapi-asp/common/source/include folder. Table below summarizes the APIs that have been implemented in oneapi-asp for OFS reference platforms.
Note: For more details about the API, its arguments and enums please refer to the aocl_mmd.h file and to section 4.1.
Table 5-12: MMD API Implemented in oneapi-asp for OFS Reference Platforms
API aocl_mmd_get_offline_info aocl_mmd_get_info aocl_mmd_open aocl_mmd_close aocl_mmd_set_interrupt_handler aocl_mmd_set_status_handler aocl_mmd_yield aocl_mmd_read aocl_mmd_write aocl_mmd_copy aocl_mmd_program aocl_mmd_host_alloc aocl_mmd_free aocl_mmd_shared_alloc aocl_mmd_shared_migrate The implementation of these APIs is in oneapi-asp/common/source/host/mmd.cpp. The functions used in the implementation are distributed across various source files. Table below provides details on source code files.
Table 5-13: MMD Source Code Files
Files/Folder Description mmd.cpp This file has the implementation for all MMD API calls listed in table 5-12 fpgaconf.hfpgaconf.c Contains bitstream reconfiguration function declaration(.h) & definition(.c) kernel_interrupt.h Contains KernelInterrupt class declaration; the class consists of functions to handle kernel interrupts kernel_interrupt.cpp Contains KernelInterrupt class constructor and function definitions mmd_device.h Contains Device class declaration, which stores device data and has functions to interact with the device mmd_device.cpp Contains Device class constructor and function definitions mmd_dma.hmmd_dma.cpp Contain DMA functions declaration(.h) & definition(.cpp) mmd_iopipes.h Contains the iopipes class declaration(.h) mmd_iopipes.cpp Contains function definitions, these functions include iopipes class constructor as well as fucntions to detect and setup CSR space for IO pipes feature in board variants that support IO pipes zlib_inflate.hzlib_inflate.c Function declaration(.h) and definition(.c) for decompressing bitstream data CMakeLists.txt CMakeLists.txt file for building MMD source code The build flow scripts build the MMD library, i.e. libintel_opae_mmd, and place them in oneapi-asp/Platform-Name/linux64/lib folder. The MMD library is specified as part of mmdlib, linklibs element in board_env.xml and used at runtime (refer to section 5-1 for sample board_env.xml file and section 2.2 for more information about board_env.xml elements).
Use of OPAE library in MMD
The MMD layer uses API from OPAE SDK for various device operations. Hence, the MMD layers requires OPAE library to be loaded to execute successfully. The mmdlib element specifies the libopae-c library to be loaded before the MMD library (demonstrated in sample board_env.xml in section 5-1).
Note: Please refer to Software Reference Manual: Open FPGA Stack for more information about OPAE SDK API. The document also has information about linux-dfl driver.
Use of Memory Properties Factory(MPF) library in MMD
In addition to OPAE, the MMD also uses API from Memory Properties Factory(MPF) software for memory operations. The libMPF library is compiled as part of oneapi-asp build flow and placed in oneapi-asp/Platform-Name/linux64/lib. This library is also loaded before the MMD library by specifying before libintel_opae_mmd in mmdlib element in board_env.xml.
Note: Memory Properties Factory(MPF) is a part of Intel\u00ae FPGA BBB. Please refer to Intel\u00ae FPGA BBB wiki for more information about MPF.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#55-oneapi-asp-utilities-implementation","title":"5.5 oneapi-asp Utilities Implementation","text":"This section covers the implementation of board utilities (refer to section 4.2 for more information on board utilities) in oneapi-asp for OFS reference platforms.
Table below shows the source code/script locations for the utilities.
Table 5-14: oneapi-asp Utilities Source Code Locations
Utility Source Location diagnose oneapi-asp/common/source/util/diagnostic program oneapi-asp/common/source/util/reprogram installuninstallinitialize> Note: These are executable scripts oneapi-asp/common/linux64/libexec/ (copied to oneapi-asp/Platform-Name/linux64/libexec/ during oneapi-asp build flow, for more information about oneapi-asp build flow, refer to section 5.2) diagnose and program are compiled as part of the oneapi-asp build flow and executables are placed in oneapi-asp/Platform-Name/linux64/libexec/. A CMakeLists.txt file is provided for building the utilities, located in oneapi-asp/common/source/util directory.
The path to all of the above utility executables is used in utilbinder element in board_env.xml (demonstrated in sample board_env.xml in section 5-1). The runtime uses this when the corresponding aocl utility is invoked.
Brief descriptions for the source code files are given in table below.
Table 5-15: Brief Descriptions of oneapi-asp Utility Routines for OFS Reference Platforms
File Description setup_permissions.sh Helper script to configure correct device port permissions, make changes to allow users to lock pages in memory and set the hugepages required for the software stack to function correctly. The helper script is used by install, initialize routines install install routine invokes the setup_permissions.sh script after the FPGA Client Driver (FCD) is setup by the runtime uninstall uninstall routine reverts the port permission, memory locking and hugepage setting changes performed by install routine and is invoked by runtime after the FCD is removed by runtime initialize initialize routine performs the following steps: * looks for the initialization binary for the board variant to be initialized * extracts the FPGA hardware configuration file from the oneAPI fat binary using clang-offload-extract command provided by oneAPI Base Toolkit (Base Kit) version 2024.0 and beyond * invokes the setup_permissions.sh script to set correct device permissions * performs partial reconfiguration of the FPGA device by invoking program routine with the initialization bitstream as an argument >Note: For more information about how initialize utility extracts FPGA hardware configuration file from oneAPI fat binary, refer to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs program program routine allocates memory and loads the supplied initialization bitstream in memory followed by a call to reprogramming function provided by oneapi-asp's MMD library. The MMD library uses fpgaReconfigureSlot API provided by OPAE library to perform device reconfiguration> Note: Please refer to Software Reference Manual: Open FPGA Stack for more information about OPAE SDK API diagnose diagnose routine scans for the available devices for the installed platform and performs DMA transactions between host & device. It also reports the DMA transfer bandwidth. diagnose routine uses functions provided by the MMD library for scanning & opening connection to available devices"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#appendix","title":"Appendix","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#appendix-a-debug-variables-and-commands","title":"Appendix A: Debug Variables and Commands","text":""},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#a1-memory-mapped-devicemmd-layer-debug-variables","title":"A.1 Memory Mapped Device(MMD) Layer Debug Variables","text":"The Memory Mapped Device(MMD) layer provides debug capability for custom ASP and oneAPI application developers. Snippet below shows how to set the variable and table below documents the environment variables that can be used to get debug information from the MMD layer at runtime.
\n\nexport MMD_<DMA/ENABLE/PROGRAM>_DEBUG = 1\n.<oneapi-binary> # Execute sample with MMD debug environment variable set \n\n
Table A-1: Environment Variable to Enable Debug Information in MMD
Environment Variable Set to Value Description MMD_DMA_DEBUG 1 Set this environment variable to see debug information for Direct Memory Access (DMA) transactions MMD_PROGRAM_DEBUG 1 Set this environment variable to see debug information when the MMD reprograms the PR region MMD_ENABLE_DEBUG 1 Set this to see debug information for all steps in the MMD layer. Maximum debug information is printed when this environment variable is set<>"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#a2-runtime-debug-variables","title":"A.2 Runtime Debug Variables","text":"In addition to the debug variables provided by the MMD layer, there are debug variables provided by the Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology used in oneAPI base toolkit. For a full list of runtime debug variables, please refer to the Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology README.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#a3-extracting-oneapi-binary-informtion-using-aocl-binedit-command","title":"A.3 Extracting oneAPI Binary Informtion using aocl binedit Command","text":"The aocl binedit utility allows you to extract the following useful information about the compiled binary:
- Compilation environment details, such as:
* oneAPI compiler version
* Quartus\u00ae Prime software version
* Compiler command used
board_spec.xml from the BSP used for compiling
- Kernel fMAX (Quartus-compiled fMAX)
- oneAPI ASP and board variant used for compiling
You can use the aocl binedit utility with the following command:
\naocl binedit <oneapi-binary> <option:list/get/print/exists> <section_name> <output_file>\n
Table A-2: Environment Variable to Enable Debug Information in MMD
| aocl binedit Command Option | Description | | list | Lists all available sections in the given binary | | print | Writes contents of the existing named section to the standard output stream for each package file in the binary | | get | Writes contents of the existing named section to the output file | | exists | Verifies if the section exists in the package files in the binary. The non-zero exit code indicates the section does not exist |
Example
You can first use the option list to see all the available sections of the oneAPI binary:
\naocl binedit <oneapi-binary> list\n
Command below shows the sections available of a binary generated targeting a oneapi-asp board variant, in this case, the sample compiled is board_test.
\n\n $ aocl binedit board_test.fpga list\n AOCX File: binedit/aocx.0\n Sections in package file:\n .acl.board, 9 bytes\n .acl.board_package, 38 bytes\n .acl.compilation_env, 2528 bytes\n .acl.rand_hash, 40 bytes\n .acl.quartus_input_hash, 163 bytes\n .acl.compileoptions, 0 bytes\n .acl.version, 49 bytes\n .acl.autodiscovery, 2048 bytes\n .acl.board_spec.xml, 2360 bytes\n .acl.kernel_arg_info.xml, 4681 bytes\n .acl.target, 4 bytes\n .acl.rtl_hash, 566 bytes\n .acl.fpga.bin, 11948560 bytes\n .acl.quartus_report, 329 bytes\n .acl.quartus_json, 347 bytes\n\n
After knowing the sections available in the oneAPI binary, you can explore the value of each section with the printoption or save the information into a file using the get option.
Command below shows an example of the use of aocl binedit to know the board variant from which the sample was created.
\naocl binedit oneapi-binary print .acl.board\n
Sample output for a board_test.fpga binary that was created with an ofs_n6001 board variant.
\n$ aocl binedit board_test.fpga print .acl.board\nAOCX File: binedit/aocx.0\nofs_n6001\n
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#appendix-b-oneapi-accelerator-support-packageasp-editor","title":"Appendix B: oneAPI Accelerator Support Package(ASP) Editor","text":"Note: This section requires readers to be faimilar with the Prerequisites (section 1.3) and the main chapters in this reference manual.
The oneapi-asp tag ofs-2024.2-1 adds a preliminary version of an editor for enabling easy parametrization of oneAPI Accelerator Support Pacakges in OFS based platforms. The features provided by this editor are same as the oneapi-asp for OFS reference platforms described in the main chapters in this manual including support for on-board global memory, Unified Shared Memory (USM), Direct Memory Access(DMA) engine, VTP (Virtual to Physical) Address Translation, I/O Pipes, User Datagram Protocol(UDP ) offload engine. In additon to this the editor handles generation of oneAPI directory structure for your board variant, generation of XML files (board_spec.xml, board_env.xml) and JSON file(oneapi_afu.json) as well as setting parameters for the oneAPI ASP hardware design. The editor is meant to abstract away some of the design steps and reduce design time, it can be used to parameterize oneAPI ASP using IP Parameter Editor GUI in Quartus Prime.
The following sections explain functionality, parameterization and user flow for the oneAPI ASP Editor. The oneAPI Accelerator Support Package (ASP): Getting Started User Guide covers the steps to demonstrate use of this editor for OFS reference platforms.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#b1-oneapi-asp-editor-overview","title":"B.1 oneAPI ASP Editor Overview","text":"The following diagram shows GUI view and summarizes the functionality provided by the editor.
Figure B-1: oneAPI ASP Editor GUI
Table below explains the functionality marked in the figure.
Table B-1 oneAPI ASP Editor GUI Functionality
# Editor Function Description 1 Parameters Window The oneAPI ASP interface settings can be set using the parameters window. Users can also derive these settings from the FIM interfaces using presets created from the FIM settings (this requires OPAE_PLATFORM_ROOT environment variable to be set, for more information, refer to the user flow in section B.2). Details of each parameter is provided in section B.3 2 Presets Window Quartus Prime provides an option to create, modify, and save parameter values as a preset file. You can then apply the parameter values in the preset file to the current component that you are parameterizing. The presets window shows the OFS reference platform presets provided in the oneapi-asp. Table B-3 summarizes the presets provided. You can also save your own preset file , e.g. to saving a custom board variant settings. Note: For more information about applying presets, please refer to Quartus\u00ae Prime Pro Edition User Guide: Getting Started 3 Generate HDL The Generate HDL button creates the directory structure for building the oneAPI ASP for selected board variant with the parameters set in the parameter window. It performs the following tasks: * Creates the directory structure and copies required platform files for the selected board variant & device * Generates the Verilog header and package files (ofs_asp.vh and ofs_asp_pkg.sv) for a board variant based on parameters set in the oneAPI ASP Editor * Generates the board_env.xml, board_spec.xml, oneapi_afu.json > Note: For more information about this XML files, please refer to section 2.0. For more information about oneapi_afu.json, refer to table 5-7 in section 5.3 * Generates parameters.tcl file used by the board_hw.tcl file > Note: For more information about the board_hw.tcl file refer to section 5.3.3 The design files for the oneAPI ASP Editor are located in oneapi-asp/oneapi_asp_editor/ directory. The directory structure of the oneapi_asp_editor folder is shown below.
\noneapi-asp/\n|--common/\n|--|--bringup/\n|--|--hardware/\n|--|--linux64/\n|--|--scripts/\n|--|--source/\n|--Platform-Directories (example : n6001, d5005, fseries-dk, iseries-dk)\n|--oneapi_asp_editor\n|--|--oneapi_asp_editor_hw.tcl\n|--|--oneapi_asp_editor.qpf\n|--|--oneapi_asp_editor.qsf\n|--|--scripts/\n|--|--ip/\n|--|--|--presets/\n|--|--|--oneapi_asp_editor.ip\n
Table below provides more information about the files in the oneapi-asp/oneapi_asp_editor directory.
Table B-2: oneapi_asp_editor Directory Contents
File/Folder Description oneapi_asp_editor_hw.tcl The TCL file providing the oneAPI ASP editor functionality scripts Helper scripts used by the oneapi_asp_editor_hw.tcl to built the oneapi-asp oneapi_asp_editor.qpfoneapi_asp_editor.qsf Dummy Quartus project files to allow parameterization of the editor using IP Parameter Editor GUI ip Contains the oneapi_asp_editor.ip file and presets folder ip/oneapi_asp_editor.ip This is the default .ip file for the oneAPI ASP editor which is used to parameterize and generate oneapi-asp design files ip/presets This contains the preset files for OFS reference platforms. Table B-3 summarizes the preset files Table B-3: Presets
Device OFS Reference Platform Plaform Name in oneapi-asp Board Variant Preset File Agilex\u00ae 7 FPGA Intel\u00ae FPGA SmartNIC N6001-PL n6001 ofs_n6001 ofs_n6001_usm ofs_n6001_iopipes ofs_n6001_usm_iopipes ofs_n6001.qprs ofs_n6001_usm.qprs ofs_n6001_iopipes.qprs ofs_n6001_usm_iopipes.qprs Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) fseries-dk ofs_fseries-dk ofs_fseries-dk_usm ofs_fseries-dk.qprs ofs_fseries-dk_usm.qprs Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) iseries-dk ofs_iseries-dk ofs_Iseries-dk_usm ofs_iseries-dk.qprs ofs_iseries-dk_usm.qprs Stratix\u00ae 10 FPGA Intel\u00ae FPGA PAC D5005 d5005 ofs_d5005 ofs_d5005_usm ofs_d5005.qprs ofs_d5005_usm.qprs Note: For more information about the board variants, please refer to Table 5-6 in section 5.3.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#b2-oneapi-asp-editor-user-flow","title":"B.2 oneAPI ASP Editor User Flow","text":"The expected user flow with the oneAPI ASP Editor is shown in figure below.
Figure B-2: oneAPI ASP Editor User Flow
The oneAPI ASP Editor can be opened in IP Parameter Editor GUI in Quartus Prime. Building oneAPI ASPs requires FIM compilation to be complete successfully and a PR tree to be generated. The OPAE_PLATFORM_ROOT environment variable must be set to point to the PR tree generated during FIM build.
You can use the oneAPI ASP Editor GUI to set parameters for your oneAPI ASP. Based on your FIM and target platform, you can use one of the options below:
- If you are building
oneapi-asp for one of the OFS reference platforms, presets are provided in the oneapi-asp/oneapi_asp_editor/ip/presets directory. These presets should show up in the Presets window as shown in figure B-1. Table B-3 summarizes the available presets. Select and apply the preset matching the reference platform you are targeting. Save the settings and click on \"Generate HDL\".
Notes: 1. For more information about how to apply presets, refer to Quartus\u00ae Prime Pro Edition User Guide: Getting Started 2. The oneAPI ASP editor generates only the selected board variant, if you would like to generate all board variants for an OFS reference platform, you will have to apply each preset and click \"Generate HDL\" for each one sequentially.
- If you have a custom FIM, you can either use the
ofs_fim.qprs or set all parameters for the oneAPI ASP yourself.
- The
ofs_fim.qprs file is created from presets generated during FIM compilation flow. This file gets created when the oneAPI ASP editor is opened in the IP Parameter Editor GUI if the OPAE_PLATFORM_ROOT environment variable is set to point to the PR tree generated during FIM build. The ofs_fim.qprs covers some of the interface settings for the oneAPI ASP, there are other parameters required by the ASP that must be set by you when using the ofs_fim.qprs. For more information about which parameters are set by the ofs_fim.qprs, please refer to section B.3.
- Another option is to set all the oneAPI ASP Editor parameters yourself, use this option with caution. Please ensure the interface settings in your ASP match your FIM design.
Once the parameters are set, you can save optionally save your own preset file (if cutomizing editor parameters); then save the .ip (File -> Save) and click Generate HDL. For more information on the steps automated by Generate HDL, refer to table B-1.
Note: For more information about saving custom presets, refer to Quartus\u00ae Prime Pro Edition User Guide: Getting Started
For more information about each parameter, refer to section B.3. For a step-by-step example of using the oneAPI ASP editor, refer to the oneAPI Accelerator Support Package (ASP): Getting Started User Guide.
"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#b3-oneapi-asp-editor-parameters","title":"B.3 oneAPI ASP Editor Parameters","text":"Figure and table below summarizes the oneAPI ASP Editor parameter settings. Figures B-3 to B-8 along with Tables B-5 to B-9 provide details on the parameters under each tab of the editor Parameter window.
The tables B-3 to B-8 have an additional column Derived from FIM settings, the parameters marked yes in this are derived from the preset file generated from the FIM as shown in figure B-2. If you have applied this preset (ofs_fim.qprs) then the parameters marked Yes under Derived from FIM settings column are correctly set to support the interface provided by FIM, you only need to provide settings marked No under this column.
Figure B-3: oneAPI ASP Editor Parameters Summary
Table B-4: oneAPI ASP Editor Parameters Summary
Parameter/Tab Description AFU ID The AFU ID is stored in a register (board_afu_id register shown in Figures 5-2 to 5-5) in the oneAPI ASP hardware design. The board AFU ID is used by the MMD layer during board discovery. * 32-bit AFU ID MSB: Upper 32 bits of the board AFU ID * 32-bit AFU ID LSB: Lower 32 bits of the board AFU ID > Note: The AFU ID is not derived from ofs_fim.qprs, if you are using the ofs_fim.qprs file, this value must be set manually. Ensure the value in MMD (*_ASP_AFU_ID in oneapi-asp/common/source/CMakeLists.txt matches the value set in this register before building the oneapi-asp using build-asp.sh) Global Memory(On-board) Tab Contains the parameter settings for configuring the global memory for the oneAPI kernel. The settings from this tab are used to generate the global memory interface in the board_spec.xml as well as parameterize the global memory interface in the oneAPI ASP hardware design > Note: For more information about global_mem element in board_spec.xml, please refer to section 2.1.5 in this manual Unified Shared Memory(USM) Tab Contains the parameter settings for configuring the Unified Shared Memory interface. The settings from this tab are used to generate the global memory interface with allocation_type=host, shared in board_spec.xml to support USM. These parameters are also used in the oneAPI ASP hardware design for the USM datapath. > Note: For more information about the USM interface support in board_spec.xml, please refer to the section 2.1.5.1.3 in this manual Direct Memory Access(DMA) Tab This tab is reserved for future use. Please do not edit the DMA settings. Keep the Number of DMA Channels set to 1 Channels Tab Contains the parameter settings for configuring I/O Pipes to/from oneAPI kernel. The settings from this tab are used to generate the channels settings in board_spec.xml as well as parameterize the I/O pipes interface in oneAPI ASP hardware design > Note: For more information about channels, please refer to section 2.1.8 in this manual Project Settings Tab In addition to interface settings, oneAPI ASP design (hardware as well as the board_spec.xml requires some more details (e.g. project names, resources) to create oneAPI ASP directory sucessfully. This tab contains some of the additional settings required Figure B-4: oneAPI ASP Editor Global Memory(On-board) Parameters
Table B-5: oneAPI ASP Editor Global Memory(On-board) Parameters
Note: All parameters in the table below except Number of Global Memory Systems are for different for each global memory in the oneAPI ASP. If the Number of Global Memory Systems is increased, a new tab (titled Global Memory <id>, where <id> starts at 0 for the lowest global memory) gets added for each global memory and you must set the parameters for each additional global memory under its tab.
Parameter/Tab Editable Derived from FIM Settings(only if ofs_fim.qprs is applied) Description Number of Global Memory Systems -(see Note in description) No Set the numbr of global memory systems (for on-board memories) > Note: oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems Name No No The name of global memory (not editable, set to device), this value is used as the value for name attribute for global_mem element in board_spec.xml > Note: For more information about global_mem element attributes, please refer to table 2-7 in section 2.1.5 in this manual Maximum theoretical bandwidth Yes Yes The maximum theoretical bandwidth for the global memory, this is used as the value for max_bandwidth attribute for global_mem element in board_spec.xml > Note: For more information about calculating max_bandwidth, please refer to table 2-7 in section 2.1.5 in this manual Number of memory banks Yes Yes Number of memory banks in a single global memory (e.g. if your board has four DDR4 banks in a single global memory, this value must be set to 4). This value is used in the oneAPI kernel interface design in the oneAPI ASP hardware design files as well as to calculate number of interface for each global memory in the board_spec.xml Permitted Values: 1, 2, 4, 8 Data width Yes Yes Width of the data bus from oneAPI kernel to on-board memory bank (interface width to individual memory bank in a global memory). This is used as the value for width of interface in global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization > Note: For more information about interface, please refer to in section 2.1.4 in this manual Address width per bank Yes Yes The address width for each individual memory bank in the global memory. This is used to calculate the size and address attributes of each interface of global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization Burst size Yes No The burst size for each individual memory bank interface in the global memory. This is used to calculate the maxburst value of interface in global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization Pipeline stages (kernel to clock crosser) Yes No Number of pipeline stages in kernel to global memory data path (controls pipe depth of pipeline bridge on this data path in kernel_wrapper.v) Waitrequest allowance No No The value is not editable through the Editor and is calculated from number of Pipeline stages (kernel to clock crosser). This is used to set the waitrequest_allowance for interface in global_mem element in board_spec.xml as well as in oneAPI ASP hardware design parameterization Pipeline stages (MBD to global memory) Yes No Number of pipeline stages in the data path from Memory Bank Divider (MBD) to the global memory interface. This value is used in oneAPI ASP hardware design parameterization AVMM write acknowledge Yes No This is used in controlling the value for bsp_avmm_write_ack for interface in global_mem element in board_spec.xml as well as in deciding the addition of avmm_wr_ack_gen_inst block in the oneAPI ASP hardware design (refer to Figure 5-2 to 5-6) Configuration address No No This value is not editable and is calculated automatically by the oneapi_asp_editor_hw.tcl for all global memories. The value is used for the config_addr attribute of global_mem element in board_spec.xml > Note: For more information about config_addr, refer to table 2-7 in section 2.1.5 Interleaved bytes No No This value is not editable and is set to 4096 in oneapi-asp tag ofs-2024.2-1, this is a known issue, please refer to release notes for more information on the workaround. This value is used for the interleaved_bytes attribute of global_mem element in board_spec.xml > Note: For more information about the calculation of interleaved_bytes refer to table 2-7 in section 2.1.5 Latency (for oneAPI compiler) Yes No This value is used for latency attribute of global_mem element in board_spec.xml Figure B-5: oneAPI ASP Editor Unified Shared Memory Parameters
Table B-6: oneAPI ASP Editor Unified Shared Memory Parameters
Parameter/Tab Editable Derived from FIM Settings(only if ofs_fim.qprs is applied) Description Unified Shared Memory Interface Yes No Used to enable or disable Unified Shared Memory(USM) interface in the oneAPI ASP design. When this box is selected, a global_mem element with allocation_type=host, shared is added to the board_spec.xml and the data path for USM is added to the oneAPI ASP hardware design (Figures 5-3, 5-5 show the implementaion of the USM data path) Name No No The name of global memory interface for USM (not editable, set to host), this value is used as the value for name attribute for global_mem element for USM interface in board_spec.xml > Notes: * For more information about global_mem element attributes, please refer to table 2-7 in section 2.1.5 in this manual * For more information about Unified Shared Memory interface, refer to section 2.1.5.1.3 Maximum theoretical bandwidth Yes No Maximum bandwidth for the USM interface. This is used as the value for max_bandwidth attribute for global_mem element in board_spec.xml Number of interfaces No No This is not editable, set to 1 for USM interface Data width Yes No The data width for the USM interface, this value is used for the width of interface in global_mem element for USM in `board_spec.xml Address width Yes No The address width for Unified Shared Memory. This is used to calculate the size attribute for interface of global_mem element for USM in board_spec.xml Burst size Yes No The burst size for USM interface. This is used to calculate the maxburst value for interface in global_mem element for USM in board_spec.xml as well as to calculate the burst count setting for this interface in oneAPI ASP hardware design Pipeline stages Yes No Number of pipeline stages in USM data path (controls pipe depth of pipeline bridge on this data path in kernel_wrapper.v) Waitrequest allowance No No The value is not editable through the Editor and is calculated from number of Pipeline stages setting. This is used to set the waitrequest_allowance for interface in global_mem element for USM in board_spec.xml Interleaved bytes No No This value is not editable and is set to 1024, this value is used for the interleaved_bytes attribute of global_mem element for USM in board_spec.xml Latency (for oneAPI compiler) Yes No This value is used for latency attribute of global_mem element for USM in board_spec.xml Figure B-6: oneAPI ASP Editor Direct Memory Access(DMA) Parameters
Table B-7: oneAPI ASP Editor Direct Memory Access(DMA) Parameters
Parameter/Tab Editable Derived from FIM Settings(only if ofs_fim.qprs is applied) Description Number of DMA Channels -(see Note in description) No Reserved for future use - This value is used in the oneAPI ASP hardware design in implementation of the data path from host to DMA engine (see Figure 5-2 to 5-5 for the data path diagram) > Note: oneapi-asp tag ofs-2024.2-1 supports only 1 DMA channel Figure B-7: oneAPI ASP Editor Channels Parameters
Table B-8: oneAPI ASP Editor Channels Parameters
Parameter/Tab Editable Derived from FIM Settings(only if ofs_fim.qprs is applied) Description Number of I/O Channels Yes Yes Number of I/O pipes to/from oneAPI kernel. This value is used in generating the channels interface in board_spec.xml as well adding the I/O pipes interface in the oneAPI ASP hardware design (Figures 5-4 to 5-5 show the I/O pipes interface from oneAPI kernel to HSSI Subsystem) Data Width for I/O Channel <id> Yes Yes The width of each individual I/O pipe interface to/from oneAPI kernel. The <id> value starts at 0 for the first channel Figure B-8: oneAPI ASP Editor Project Settings Parameters
Table B-9: oneAPI ASP Editor Project Settings Parameters
Parameter/Tab Editable Derived from FIM Settings(only if ofs_fim.qprs is applied) Description Device Family Yes Yes Set to target device family for your platform, options are: * Stratix\u00ae 10 FPGA * Agilex\u00ae 7 FPGA Platform name Yes No Sets the name of the top-level directory that stores oneAPI ASP design files for your target platform and board variant Board variant name Yes No Sets the name for the board variant directory Quartus version Yes Yes Sets the Quartus version for the oneAPI ASP OneAPI compiler version Yes No Sets the oneAPI compiler version, this is used as the value for version attribute in board_env.xml > Note: For more information about board_env.xml attributes, refer to section 2.2 in this manual Snoop port enable Yes No When this is enabled, it enables the snoop port in the Memory Bank Divider (acl_asp_snoop_port) > Note: For more information about Memory Bank Divider, refer to section 3.1.1 Quartus project name Yes No Sets the value of project attribute of compile element in board_spec.xml > Note: For more information about the compile element attributes, refer to section 2.1.2 in this manual Quartus revision name Yes No Sets the value of revision attribute of compile element in board_spec.xml > Note: For more information about the compile element attributes, refer to section 2.1.2 in this manual Device model Yes No Specify the name of the device model file. This value is used to set the file name for device_model attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 ALMs Yes Yes Adaptive Logic Modules(ALMs) that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 FFs Yes Yes Flip-flops(FFs) that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 DSPs Yes Yes Digital Signal Processing(DSPs) blocks that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 M20Ks Yes Yes RAM (M20Ks) blocks that the board design consumes in the absence of any kernel. This value is used for populating used_resources attribute of device element > Note: For more information about device element attributes, refer to section 2.1.3 Kernel CRA data width No No This is not editable and is set to 64. This parameter sets the width of Kenel CRA interface and is used as the width value of kernel_cra port in interfaces element in board_spec.xml > Notes: * For more information about kernel_cra, please refer to table 3-4 in section 3.2.1 * For more information about interfaces element, refer to section 2.1.7 * For more information about interface attribute, refer to section 2.1.4 Kernel CRA pipeline stages Yes No Number of pipeline stages in host to kernel CRA Avalon\u00ae interface (controls pipe depth of pipeline bridge on this control path in kernel_wrapper.v) > Note: For more information about kernel_cra, please refer to table 3-4 in section 3.2.1 Kernel CRA waitrequest allowance No No The value is not editable through the Editor and is calculated from number of Kernel CRA pipeline stages setting. This is used to set the waitrequest_allowance of kernel_cra port in interfaces element in board_spec.xml Range for low clock No No Sets the lower threshold for the user clock frequency (kernel clock) range. This value is used in the creation of oneapi_afu.json file. This is not editable and is set to the following values based on target device: * Stratix\u00ae 10 FPGA: auto-400 * Agilex\u00ae 7 FPGA: auto-600 > Note: For more information about oneapi_afu.json, refer to table 5-7 in section 5.3 Range for high clock No No Sets the upper threshold for the user clock frequency (kernel clock) range. This value is used in the creation of oneapi_afu.json file. This is not editable and is set to the following values based on target device: * Stratix\u00ae 10 FPGA: auto-800 * Agilex\u00ae 7 FPGA: auto-1200 > Note: For more information about oneapi_afu.json, refer to table 5-7 in section 5.3"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#document-revision-history","title":"Document Revision History","text":"Date Release Changes May 26, 2023 2023.1 First release of oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack on https://ofs.github.io/ September 15, 2023 2023.2 1. Section 1: * Updated Figure 1-3 to add HSSI path 2. Section 2: * Added channels element in figure 2-1 and table 2-1, added information about board variants below this table * Added Section 2.1.8 on channels element 3. Section 4: * Added information about MMD API (table 4-1) and new sections 4.1.1 to 4.1.21 4. Section 5: * Moved oneapi-asp build flow information into new section 5.2 * Added new table 5-6 with oneAPI ASP board variants information * Added hardware design diagrams and information about new board variants with I/O pipes support (Hardware Design with IO Pipes and Hardware Design with IO Pipes and USM) * Updated hardware design diagrams to show PF/VF Mux/De-mux and added information about PF/VF mapping in section 5.3 * Added new section on UDP engine (section 5.3.4) * Updated figure 5-10 to remove import_opencl_kernel.tcl and add_bbb_to_pr_project.tcl * Updated table 5-13 to add mmd_iopipes.h and mmd_iopipes.cpp files Dec 13, 2023 2023.3-1 1. Section 2: * Update table 2-5 to add information about port parameter * Added new section 2.1.4.1 on port parameter * Update table & figure numbers * Changed \"master\" and \"slave\" ports to \"host\" & \"agent\" ports in figures 2. Section 4: * Added new executable call supported by initialize utility 3. Section 5: * Updated initialize utility information to add clang-offload-extractcommand usage * Updated table with hardware design files information, added information about ofs_asp.sdc * Updated compile flow diagram, added information about new gen-asp-quartus-report.tcl script * Updated hardware implementation diagrams for signal name and IP name changes, replaced mem_if_vtp block with host_mem_if_vtp * Updated OpenCL Memory Bank Divider to Memory Bank Divider * Updated OpenCL Kernel Interface to Kernel Interface Feb 2, 2024 2023.3-2 1. Section 5: * Updated Table 5-8 to add PF/VF mapping for different FIM configurations Mar 19, 2024 2024.1 1. Section 1: * Added links for FIM developer guides for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) * Added link to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs * Updated figure 1-3 to rename OpenCL Memory Bank Divider to Memory Bank Divider 2. Section 2: * Replaced links to FPGA Optimization Guide for Intel\u00ae oneAPI Toolkits with links to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs * Fixed format for table 2-6 * Updated figure 2-6 to rename OpenCL Memory Bank Divider to Memory Bank Divider * Replaced broken link for USM (Unified Shared Memory) 3. Section 3: * Updated Memory Bank Dividersection to add information about memory_bank_divider_hw.tcl. Replaced figure showing GUI for the IP with parameters in memory_bank_divider_hw.tcl and added information about these parameters in table 3-1 * Updated port name acl_bsp_snoop to acl_asp_snoopand acl_bsp_memorg_host to acl_asp_memorg_host in table 3-2 * Updated Kernel Interface section to add information about kernel_interface_hw.tcl. Replaced figure showing GUI for the IP with parameters in kernel_interface_hw.tcl and added information about these parameters in table 3-3 4. Section 5: * Replaced board_env.xml figure with code snippet * Added links to README for fseries-dk & iseries-dk oneapi-asp folders in section 5.1 * Updated directory structure in section 5.1 and added information about design files common to oneapi-asp for OFS reference platforms * Update file paths in table 5-2 * Update section for script file name changes from build-bsp.sh to build-asp.sh, setup-bsp.py to setup-asp.py, build-bsp-sw.sh to build-asp-sw.sh * Updated figure 5-2 to add a step in build-asp.sh script to copy common files to platform directory * Added Note on OPAE_SDK_REPO_BRANCH environment variable in table 5-5 * Added new OFS reference platforms (fseries-dk and iseries-dk) in table 5-6 * Update section for design file name changes from board.qsys to board_hw.tcl, ddr_board.qsys to ddr_board_hw.tcl, ddr_channel.qsys to ddr_channel_hw.tcl, bsp_logic.sv to asp_logic.sv, bsp_design_files.tcl to asp_design_files.tcl * Updated signal names bsp_mem_if to asp_mem_if, acl_bsp_memorg_host to acl_asp_memorg_host, host_mem_if_pa_bsp to host_mem_if_pa_asp * Updated hardware implementation diagrams (figure 5-2 to 5-5) for new file, signal names * Added information about RTL files to table 5-7 * Added iseries-dk_1pf_1vf.ofss in table 5-8 * Fixed section 5.3.1 header and added more information about PIM blocks in section 5.3.1 * Added new section 5.3.3 on board_hw.tcl * Updated figure 5-10 to add information about ip-deploy and qsys-generate for board_hw.tcl 5. Appendix: * Added appendix section with MMD debug variable information July 15, 2024 2024.2 1. Section 1: * Added document overview table (Table 1-1) 2. Section 2: * Added a section number 2.1.5.1.3 to Unified Shared Memory Section 3. Section 5: * Added information about oneapi_afu.json to table 5-7 * Added information about new parameters in board_hw.tcl in table 5-10(NUMBER_OF_DMA_CHANNELS, DATA_WIDTH, MAX_BURST_SIZE, KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE) and parameters.tcl * Added Note about acl_quartus_report.txt in section 5.3.5 * Updated table 5-10 for parameters for multiple global memories (NUMBER_OF_GLOBAL_MEMORY_SYSTEMS) 4. Appendix A: * Modified appendix to split into sections explaining debug variables for MMD and runtime * Added new section for Extracting oneAPI Binary Informtion usingaocl bineditCommand 5. Appendix B: * Added Appendix B to describe new GUI based flow using oneAPI ASP Editor"},{"location":"hw/common/reference_manual/oneapi_asp/oneapi_asp_ref_mnl/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/","title":"Software Installation Guide: Open FPGA Stack for PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the PCIe Attach software stack on the host. This document will not cover the process of board installation or platform bring-up. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Build and install the OPAE Software Development Kit (SDK) on the host
- Build and install the Linux DFL driver stack on the host
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating a PCIe Attach shell. The PCIe Attach shell design is supported on a number of board offerings, including the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile), Intel\u00ae FPGA SmartNIC N6000/1-PL, and Intel\u00ae FPGA PAC D5005.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-2-software-and-component-version-summary-for-ofs-pcie-attach","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach","text":"The OFS PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are where the source code resides for each of the components discussed in this document.
Component Version Download Link Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 link OPAE SDK 2.13.0-3 2.13.0-3 Linux DFL intel-1.11.0-2 intel-1.11.0-2"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-3-release-pages-for-each-pcie-attach-platform","title":"Table 3: Release Page(s) for each PCIe Attach Platform","text":"This is a comprehensive list of the platform(s) whose software build and installation steps are covered in this document.
Platform Release Page Link Stratix\u00ae 10 FPGA https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Intel\u00ae FPGA SmartNIC N6001-PL https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#12-server-requirements","title":"1.2 Server Requirements","text":""},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#121-host-bios","title":"1.2.1 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
- Intel VT for Directed I/O (VT-d) must be enabled
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#122-host-server-kernel-and-grub-configuration","title":"1.2.2 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
- OS: RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10
- Kernel: 4.18.0-dfl
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#20-ofs-software-overview","title":"2.0 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack, on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions, and currently only supported the PCIe Attach release. Its usage is detailed in the relevant Quick Start Demonstration Guideline for your platform and will not be covered here.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#31-ofs-dfl-backport-kernel-driver-installation-environment-setup","title":"3.1 OFS DFL Backport Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL Backport GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.2.2 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 3.3 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
-
It is recommended you lock your Red Hat release version to 8.10 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
-
Install the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n\nsudo dnf install kernel-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-headers-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-devel-4.18.0-553.5.1.el8_10.x86_64\n
Now you have the choice to either follow the steps in section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source or 3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#32-building-and-installing-the-ofs-dfl-backport-kernel-drivers-from-source","title":"3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Select the 4.18.0-553.5.1.el8_10 kernel as your next boot target on the build machine, then reboot.
# For multiple boots (until overwritten), use the following\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\n# Or select the kernel you want during boot time\nsudo reboot\n
1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n
2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nintel-1.11.0-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n
3. Build the kernel.
```bash\ncd /home/OFS/linux-dfl-backport\nmake && make rpm\n```\n
4. Install the newly compiled RPM package and reboot.
```bash\nsudo rpm -i intel-fpga-dfl-dkms-1.11.0-2.2024.07.25.g276007e.noarch.rpm\n\nsudo reboot\n```\n
5. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/4.18.0-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\n
If an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n
6. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n
7. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n
8. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n
9. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-4.18.0-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\n
A list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#33-installing-the-ofs-dfl-backport-kernel-drivers-from-pre-built-packages","title":"3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page under the Artifacts tab. The name will resemble kernel-*.tar.gz. For the backport repository you can also choose to download packages under the GitHub action Build dkms packages, although your version will differ from the release version of the software.
- Download and install the pre-built kernel package.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\ntar xf kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\nsudo rpm -i kernel-4.18.0_dfl_2024.06.14_276007e/intel-fpga-dfl-dkms-1.11.0-2.2024.06.14.g276007e.noarch.rpm\n\n# Set this kernel as your new default boot target\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\nsudo reboot\n
Continue from step 5 of Section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 4.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#41-opae-sdk-installation-environment-setup","title":"4.1 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package. -
Remove any currently installed OPAE packages.
sudo dnf remove opae*\n
-
Initialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\n
-
Verify that the correct tag/branch have been checkout out.
git describe --tags\n2.13.0-3\n
-
Set up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\n
The following packages will be built in the same directory as create:
-
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\n
-
Check that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\n
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#42-installing-the-opae-sdk-with-pre-built-packages","title":"4.2 Installing the OPAE SDK with Pre-Built Packages","text":"You can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.13.0-3.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.13.0-3.x86_64-*.tar.gz\n
For a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\n
"},{"location":"hw/common/sw_installation/pcie_attach/sw_install_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/","title":"Software Installation Guide: Intel Agilex 7 SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the OFS SoC Attach software stack on the host and platform. After reviewing this document, a user shall be able to:
- Set up their server environment according to the Best Known Configuration (BKC)
- Build and install the OPAE Software Development Kit (SDK) on the host
- Build and load a Yocto image with the OPAE SDK and Linux DFL Drivers included on the SoC
This document does not cover first time setup of the IPU Platform F2000X-PL platform.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating an SoC Attach release. This document will cover key topics related to initial bring up of the IPU Platform F2000X-PL software stack, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#table-2-software-component-version-summary-for-soc-attach","title":"Table 2: Software Component Version Summary for SoC Attach","text":"Name Location META-OFS https://github.com/OFS/meta-ofs, tag: ofs-2024.1-2 Host Operating System Ubuntu 22.04 LTS Host OPAE SDK 2.12.0-5 SoC OS meta-intel-ese Reference Distro 1.0-ESE (kirkstone) SoC Kernel Version 6.1.78-dfl SoC OPAE SDK 2.12.0-5 SoC Linux DFL ofs-2024.1-6.1-2 Not all components shown in Table 2 will have an update available upon release. The OPAE SDK and Linux DFL software stacks are incorporated into a Yocto image and do not need to be downloaded separately.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#20-updating-the-ipu-platform-f2000x-pl","title":"2.0 Updating the IPU Platform F2000X-PL","text":"Every IPU Platform F2000X-PL ships with pre-programmed firmware for the FPGA user1, user2, and factory images, the Cyclone 10 BMC RTL and FW, the SoC NVMe, and the SoC BIOS. In this software installation guide, we will only be focusing on the building and loading of a new SoC NVMe image. Board setup and configuration for the IPU Platform F2000X-PL is discussed in the Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#30-compiling-a-custom-yocto-soc-image","title":"3.0 Compiling a Custom Yocto SoC Image","text":"Current Yocto image architecture for SoC Attach is based off of the IOTG Yoct-based ESE BSP, with the addition of the Linux DFL kernel including the latest DFL drivers for FPGA devices alongside the OPAE SDK user space software. The image targets x86_64 SoC FPGA devices but should boot on most UEFI-based machines. The source code and documentation for this image is hosted on the meta-ofs repository.
Build requirements exceed 100 GiB of disk space, depending on which image is built. As a reference point, on a system with two Intel(R) Xeon(R) E5-2699 v4 for a total of 44 CPU cores, the initial, non-incremental build takes less than an hour of wall time.
The repo tool is needed to clone the various Yocto layer repositories used in this example.
Note: If you are behind a firewall that prevents you from accessing references using the git:// protocol, you can use the following to redirect Git to use the corresponding https repositories for Yocto only: git config --global url.https://git.yoctoproject.org/.insteadOf git://git.yoctoproject.org/.
To compile the image as-is, use the following steps (as provided in meta-ofs):
-
Create and initialize the source directory.
mkdir ofs-yocto && cd ofs-yocto\ngit clone --recurse-submodules --shallow-submodules https://github.com/OFS/meta-ofs\ncd meta-ofs\ngit checkout tags/ofs-2024.1-2\n
-
Build packages and create an image.
cd examples/iotg-yocto-ese\nTEMPLATECONF=$PWD/conf source openembedded-core/oe-init-build-env build\nbitbake mc:x86-2022-minimal:core-image-full-cmdline\n
The resulting GPT disk image is available in uncompressed (.wic) and compressed form (.wic.gz) in meta-ofs/examples/iotg-yocto-ese/build/tmp-x86-2021-minimal-glibc/deploy/images/intel-corei7-64/. With no changes the uncompressed image size is ~21 GB.
The image type core-image-full-cmdline includes the familiar GNU core utilities, as opposed to core-image-minimal which uses BusyBox instead.
The example build configuration files under build/conf/ are symlinked from examples/iotg-yocto-ese/. To customize the image, start by modifying local.conf and bblayers.conf.
The uncompressed Yocto image can be loaded onto a flash drive as discussed in section 1.5.5 Creating a Bootable USB Flash Drive for the SoC and written to NVMe as the default boot target for the SoC as demonstrated in section 2.1 Updating the F2000X-PL ICX-D SoC NVMe.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#40-verifying-the-icx-d-soc-software-stack","title":"4.0 Verifying the ICX-D SoC Software Stack","text":"The reference SoC Attach FIM and unaltered FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Full supported functionality of the HEMs is documented in this platform's associated Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs and will not be covered here.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#50-setting-up-the-host","title":"5.0 Setting up the Host","text":"External SoC Attach supports testing Host to FPGA latency, MMIO latency, and MMIO bandwidth. This testing is accomplished using the utility host_exerciser on the host, which is included as a part of OPAE. This section will cover the installation and verification flow for a host interacting with the SoC Attach workload.
Review Section 1.2 Server Requirements of the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL for a list of changes required on the host to support an IPU Platform F2000X-PL and for a list of supported OS distributions. Installation will require an active internet connection to resolve dependencies.
The following software checks may be run on the host to verify the FPGA has been detected and has auto-negotiated the correct PCIe link width/speed. These commands do not require any packages to be installed. We are using PCIe BDF b1:00.0 as an example address.
# Check that the board has enumerated successfully.\n# Your PCIe BDF may differ from what is shown below.\n$ lspci | grep accel\nb1:00.0 Processing accelerators: Intel Corporation Device bcce b1:00.1 Processing accelerators: Intel Corporation Device bcce\n\n# Check PCIe link status and speed. Width should be x16, and speed whould be 16GT/s\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.0 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Width.{0,4}'\nsudo lspci -s b1:00.1 -vvv | grep LnkSta | grep -o -P 'Speed.{0,7}'\n
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#60-installing-the-opae-sdk-on-the-host","title":"6.0 Installing the OPAE SDK On the Host","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit the opae.io page, and the Software Reference Manual
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access. If you wish to install pre-built artifacts instead of building the release yourself, skip steps 3 and 4.
-
Before Installing the newest version of OPAE you must remove any prior OPAE framework installations.
$ sudo apt-get remove opae*\n
-
The following system and Python3 package dependencies must be installed before OPAE may be built.
$ sudo apt-get install bison flex git ssh pandoc devscripts debhelper cmake python3-dev libjson-c-dev uuid-dev libhwloc-dev doxygen libtbb-dev libncurses-dev libspdlog-dev libspdlog1 python3-pip libedit-dev pkg-config libcli11-dev libssl-dev dkms libelf-dev gawk openssl libudev-dev libpci-dev libiberty-dev autoconf llvm\n\n$ python3 -m pip install setuptools pybind11 jsonschema\n
-
Clone the OPAE SDK repo. In this example we will use the top level directory OFS for our package installs.
$ mkdir OFS && cd OFS\n$ git init\n$ git clone https://github.com/OFS/opae-sdk\n$ cd opae-sdk\n$ git checkout tags/2.12.0-5\n\n# Verifying we are on the correct release tag\n$ git describe --tags\n2.12.0-5\n
-
Navigate to the automatic DEB package build script location and execute.
$ cd OFS/opae-sdk/packaging/opae/deb $ ./create\n\n# Verify all packages are present\n$ ls | grep opae.*.deb\nopae_2.12.0-5_amd64.deb\nopae-dbgsym_2.12.0-5_amd64.ddeb\nopae-devel_2.12.0-5_amd64.deb\nopae-devel-dbgsym_2.12.0-5_amd64.ddeb\nopae-extra-tools_2.12.0-5_amd64.deb\nopae-extra-tools-dbgsym_2.12.0-5_amd64.ddeb\n
-
Install your newly built OPAE SDK packages.
$ cd OFS/opae-sdk/packaging/opae/deb\n$ sudo dpkg -i opae*.deb\n
The OPAE SDK version installed the host are identical to those installed on the SoC. A set of pre-compiled OPAE SDK artifacts are included in this release. These can be downloaded from ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell and installed without building/configuring.
$ tar xf opae-*.tar.gz\n$ sudo dpkg -i opae*.deb\n
-
Enable iommu=on, pcie=realloc, and set hugepages as host kernel parameters.
# Check if parameters are already enabled\n$ cat /proc/cmdline\n
If you do not see intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200, then add them manually.
$ sudo vim /etc/default/grub\n\n# Edit the value for GRUB_CMDLINE_LINUX, add the values at the end of the variable inside of the double quotes. Example: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/rhel00-swap rd.lvm.lv=rhel00/root rd.lvm.lv=rhel00/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\"\n# Save your changes, then apply them with the following\n$ sudo grub2-mkconfig\n$ sudo reboot now\n
After rebooting, check that proc/cmdline reflects your changes.
"},{"location":"hw/common/sw_installation/soc_attach/sw_install_soc_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#11-find-and-connect-to-afu","title":"1.1. Find and connect to AFU","text":"Here is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\n
In main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#12-map-mmio-optional","title":"1.2. Map MMIO (optional)","text":"Mapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\n
The below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#13-allocate-shared-memory-buffers","title":"1.3. Allocate Shared Memory Buffers","text":"The below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\n
In main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#14-perform-acceleration","title":"1.4. Perform Acceleration","text":"The host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#15-cleanup","title":"1.5. Cleanup","text":"When the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#2-building-the-host-application","title":"2. Building the Host Application","text":"A Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
- Path to common_include.mk (from examples-afu)
- TEST name
- Source files: SRCS
- Path to .json file (relative to Makefile directory)
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#3-running-the-host-application","title":"3. Running the Host Application","text":"To run the host application, you will need to:
- Load AFU onto the FIM
- Create VF's
- Bind VF's using the OPAE Drivers
- Run application
See the associated AFU Developer Guide for details.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_host_software/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/","title":"AFU Developer Guide: OFS for Agilex\u00ae 7 FPGA PCIe Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#1-introduction","title":"1. Introduction","text":"This document is a design guide for the creation of an Accelerator Functional Unit (AFU) using Open FPGA Stack (OFS) for Agilex\u00ae 7 FPGAs PCIe Attach. The AFU concept consists of separating out the FPGA design development process into two parts, the construction of the foundational FPGA Interface Manager (FIM), and the development of the Acceleration Function Unit (AFU), as shown in the diagram below.
This diagram shows the separation of FPGA board interface development from the internal FPGA workload creation. This separation starts with the FPGA Interface Manager (FIM) which consists of the external interfaces and board management functions. The FIM is the base system layer and is typically provided by board vendors. The FIM interface is specific to a particular physical platform. The AFU makes use of the external interfaces with user defined logic to perform a specific application. By separating out the lengthy and complicated process of developing and integrating external interfaces for an FPGA into a board allows the AFU developer to focus on the needs of their workload. OFS for Agilex\u00ae 7 FPGAs PCIe Attach provides the following tools for rapid AFU development:
- Scripts for both compilation and simulation setup
- Optional Platform Interface Manager (PIM) which is a set of SystemVerilog shims and scripts for flexible FIM to AFU interfacing
- Acceleration Simulation Environment (ASE) which is a hardware/software co-simulation environment scripts for compilation and Acceleration
- Integration with Open Programmable Acceleration Engine (OPAE) SDK for rapid software development for your AFU application
Please notice in the above block diagram that the AFU region consists of static and partial reconfiguration (PR) regions where the PR region can be dynamically reconfigured while the remaining FPGA design continues to function. Creating AFU logic for the static region is described in Shell Developer Guide for the associated platform. This guide covers logic in the AFU Main region.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#11-document-organization","title":"1.1. Document Organization","text":"This document is organized as follows:
- Description of design flow
- Interfaces and functionality provided in the Agilex\u00ae 7 FPGAs PCIe Attach FIM
- Setup of the AFU Development environment
- Synthesize the AFU example
- Testing the AFU example on the Intel\u00ae FPGA SmartNIC N6001-PL card
- Hardware/Software co-simulation using ASE
- Debugging an AFU with Remote Signal Tap
This guide provides theory followed by tutorial steps to solidify your AFU development knowledge.
NOTE:
This guide uses the Intel\u00ae FPGA SmartNIC N6001-PL as the platform for all tutorial steps. Additionally, this guide and the tutorial steps can be used with other platforms, such as the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile).
Some of the document links in this guide are specific to the Intel\u00ae FPGA SmartNIC N6001-PL. If using a different platform, please use the associated documentation for your platform as there could be differences in building the FIM and downloading FIM images.
If you have worked with previous Altera\u00ae Programmable Acceleration products, you will find out that OFS for Agilex\u00ae 7 FPGAs PCIe Attach is similar. However, there are differences and you are advised to carefully read and follow the tutorial steps to fully understand the design tools and flow.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#12-prerequisite","title":"1.2. Prerequisite","text":"This guide assumes you have the following FPGA logic design-related knowledge and skills:
- FPGA compilation flows including the Quartus\u00ae Prime Pro Edition design flow
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition software, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL and coding practices to create synthesizable logic.
- Understanding of AXI and Avalon memory mapped and streaming interfaces.
- Simulation of complex RTL using industry standard simulators (Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae).
- Signal Tap Logic Analyzer tool in the Quartus\u00ae Prime Pro Edition software.
You are strongly encouraged to review the Shell Developer Guide for the associated platform.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#13-acceleration-functional-unit-afu-development-flow","title":"1.3. Acceleration Functional Unit (AFU) Development Flow","text":"The AFU development flow is shown below:
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#131-understanding-platform-capabilities","title":"1.3.1. Understanding Platform Capabilities","text":"The block diagram of the N6001 Board is shown below:
The N6001 FIM provided with this release is shown below:
This release FIM provides the following features:
- Host interface
- PCIe Gen4 x 16
- 5 - PF, 4 - VF, AXI-S TLP packets
- MSI-X interrupts
- Logic to demonstrate simple PCIe loopback
- Network interface
- 2 - QSFP28/56 cages
- 2 x 4 x 25 GbE with exerciser logic demonstrating traffic generation/monitoring
- External Memory - DDR4 - 2400
- HPS - 1GB organized as 256 Mb x 32 with 256 Mb x 8 ECC
- Channel 0, 1 - 4 GB organized as 1 Gb x 32
- Channel 2, 3 - 4 GB organized as 1 Gb x 32 with 1 Gb x 8 ECC (ECC is not implemented in this release)
- Memory exerciser logic demonstrating external memory operation
- Board Management
- SPI interface
- FPGA configuration
- Example logic showing DFH operation
- Remote Signal Tap logic
- Partial reconfiguration control logic
- ARM HPS subsystem with embedded Linux
- HPS Copy engine
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#132-high-level-data-flow","title":"1.3.2. High Level Data Flow","text":"The OFS high level data flow is shown below:
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#133-considerations-for-pim-usage","title":"1.3.3. Considerations for PIM Usage","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
The following resources are available to assist in creating an AFU:
PIM Core Concepts provides details on using the PIM and its capabilities.
PIM Based AFU Developer Guide provides details on interfacing your AFU to the FIM using the PIM.
Multi-PCIe Link AFUs provides details on encapsulation of multiple FPGA device connections as a single OPAE handle.
The examples-afu repo provides several AFU examples:
Example Description PIM-based Hybrid Native clocks Example AFU using user configurable clocks. X copy_engine Example AFU moving data between host channel and a data engine. X dma Example AFU moving data between host channel and local memory with a DMA. X hello_world Example AFU sending \"Hello World!\" to host channel. X X X local_memory Example AFU reading and writing local memory. X X These examples can be run with the current OFS FIM package. There are three AFU types of examples provided (PIM based, hybrid and native). Each example provides the following:
- RTL, which includes the following interfaces:
- Host Channel:
- Host memory, providing a DMA interface.
- MMIO, providing a CSR interface.
- Local Memory
- Host software example interfacing to the CSR interface and host memory interface, using the OPAE C API.
- Accelerator Description File .json file
- Source file list
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#134-afu-interfaces-included-with-intel-fpga-smartnic-n6001-pl","title":"1.3.4. AFU Interfaces Included with Intel\u00ae FPGA SmartNIC N6001-PL","text":"The figure below shows the interfaces available to the AFU in this architecture. It also shows the design hierarchy with module names from the fim (top.sv) to the PR region AFU (afu_main.sv). One of the main differences from the Stratix 10 PAC OFS architecture to this one is the presence of the static port gasket region (port_gasket.sv) that has components to facilitate the AFU and also consists of the PR region (afu_main.sv) via the PR slot. The Port Gasket contains all the PR specific modules and logic, e.g., PR slot reset/freeze control, user clock, remote STP etc. Architecturally, a Port Gasket can have multiple PR slots where user workload can be programmed into. However, only one PR slot is supported for OFS Release for Agilex\u00ae. Everything in the Port Gasket until the PR slot should be provided by the FIM developer. The task of the AFU developer is to add their desired application in the afu_main.sv module by stripping out unwanted logic and instantiating the target accelerator. As shown in the figure below, here are the interfaces connected to the AFU (highlighted in green) via the PCIe Attach FIM:
- AXI Streaming (AXI-S) interface to the Host via PCIe Gen4x16
- AXI Memory Mapped Channels (4) to the DDR4 EMIF interface
- AXI Streaming (AXI-S) interface to the HSSI 25 Gb Ethernet
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#2-set-up-afu-development-environment","title":"2. Set Up AFU Development Environment","text":"This section covers the setup of the AFU development environment.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#21-afu-development-environment-overview","title":"2.1. AFU development environment overview","text":"A typical development and hardware test environment consists of a development server or workstation with FPGA development tools installed and a separate server with the target OFS compatible FPGA PCIe card installed. The typical usage and flow of data between these two servers is shown below:
Note: both development and hardware testing can be performed on the same server if desired.
This guide uses Intel\u00ae FPGA SmartNIC N6001-PL as the target OFS compatible FPGA PCIe card for demonstration steps. The Intel\u00ae FPGA SmartNIC N6001-PL must be fully installed following the Board Installation Guide: OFS for Agilex\u00ae 7 PCIe Attach Development Kits. If using a different OFS FPGA PCIe card, contact your supplier for instructions on how to install and operate user developed AFUs.
The following is a summary of the steps to set up for AFU development:
- Install Quartus Prime Pro Version 24.1 for Linux with Agilex device support and required Quartus patches.
- Make sure support tools are installed and meet version requirements.
- Install OPAE SDK.
- Download the Basic Building Blocks repository.
- Build or download the relocatable AFU PR-able build tree based on your Agilex\u00ae FPGA PCIe Attach FIM.
- Download FIM to the Agilex\u00ae FPGA PCIe Attach platform.
Building AFUs with OFS for Agilex\u00ae requires the build machine to have at least 64 GB of RAM.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#22-installation-of-quartus-and-required-patches","title":"2.2. Installation of Quartus and required patches","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#23-installation-of-support-tools","title":"2.3. Installation of Support Tools","text":"Make sure support tools are installed and meet version requirements.
The OFS provided Quartus build scripts require the following tools. Verify these are installed in your development environment.
Item Version Python 3.6.8 GCC 8.5.0 cmake 3.15 git 1.8.3.1 perl 5.8.8"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#24-installation-of-opae-sdk","title":"2.4. Installation of OPAE SDK","text":"Working with the Intel\u00ae FPGA SmartNIC N6001-PL card requires opae-2.13.0-3. Follow the instructions in the Follow the instructions in the Software Installation Guide: OFS for PCIe Attach FPGAs to build and install the required OPAE SDK for the Intel\u00ae FPGA SmartNIC N6001-PL. Make sure to check out the cloned repository to tag 2.13.0-3 and branch release/2.13.0.
$ git checkout tags/2.13.0-3 -b release/2.13.0\n
Note: The tutorial steps provided in the next sections assume the OPAE SDK is installed in default system locations, under the directory, /usr. In most system configurations, this will allow the OS and tools to automatically locate the OPAE binaries, scripts, libraries and include files required for the compilation and simulation of the FIM and AFUs.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#25-download-the-basic-building-blocks-repositories","title":"2.5. Download the Basic Building Blocks repositories","text":"The ofs-platform-afu-bbb repository contains the PIM files as well as example PIM-based AFUs that can be used for testing and demonstration purposes. This guide will use the host_chan_mmio AFU example in the ofs-platform-afu-bbb repository and the hello_world sample accompanying the examples-afu repository to demonstrate how to synthesize, load, simulate, and test a PIM-based AFU using the Intel\u00ae FPGA SmartNIC N6001-PL card with the PCIe Attach FIM.
Execute the next commands to clone the BBB repository.
# Create top level directory for AFU development\n$ mkdir OFS_BUILD_ROOT\n$ cd OFS_BUILD_ROOT\n$ export OFS_BUILD_ROOT=$PWD\n# Clone the ofs-platform-afu-bbb repository.\n$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/ofs-platform-afu-bbb.git\n$ cd ofs-platform-afu-bbb\n$ git checkout tags/ofs-2024.2-1\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n# Verify retrieval\n$ ls\nLICENSE plat_if_develop plat_if_release plat_if_tests README.md\n
The documentation in the ofs-platform-afu-bbb repository further addresses - The PIM concept. - The structure of the PIM-based AFU examples. - How to generate a release and configure the PIM. - How to connect an AFU to an FIM.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#26-build-or-download-the-relocatable-pr-build-tree","title":"2.6. Build or download the relocatable PR build tree","text":"A relocatable PR build tree is needed to build the AFU partial reconfiguration area for the intended FIM. The tree is relocatable and may be copied to a new location. It does not depend on files in the original FIM build.
You can use the Intel\u00ae FPGA SmartNIC N6001-PL release package and download the PR build tree and FIM images, to develop your AFU. These are located at OFS-N6001 release
Or you can build your own FIM and generate the PR build tree during the process.
To download and untar the pr_build_template:
$ cd $OFS_BUILD_ROOT\n$ wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/n6001-images_ofs-2024-2-1.tar.gz\n$ tar -zxvf n6001-images_ofs-2024-2-1.tar.gz\n$ cd n6001-images_ofs-2024-2-1/\n$ cd pr_build_template\n$ export OPAE_PLATFORM_ROOT=$PWD\n
To build your own FIM and generate the PR build tree for the Intel\u00ae FPGA SmartNIC N6001-PL platform, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs and follow the Out-of-Tree PR FIM build flow. If you are using a different platform, refer to the Shell Developer Guide for your platform and follow the Out-of-Tree PR FIM build flow.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#27-download-fim-to-fpga","title":"2.7. Download FIM to FPGA","text":"The AFU requires that the FIM from which the AFU is derived be loaded onto the FPGA.
If you are using the Intel\u00ae FPGA SmartNIC N6001-PL release package downloaded in the previous section:
$ cd $OFS_BUILD_ROOT/n6001-images_ofs-${{ env.COMMON_OFS_RELEASE_TAR }}/\n
If you are generating your own FIM, use the unsigned FPGA binary images from your FIM build.
Downlaod the FIM to the Intel\u00ae FPGA SmartNIC N6001-PL platform. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.
$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin <N6001 SKU2 PCIe b:d.f>\n$ sudo fpgasupdate ofs_top_page2_unsigned_user2.bin <N6001 SKU2 PCIe b:d.f>\n$ sudo rsu fpga --page=user1 <N6001 SKU2 PCIe b:d.f>\n
If you are using a different platform, refer to the documentation for your platform to download the FIM images onto your Agilex\u00ae PCIe Attach Platform.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#28-set-up-required-environment-variables","title":"2.8. Set up required Environment Variables","text":"Set the required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks. You can create a simple script to set these variables and save time going forward.
# If not already done, export OFS_BUILD_ROOT to the top level directory for AFU development\n$ export OFS_BUILD_ROOT=<path to ofs build directory>\n\n# If not already done, export OPAE_PLATFORM_ROOT to the PR build tree directory\n$ export OPAE_PLATFORM_ROOT=<path to pr build tree>\n\n# If not already done, export OFS_PLATFORM_AFU_BBB to the clone of ofs-platform-afu-bbb repository which contains PIM files and AFU examples.\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu-bbb>\n\n# Quartus Tools\n# Note, QUARTUS_HOME is your Quartus installation directory, e.g. $QUARTUS_HOME/bin contains Quartus executable.\n$ export QUARTUS_HOME=<user_path>/intelFPGA_pro/24.1/quartus\n$ export QUARTUS_ROOTDIR=$QUARTUS_HOME\n$ export QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\n$ export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n$ export IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys\n$ export PATH=$QUARTUS_HOME/bin:$QSYS_ROOTDIR/bin:$QUARTUS_HOME/../sopc_builder/bin/:$PATH\n# OPAE SDK release\n$ export OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# OPAE and MPF libraries must either be on the default linker search paths or on both LIBRARY_PATH and LD_LIBRARY_PATH. \n$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#3-compiling-an-afu","title":"3. Compiling an AFU","text":"In this section, you will use the relocatable PR build tree created in the previous steps from the FIM to compile an example PIM-based AFU. This section will be developed around the host_chan_mmio and hello_world AFU examples to showcase the synthesis of a PIM-based AFU.
The build steps presented below demonstrate the ease in building and running an actual AFU on the board. To successfully execute the instructions in this section, you must have set up your development environment and have a relocateable PR Build tree as instructed in section 2 of this document.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#31-creating-the-afu-synthesis-environment","title":"3.1. Creating the AFU Synthesis Environment","text":"The PIM flow provides the script afu_synth_setup to create the synthesis environment to build the AFU examples. See how to use it below.
usage: afu_synth_setup [-h] -s SOURCES [-p PLATFORM] [-l LIB] [-f] dst\n\nGenerate a Quartus build environment for an AFU. A build environment is\ninstantiated from a release and then configured for the specified AFU. AFU\nsource files are specified in a text file that is parsed by rtl_src_config,\nwhich is part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA platform name.\n -l LIB, --lib LIB FPGA platform release hw/lib directory. If not\n specified, the environment variables OPAE_FPGA_HW_LIB\n and then BBS_LIB_PATH are checked.\n -f, --force Overwrite target directory if it exists.\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#32-building-and-running-host_chan_mmio-example-afu","title":"3.2. Building and Running host_chan_mmio example AFU","text":"The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using Avalon and AXI interfaces. However, this guide will use the AXI version of the host_chan_mmio AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the actual AFU hardware.
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#321-build-the-host_chan_mmio-example-afu","title":"3.2.1. Build the host_chan_mmio example AFU","text":"Execute afu_synth_setup as follows to create the synthesis environment for a host_chan_mmio AFU that fits the PCIe Attach FIM previously constructed.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_dev\n
Now, move into the synthesis environment afu_dev directory just created. From there, execute the afu_synth command. The successful completion of the command will produce the host_chan_mmio.gbs file under the synthesis environment directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev. $ cd afu_dev\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating host_chan_mmio.gbs\n==================================\n...\n...\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\n
The previous output indicates the successful compilation of the AFU and the compliance with the timing requirements. Analyze the reports generated in case the design does not meet timing. The timing reports are stored in the directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev/build/syn/board/n6001/syn_top/output_files/timing_report.
Once the compilation finishes successfully, load the new host_chan_mmio.gbs bitstream file into the partial reconfiguration region of the target Intel\u00ae FPGA SmartNIC N6001-PL board. Keep in mind, that the loaded image is dynamic - this image is not stored in flash and if the card is power cycled, then the PR region is re-loaded with the default AFU.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#322-download-the-host_chan_mmio-example-afu","title":"3.2.2. Download the host_chan_mmio example AFU","text":"To test the AFU in actual hardware, load the host_chan_mmio.gbs to the Intel\u00ae FPGA SmartNIC N6001-PL card. For this step to be successful, the PCIe Attach FIM must have already been loaded to the Intel\u00ae FPGA SmartNIC N6001-PL card following the steps described in Section 2 of this document. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.
Verify Board and PCIe b.d.f. For the following example, the N6001 SKU2 PCIe b:d.f is B1:00.0, however this may be different in your system.
$ fpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\n...\n
Download AFU.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev\n$ sudo fpgasupdate host_chan_mmio.gbs B1:00.0\n[sudo] password for <<Your username>>: [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#323-set-up-host-to-interface-with-example-afu","title":"3.2.3. Set up host to interface with example AFU","text":"Set up host to interface with the newly loaded AFU.
List the PFs available, the default N6001 FIM has 5 PFs.
$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
Create the Virtual Functions (VFs) provided by the FIM, the default N6001 FIM has 3 VFs. If your FIM uses only PFs, skip this step.
$ sudo pci_device B1:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.5 Processing accelerators: Intel Corporation Device bccf\nB1:00.6 Processing accelerators: Intel Corporation Device bccf\nB1:00.7 Processing accelerators: Intel Corporation Device bccf\n
Bind PFs and VFs to VFIO driver (except PF0/B1:00.0, which is the FME PF).
# Enter your username.\n$ sudo opae.io init -d 0000:b1:00.1 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 183\nAssigning /dev/vfio/183 to ceg\nChanging permissions for /dev/vfio/183 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.2 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 184\nAssigning /dev/vfio/184 to ceg\nChanging permissions for /dev/vfio/184 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.3 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x1af4,0x1000) at 0000:b1:00.3 from virtio-pci\nBinding (0x1af4,0x1000) at 0000:b1:00.3 to vfio-pci\niommu group for (0x1af4,0x1000) at 0000:b1:00.3 is 185\nAssigning /dev/vfio/185 to ceg\nChanging permissions for /dev/vfio/185 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.4 <<Your username>>\n[sudo] password for <<Your username>>: Unbinding (0x8086,0xbcce) at 0000:b1:00.4 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.4 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.4 is 186\nAssigning /dev/vfio/186 to ceg\nChanging permissions for /dev/vfio/186 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.5 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.5 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.5 is 315\nAssigning /dev/vfio/315 to ceg\nChanging permissions for /dev/vfio/315 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.6 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.6 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.6 is 316\nAssigning /dev/vfio/316 to ceg\nChanging permissions for /dev/vfio/316 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.7 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 317\nAssigning /dev/vfio/317 to ceg\nChanging permissions for /dev/vfio/317 to rw-rw----\n
Verify the new AFU is loaded. The host_chan_mmio AFU GUID is \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\".
$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xEC00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 76d7ae9c-f66b-461f-816a-5428bcebdbc5\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x6098000000000000\nPCIe s:b:d.f : 0000:B1:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 1aae155c-acc5-4210-b9ab-efbd90b970c4\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#324-run-the-host_chan_mmio-example-afu","title":"3.2.4. Run the host_chan_mmio example AFU","text":"Now, navigate to the directory of the host_chan_mmio AFU containing the host application's source code, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw. Once there, compile the host_chan_mmio host application and execute it on the host server to excercise the functionality of the AFU.
Note: If OPAE SDK libraries were not installed in the default systems directories under /usr, you need to set the OPAE_LOC, LIBRARY_PATH, and LD_LIBRARY_PATH environment variables to the custom locations where the OPAE SDK libraries were installed.
# Move to the sw directory of the the host_chan_mmio AFU. This directory holds the source for the host application.\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw\n$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n$ make\n\n# Run the application\n$ ./host_chan_mmio\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#33-building-and-running-the-hello_world-example-afu","title":"3.3. Building and running the hello_world example AFU","text":"The platform-independent examples-afu repository also provides some interesting example AFUs. In this section, you will compile and execute the PIM based hello_world AFU. The RTL of the hello_world AFU receives from the host application an address via memory mapped I/O (MMIO) write and generates a DMA write to the memory line at that address. The content written to memory is the string \"Hello world!\". The host application spins, waiting for the memory line to be updated. Once available, the software prints out the string.
The hello_world example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using CCIP, Avalon, and AXI interfaces. However, this guide will use the AXI version of the AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the AFU hardware.
hello_world\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 ccip\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u2514\u2500\u2500 hello_world.json\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 hello_world\n \u251c\u2500\u2500 hello_world.c\n \u251c\u2500\u2500 Makefile\n \u2514\u2500\u2500 obj\n \u251c\u2500\u2500 afu_json_info.h\n \u2514\u2500\u2500 hello_world.o\n
The following instructions can be used to compile other AFU samples accompanying this repository.
If not done already, download and clone the examples-afu repository.
$ cd $OFS_BUILD_ROOT $ git clone https://github.com/OFS/examples-afu.git\n$ cd examples-afu\n$ git checkout tags/ofs-2024.2-1\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#331-build-the-hello_world-example-afu","title":"3.3.1. Build the hello_world example AFU","text":"Compile the hello_word sample AFU.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_synth_setup --source hw/rtl/axi/sources.txt afu_dev\n$ cd afu_dev\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating hello_world.gbs\n==================================\n.\n.\n.\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'hello_world.gbs'\nDesign meets timing\n===========================================================================\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#332-download-the-hello_world-example-afu","title":"3.3.2. Download the hello_world example AFU","text":"To test the AFU in actual hardware, load the hello_world.gbs to the Intel\u00ae FPGA SmartNIC N6001-PL card. For this step to be successful, the PCIe Attach FIM must have already been loaded to the Intel\u00ae FPGA SmartNIC N6001-PL card following the steps described in Section 2 of this document. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.
Verify Board and PCIe b.d.f. For the following example, the N6001 SKU2 PCIe b:d.f is B1:00.0, however this may be different in your system.
$ fpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\n...\n
Download AFU.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_dev\n$ sudo fpgasupdate hello_world.gbs B1:00.0\n [sudo] password for <<Your username>>: [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#333-set-up-host-to-interface-with-example-afu","title":"3.3.3. Set up host to interface with example AFU","text":"Set up your Intel\u00ae FPGA SmartNIC N6001-PL board to work with the newly loaded hello_world.gbs file.
# List the PF's available, the default N6001 FIM has 5 PF's\n$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
Download AFU.
# Create the Virtual Functions (VFs) provided by the FIM, the default N6001 FIM has 3 VFs. \n# If your FIM uses only PFs, skip this step.\n$ sudo pci_device B1:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s B1:00\nB1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)\nB1:00.5 Processing accelerators: Intel Corporation Device bccf\nB1:00.6 Processing accelerators: Intel Corporation Device bccf\nB1:00.7 Processing accelerators: Intel Corporation Device bccf\n
Bind PFs and VFs to VFIO driver (except PF0/B1:00.0, which is the FME PF).
#Enter your username\n$ sudo opae.io init -d 0000:b1:00.1 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 183\nAssigning /dev/vfio/183 to ceg\nChanging permissions for /dev/vfio/183 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.2 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 184\nAssigning /dev/vfio/184 to ceg\nChanging permissions for /dev/vfio/184 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.3 <<Your username>>\n[sudo] password for <<Your username>>:\nUnbinding (0x8086,0xbcce) at 0000:b1:00.3 from virtio-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.3 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.3 is 185\nAssigning /dev/vfio/185 to ceg\nChanging permissions for /dev/vfio/185 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.4 <<Your username>>\n[sudo] password for <<Your username>>: Unbinding (0x8086,0xbcce) at 0000:v1:00.4 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.4 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.4 is 186\nAssigning /dev/vfio/186 to ceg\nChanging permissions for /dev/vfio/186 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.5 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.5 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.5 is 315\nAssigning /dev/vfio/315 to ceg\nChanging permissions for /dev/vfio/315 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.6 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.6 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.6 is 316\nAssigning /dev/vfio/316 to ceg\nChanging permissions for /dev/vfio/316 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.7 <<Your username>>\n[sudo] password for <<Your username>>:\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 317\nAssigning /dev/vfio/317 to ceg\nChanging permissions for /dev/vfio/317 to rw-rw----\n
Verify the new AFU is loaded. The hello_world AFU GUID is \"c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\".
$ fpgainfo port\n\n//****** PORT ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE098000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0xC098000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0xA098000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n//****** PORT ******//\nObject Id : 0x8098000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x6098000000000000\nPCIe s:b:d.f : 0000:B1:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 1aae155c-acc5-4210-b9ab-efbd90b970c4\n//****** PORT ******//\nObject Id : 0x4098000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x2098000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#334-run-the-hello_world-example-afu","title":"3.3.4. Run the hello_world example AFU","text":"Compile and execute the host application of the hello_world AFU. You should see the application outputs the \"Hello world!\" message in the terminal.
# Move to the sw directory of the hello_world AFU\n$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n\n$ make\n\n# Launch the host application\n$ ./hello_world\n Hello world!\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#34-modify-the-afu-user-clocks-frequency","title":"3.4. Modify the AFU user clocks frequency","text":"An OPAE compliant AFU specifies the frequency of the uclk_usr and uclk_usr_div2 clocks through the JSON file for AFU configuration located under the <afu_example>/hw/rtl directory of an AFU design. For instance, the AFU configuration file of the host_chan_mmio example is $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/host_chan_mmio.json.
The AFU specifies the frequency for uClk_usr in its platform configuration file using the following key:value pairs:
\"clock-frequency-high\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\n \"clock-frequency-low\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\n
These key:value tuples are used to configure the PLL of the target platform that provides the user clocks through the AFU clocks interface. In addition, the specified frequency affects the timing closure process on the user clocks during AFU compilation.
Setting the value field to a float number (e.g., 315.0 to specify 315 MHz) drives the AFU generation process to close timing within the bounds set by the low and high values and sets the AFU's JSON metadata to specify the user clock PLL frequency values.
The following example shows the JSON file of the host_chan_mmio to set the AFU uClk to 500 MHz and uClk_div2 to 250 MHz.
{\n \"version\": 1,\n \"afu-image\": {\n \"power\": 0,\n \"clock-frequency-high\": 500,\n \"clock-frequency-low\": 250,\n \"afu-top-interface\":\n {\n \"class\": \"ofs_plat_afu\"\n },\n \"accelerator-clusters\":\n [\n {\n \"name\": \"host_chan_mmio\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\"\n }\n ]\n }\n}\n
Save the changes to host_chan_mmio.json file, then execute the afu_synth_setup script to create a new copy of the AFU files with the modified user clock settigns.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_clks\n\nCopying build from /home/<user_area>/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/build...\nConfiguring Quartus build directory: build_n6001_afu_clks/build\nLoading platform database: /home/<user_area>/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting platform/platform_afu_top_config.vh\nWriting platform/platform_if_addenda.qsf\nWriting ../hw/afu_json_info.vh\n
Compile the host_chan_mmio AFU with the new frequency values. $ cd afu_clks\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\n
During the compilation phase, you will observe the Timing Analyzer uses the specified user clock frequency values as the target to close timing.
AFU developers must ensure the AFU hardware design meets timing. The compilation of an AFU that fails timing shows a message similar to the following.
.\n.\n.\n\nWrote host_chan_mmio.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\n*** Design does not meet timing\n *** See build/syn/board/n6001/syn_top/output_files/timing_report\n\n===========================================================================\n
The previous output indicates the location of the timing reports for the AFU designer to identify the failing paths and perform the necessary design changes. Next, is a listing of the timing report files from a host_chan_mmio AFU that fails to meet timing after modifying the user clock frequency values.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/n6001/afu_clks\n$ ls build/syn/board/n6001/syn_top/output_files/timing_report\n\nclocks.rpt clocks.sta.fail.summary clocks.sta.pass.summary\n
Warning: AFU developers must inform software developers of the maximum operating frequency (Fmax) of the user clocks to avoid any unexpected behavior of the accelerator and potentially of the overall system.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#4-simulating-an-afu-using-ase","title":"4. Simulating an AFU using ASE","text":"The Application Simulation Environment (ASE) is a hardware/software co-simulation environment for your AFU. See diagram below illustrating ASE operation:
ASE uses the simulator Direct Programming Interface (DPI) to provide HW/SW connectivity. The PCIe connection to the AFU under testing is emulated with a transactional model.
The following list describes ASE operation:
- Attempts to replicate the transactions that will be seen in real system.
- Provides a memory model to AFU, so illegal memory accesses can be identified early.
- Not a cache simulator.
- Does not guarantee synthesizability or timing closure.
- Does not model system latency.
- No administrator privileges are needed to run ASE. All code is user level.
The remainder of this section is a tutorial providing the steps on how to run ASE with either Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae using an example AFU and the AFU build tree previously created in this guide.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#41-set-up-steps-to-run-ase","title":"4.1. Set Up Steps to Run ASE","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#411-install-opae-sdk","title":"4.1.1. Install OPAE SDK","text":"The N6001 SKU2 card requires 2.13.0-3. Follow the instructions provided in the Follow the instructions in the Software Installation Guide: OFS for PCIe Attach FPGAs to build and install the required OPAE SDK for the Intel\u00ae FPGA SmartNIC N6001-PL. Make sure to check out the cloned repository to tag 2.13.0-3 and branch release/2.13.0.
$ git checkout tags/2.13.0-3 -b release/2.13.0\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#412-install-ase-tools","title":"4.1.2 Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK.
-
If not done already, set the environment variables as described in section, Set Up AFU Development Environment.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim $ git checkout tags/2.13.0-2 -b release/2.13.0\n
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#413-setup-required-ase-environment-variables","title":"4.1.3. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ cd /usr/bin\n$ export PATH=$PWD:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ cd /usr/lib\n$ export LIBRARY_PATH=$PWD\n$ cd /usr/lib64\n$ export LD_LIBRARY_PATH=$PWD\n$ cd $OFS_BUILD_ROOT/ofs-platform-afu-bbb\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to Modelsim installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#42-simulating-the-host_chan_mmio-afu","title":"4.2. Simulating the host_chan_mmio AFU","text":"The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files:
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\n
This example AFU contains examples using both Avalon and AXI interface buses. This guide will use the AXI version of the host_chan_mmio AFU.
ASE uses client-server application architecture to deliver hardware/software co-simulation. You require one shell for the hardware based simulation and another shell where the software application is running. The hardware is started first with a simulation compilation and simulator startup script, once the simulator has loaded the design, it will wait until the software process starts. Once the software process starts, the simulator proceeds. Transaction logging and waveform capture is performed.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#421-set-up-and-run-the-hw-simulation-process","title":"4.2.1 Set Up and Run the HW Simulation Process","text":"You will run the afu_sim_setup script to create the scripts for running the ASE environment. The afu_sim_setup script has the following usage:
usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
Run afu_sim_setup to create the ASE simulation environment for the host_chan_mmio example AFU. The '-t VCS' option indicates to prepare the ASE simulation environment for Synopsys\u00ae VCS\u00ae.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n\n$ afu_sim_setup -s ./hw/rtl/test_mmio_axi1.txt -t VCS afu_sim\n\nCopying ASE from /opae-sdk/install-opae-sdk/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /ofs-agx7-pcie-attach/work_pr/build_tree/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup creates the ASE scripts in the directory host_chan_mmio_sim where the afu_sim_setup script was run. Start the simulator as shown below:
$ cd afu_sim\n$ make\n$ make sim\n
This process launches the AFU hardware simulator. Before moving to the next section, pay attention to the simulator output highlighted in the image below.
The simulation artifacts are stored in host_chan_mmio/work and consist of:
log_ase_events.tsv\nlog_ofs_plat_host_chan.tsv log_ofs_plat_local_mem.tsv log_pf_vf_mux_A.tsv log_pf_vf_mux_B.tsv
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#422-set-up-and-run-the-sw-process","title":"4.2.2 Set Up and Run the SW Process","text":"Open an additional shell to build and run the host application that communicates with the actual AFU hardware. Set up the same environment variable you have set up in the shell you have been working on until this point.
Additionally, as indicated by the hardware simulator output that is currently executing in the \"simulator shell\", copy and paste the line \"export ASE_WORKDIR=...\", into the new \"software shell\". See the last image of the previous section.
$ export ASE_WORKDIR=$OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_sim/work\n
Then, go to the sw directory of the host_chan_mmio AFU example to compile the host application. $ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw $ make\n\nafu_json_mgr json-info --afu-json=../hw/rtl/host_chan_mmio.json --c-hdr=obj/afu_json_info.h\nWriting obj/afu_json_info.h\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c main.c -o obj/main.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c test_host_chan_mmio.c -o obj/test_host_chan_mmio.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/connect.c -o obj/connect.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/csr_mgr.c -o obj/csr_mgr.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/hash32.c -o obj/hash32.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/test_data.c -o obj/test_data.o\ncc -o host_chan_mmio obj/main.o obj/test_host_chan_mmio.o obj/connect.o obj/csr_mgr.o obj/hash32.o obj/test_data.o -z noexecstack -z relro -z now -pie -luuid -lopae-c-ase\n
Now, launch the host application to exercise the AFU hardware running on the simulator shell. The next image shows the AFU hardware simulation process on the left side shell. The right hand shell shows the host application's output of a successful simulation.
$ with_ase ./host_chan_mmio\n [APP] Initializing simulation session ...\nRunning in ASE mode\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n [APP] Deinitializing simulation session\n [APP] Took 1,003,771,568 nsec\n [APP] Session ended\n
Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
$ make wave\n
This brings up the VCS\u00ae simulator GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | afu , as shown below.
Right click on the afu (afu) entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#43-simulating-the-hello_world-afu","title":"4.3 Simulating the hello_world AFU","text":"In this section you will quickly simulate the PIM-based hello_world sample AFU accompanying the examples-afu repository.
-
Set the environment variables as described in section 4.1. Set Up Steps to Run ASE.
-
Prepare an RTL simulation environment for the AXI version of the hello_world AFU.
Simulation with ASE requires two software processes, one to simulate the AFU RTL and the other to run the host software that excercises the AFU. To construct an RTL simulation environment under the directory simulation, execute the following.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_sim_setup -s ./hw/rtl/axi/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/<user_area>/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup script constructs an ASE environment in the hello_world_sim subdirectory. If the command fails, confirm that the path to the afu_sim_setup is on your PATH environment variable (in the OPAE SDK bin directory) and that your Python version is at least 2.7.
- Build and execute the AFU RTL simulator.
$ cd afu_sim\n$ make\n$ make sim
The previous commands will build and run the Synopsys\u00ae VCS\u00ae RTL simulator, which prints a message saying it is ready for simulation. The simulation process also prints a message instructing you to set the ASE_WORKDIR environment variable in a second shell.
-
Open a second shell where you will build and execute the host software. In this new \"software shell\", set up the environment variables you have set up so far in the \"hardware simulation\" shell.
-
Also, set the ASE_WORKDIR environment variable following the instructions given in the \"hardware simulation\" shell.
$ export ASE_WORKDIR=$OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_sim/work\n
6. Then, move to the sw directory of the hello_world AFU sample to build the host software. $ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n$ make
- Run the
hello_world host application to resume the work of the RTL simulation. The host software process and the RTL simulation execute in lockstep. If successful, you should see the Hello world! output.
$ with_ase ./hello_world\n\n[APP] Initializing simulation session ...\nHello world!\n[APP] Deinitializing simulation session\n[APP] Took 43,978,424 nsec\n[APP] Session ended\n
The image below shows the simulation of the AFU hardware and the execution of the host application side-by-side.
- Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
make wave\n
This brings up the DVE GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | hello_afu, as shown below.
Right click on the hello_afu entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#5-adding-remote-signal-tap-logic-analyzer-to-debug-the-afu","title":"5. Adding Remote Signal Tap Logic Analyzer to debug the AFU","text":"The OPAE SDK provides a remote Signal Tap facility. It also supports the following in system debug tools included with the Quartus Prime Pro Edition:
- In-system Sources and Probes
- In-system Memory Content Editor
- Signal Probe
- System Console
This section is a short guide on adding remote Signal Tap instances to an AFU for in system debugging. You can follow the steps in the following sections, in order of execution to create an instrumented AFU. The host_chan_mmio AFU is used in this guide as the target AFU to be instrumented.
You need a basic understanding of Signal Tap. Please see the Signal Tap Logic Analyzer: Introduction & Getting Started Web Based Training for more information.
You will run with a Signal Tap GUI running locally on the server with the Intel\u00ae FPGA SmartNIC N6001-PL as shown below:
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#51-adding-rstp-to-the-host_chan_mmio-afu","title":"5.1. Adding RSTP to the host_chan_mmio AFU","text":"RSTP is added to an AFU by:
- Defining signals to be instrumented in Signal Tap. This creates a new *.stp file.
- Modify ofs_top.qpf to include the new *.stp file
- Modify ofs_top.qsf
- Modify ofs_pr_afu.qsf
- Run $OPAE_PLATFORM_ROOT/bin/afu_synth to build the PR-able image containing the RSTP instance
You can use these detailed steps to add Signal Tap to your AFU.
-
Set path to platform root directory and create the host_chan_mmio AFU Quartus project for adding Signal Tap.:
$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n\n# we will now build a new host_chahnel_mmio example based on Signal Tap\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_stp\n
-
Navigate to host_chan_mmio AFU Quartus project and open the project using Quartus GUI.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top\n$ quartus ofs_top.qpf &\n
-
Once the project is loaded in Quartus, run Analysis & Synthesis Processing | Start | Start Analysis & Synthesis. When complete, review the project hierarchy as shown in the Project Navigator. This example will add Signal Tap probe points to the AFU region. Reviewing the code will give insight into the function of this block. You can bring up the code in the Project Navigator by expanding afu_top - port_gasket - pr_slot - afu_main - port_afu_instances - ofs_plat_afu, then select instance afu, right click, select Locacte Node - Locate in Design File as shown below.
-
Bring up Signal Tap to create the *.stp file. In the Quartus GUI, go to Tools - Signal Tap Logic Analyzer. In the New File from Template pop up, click Create to accept the default template. The Signal Tap Logic Analyzer window comes up.
-
Set up the clock for the Signal Tap logic instance by clicking ... button as shown below:
-
In the Hierarchy Level, navigate to top - afu_top - pg_afu.port_gasket - pr_slot - afu_main - port_afu_instances - ofs_plat_afu, then select instance afu.
-
Enter *clk* in the Named: box and click Search. This brings up matching terms. Click clk and >. Verify your Node Finder is as shown below and then click Ok:
-
Double click the Double-click to add nodes as shown below:
-
The Node Finder comes up. Once again navigate to top - afu_top - port_gasket - pr_slot - afu_main - port_afu_instances - ofs_plat_afu, then select instance afu and click Ok. Enter mmio64_reg* and click Search. Then click >> to add these signals to the STP instance as shown below:
-
Then click Insert and Close.
-
Save the newly created STP by clicking File - Save As and in the save as navigate to $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top and save the STP file as host_chan_mmio.stp as shown below:
Select Yes when asked to add host_chan_mmio.stp to current project. Close Signal Tap window.
-
Edit ofs_pr_afu.qsf to add host_chan_mmio.stp file and enable STP. Open $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top/ofs_pr_afu.qsf in an editor and add the lines shown below:
set_global_assignment -name VERILOG_MACRO \"INCLUDE_REMOTE_STP\"\nset_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE host_chan_mmio.stp\nset_global_assignment -name SIGNALTAP_FILE host_chan_mmio.stp\n
Save the ofs_pr_afu.qsf and close Quartus. -
The host_chan_mmio AFU Quartus project is ready to be built. In your original build shell enter the following commands:
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\n\n...\n...\nWrote host_chan_mmio.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\n
-
Once compilation completes, the new host_chan_mmio.gbs file that contains the Signal Tap instance can be loaded.
# For the following example, the N6001 SKU2 PCIe b:d.f is assumed to be b1:00.0,\n# however this may be different in your system\n# Enusre FIM is loading prior to loading AFU\n# Load AFU\n$ sudo fpgasupdate host_chan_mmio.gbs b1:00.0\n[2021-12-04 07:16:59,101] [WARNING ] Update starting. Please do not interrupt.\n[2021-12-04 07:16:59,740] [INFO ] Partial Reconfiguration OK\n
-
Use the OPAE SDK mmlink tool to create a TCP/IP connection to your Agilex\u00ae card under test. The mmlink command has the following format:
Usage:\nmmlink\n<Segment> --segment=<SEGMENT NUMBER>\n<Bus> --bus=<BUS NUMBER> OR -B <BUS NUMBER>\n<Device> --device=<DEVICE NUMBER> OR -D <DEVICE NUMBER>\n<Function> --function=<FUNCTION NUMBER> OR -F <FUNCTION NUMBER>\n<Socket-id> --socket-id=<SOCKET NUMBER> OR -S <SOCKET NUMBER>\n<TCP PORT> --port=<PORT> OR -P <PORT>\n<IP ADDRESS> --ip=<IP ADDRESS> OR -I <IP ADDRESS>\n<Version> -v,--version Print version and exit\n
Enter the command below to create a connection using port 3333:
$ sudo mmlink -P 3333 -B 0xb1\n\n------- Command line Input START ----\n\nSocket-id : -1\n Port : 3333\nIP address : 0.0.0.0\n ------- Command line Input END ----\n\nPORT Resource found.\nServer socket is listening on port: 3333\n
Leave this shell open with the mmlink connection.
-
In this step you will open a new shell and enable JTAG over protocol. You must have Quartus 24.1 Programmer loaded on the N6001 server for local debugging.
$ jtagconfig --add JTAG-over-protocol sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0\n\n# Verify connectivity with jtagconfig --debug\n$ jtagconfig --debug\n1) JTAG-over-protocol [sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0]\n(JTAG Server Version 21.4.0 Build 67 12/06/2021 SC Pro Edition)\n020D10DD VTAP10 (IR=10)\nDesign hash 86099113E08364C07CC4\n + Node 00406E00 Virtual JTAG #0\nCaptured DR after reset = (020D10DD) [32]\nCaptured IR after reset = (155) [10]\nCaptured Bypass after reset = (0) [1]\nCaptured Bypass chain = (0) [1]\n
-
Start Quartus Signal Tap GUI, connect to target, load stp file by navigating to $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top. The Quartus Signal Tap must be the same version of Quartus used to compile the host_chan_mmio.gbs. Quartus Prime Pro Version 24.1 is used in the steps below:
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_stp/build/syn/board/n6001/syn_top\n$ quartus_stpw host_chan_mmio.stp &\n
This command brings up Signal Tap GUI. Connect to the Signal Tap over protocol by selecting the Hardware button on the right side of the GUI and click the \"Please Select\" pull down as shown below:
JTAG over protocol selected:
This connection process will take approximately 2-3 minutes for the Signal Tap instance to indicate \"Ready to acquire\".
-
Set the trigger condition for a rising edge on signal awvalid signal.
-
In the Signal Tap window, enable acquisition by pressing key F5, the Signal Tap GUI will indicate \"Acquisition in progress\". Create and bind the VFs, then run the host_chan_mmio application following 3.2. Loading and Running host_chan_mmio example AFU, and observe that the Signal Tap instance has triggered. You should see signals being captured in the Signaltap GUI.
See captured image below:
To end your Signal Tap session, close the Signal Tap GUI, then in the mmlink shell, enter ctrl c to kill the mmlink process. To remove the JTAG over protocol connection:
# This is assuming the JTAG over protocol is instance '1', as shown during jtagconfig --debug\n$ jtagconfig --remove 1\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#6-disabling-the-flr-function-level-reset-during-afu-debugging","title":"6. Disabling the FLR (Function Level Reset) during AFU Debugging","text":"The vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\n
Enable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\n
If you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\n
Sample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#7-how-to-modify-the-pfvf-mux-configuration","title":"7. How to modify the PF/VF MUX configuration","text":"For information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_ofs_agx7_pcie_attach/ug_dev_afu_ofs_agx7_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
- C/C++
- Verilog/SystemVerilog
- RTL simulators such as Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
- Verilog
- SystemVerilog
- VHDL
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
- Software: OPAE API implemented in the C programming language.
- Hardware: PCIe SS TLP specification implemented in SystemVerilog.
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":" - The ASE provides a protocol checker to ensure protocol correctness. The ASE also provides methods to identify potential issues early, before in-system deployment.
- The ASE can help identify certain lock conditions and Configuration and Status Registers (CSR) address mapping and pointer math errors.
- The ASE tracks memory requested from the accelerator. The memory model immediately flags illegal memory transactions to locations outside of requested memory spaces. Consequently, you can fix incorrect memory accesses early, during the simulation phase.
- The ASE does not guarantee that you can synthesize an AFU. After you verify the AFU RTL functionality in the ASE, use the ASE and the Quartus\u00ae Prime Pro Edition software iteratively to generate the Accelerator Function (AF).
- The ASE does not require administrator privileges. After installing all the required tools, you can run the ASE on a plain vanilla user Linux machine.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#23-ase-limitations","title":"2.3. ASE Limitations","text":"When using ASE in the application development cycle, consider the following limitations:
- The ASE is a transaction-level simulator. It does not model either Intel UPI- or PCIe-specific packet structures and protocol layers.
- The ASE does not simulate caching and is not a cache simulator. It cannot reliably simulate cache collisions or capacity issues.
- Although ASE models some latency parameters, it cannot model real-time system-specific latency. It is also not an accurate timing simulation of the design or latency and bandwidth of the real system. The ASE models enable you to develop functionally correct accelerators.
- The ASE does not simulate multi-AFU or multi-socket configurations.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#24-ase-based-afu-design-workflow","title":"2.4 ASE-Based AFU Design Workflow","text":"AFU development using the ASE includes the following four stages:
-
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
-
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
-
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
-
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
- The AFU RTL code is synthesizable.
- The AFU RTL code meets timing.
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
- C-Compiler: gcc 8.5.0 or above
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
- CMake: version 3.15 or above
- Python: version 3.6.8 or above
- Intel Quartus Prime Pro 24.1: The ASE must find the
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.
The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
- Compilation of the SystemVerilog Direct Programming Interface (DPI) constructs
- Compilation of the standard examples that are included in the installation
- Support for SystemC
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#4-package-description","title":"4. Package Description","text":"The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\n
This directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#41-ase-scripts","title":"4.1. ASE Scripts","text":"The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#411-simulation-tool-set-up","title":"4.1.1. Simulation Tool Set Up","text":"Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#412-ase-environment-check","title":"4.1.2. ASE Environment Check","text":"This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#413-afu-simulation-using-the-ase","title":"4.1.3. AFU Simulation Using the ASE","text":"Before configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#4131-afu_sim_setup","title":"4.1.3.1. afu_sim_setup","text":"The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\n
* The only required argument to the afu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.
afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.
generate_ase_environment.py: This script instantiates your simulated platform configuration.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#4132-rtl_src_configpy","title":"4.1.3.2. rtl_src_config.py","text":"The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#4133-generate_ase_environmentpy","title":"4.1.3.3. generate_ase_environment.py","text":"The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
- Synopsys: Creates
synopsys_sim.setup and vcs_run.tcl in the configuration directory. - Siemens: Creates
vsim_run.tcl in the configuration directory.
The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#5-ase-usage","title":"5. ASE Usage","text":"The AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
- Server: A makefile in the
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code. - Client: The main
cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#51-ase-build-instructions","title":"5.1. ASE Build Instructions","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#513-install-ase-tools","title":"5.1.3. Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\n
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#514-ase-simulator-server-build-instructions","title":"5.1.4. ASE Simulator (Server) Build Instructions","text":"ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
Change directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n...
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#514-ase-runtime-instructions","title":"5.1.4. ASE Runtime Instructions","text":"The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
- Terminal 1: In the simulation directroy
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.
Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\n
You can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
- Terminal 2: First set the environment variable
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c.
# Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\n
Note: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\n
Upon completion, the simulation generates the following files:
- Waveform dump:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#52-ase-makefile-targets","title":"5.2. ASE Makefile Targets","text":"COMMAND DESCRIPTION make Build the HW Model using RTL supplied make sim Run simulator - ASE can be run in one of 4 modes set in ase.cfg - A regression mode can be enabled by writing ASE_MODE = 4 in ase.cfg and supplying an ase_regress.sh script make wave Open the waveform (if created) to be run after simulation completes make clean Clean simulation files make distclean Clean ASE sub-distribution"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#53-ase-makefile-variables","title":"5.3. ASE Makefile Variables","text":"Makefile switch DESCRIPTION ASE_CONFIG Directly input an ASE configuration file path (ase.cfg) ASE_SCRIPT Directly input an ASE regression file path (ase_regress.sh, for ASE_MODE=4) SIMULATOR Directly input a simulator brand (select between 'VCS' or 'QUESTA') ASE_DISABLE_CHECKER Legacy - Disable CCI-P protocol checker module (set to '1' might speed up simulation) WARNING => NO warnings on hazards, protocol checks, timeouts will be generated. This option must be ONLY used if the design is already CCI-P compliant and fast simulation of app-specific logic is needed"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#54-ase-runtime-configuration-options","title":"5.4. ASE Runtime Configuration Options","text":"The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
Switch Name Default Description ASE_MODE 1 ASE mode has the following valid values: 1 : Standard Server-Client Mode2 : Simulator stops after ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
- ASE_INFO: Prints mandatory information messages required to specify operation.
- ASE_ERR: Prints error messages during operation.
- ASE_MSG: Prints general messages indicating check points in the ASE. Suppress these messages by setting the environment variable
ASE_LOG to 0.
Two log levels are supported in ASE, controlled by env(ASE_LOG)
- ASE_LOG=0 | ASE_LOG_SILENT : Only INFO, ERR messages are posted
- ASE_LOG!=0 | ASE_LOG_MESSAGE : All MSG, INFO, ERR messages are posted
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\n
You cannot suppress warnings and errors."},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#56-troubleshooting-and-error-reference","title":"5.6. Troubleshooting and Error Reference","text":"The following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If using ASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_afu_sim_env/images/readme/","title":"Readme","text":"images
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
- AFU Build environment
- Using the PIM to interface with an AFU
- AFU Design
- Software Development
- Packaging the AFU
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X A Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\n
The OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
- ofs_plat_if.vh: PIM's top level wrapper interface for passing all top-level interfaces into an AFU and is copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.
- PIM interfaces are defined in base_ifcs and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).
- PIM modules are defined in ifcs_classes and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support: - host_chan
- local_memory
- hssi
- Other
- PIM utilities are defined in utils and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#3-using-pim-to-interface-with-an-afu","title":"3. Using PIM to interface with an AFU","text":"To interface the PIM with an AFU:
- Create top level module ofs_plat_afu.sv.
- For each Subsystem used by your AFU, create individual channel interfaces using your selected bus protocols and connect the channel PIM Shims based on selected bus protocols.
- PCIe - Host Channel
- Local Memory
- HSSI
- Tie off all unused channels/ports.
- Connect the channel interfaces to the AFU module.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#31-top-level-module-ofs_plaf_afu","title":"3.1. Top Level Module - ofs_plaf_afu","text":"For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\n
The SystemVerilog interface ofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
- AVMM
- AVMM-RDWR
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n
The Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
- AVMM
- AXI-Lite
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#322-connect-the-host-channel-to-the-pim-shim","title":"3.2.2. Connect the host channel to the PIM Shim","text":"Using the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#323-avalon-example","title":"3.2.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#33-local-memory","title":"3.3. Local Memory","text":"Local memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
- AVMM
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#332-connect-local-memory-to-the-pim-shim","title":"3.3.2. Connect local memory to the PIM Shim","text":"Using the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#333-avalon-example","title":"3.3.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#34-high-speed-serial-interface-hssi","title":"3.4. High Speed Serial Interface (HSSI)","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#35-tie-off-unused-ports","title":"3.5. Tie Off Unused ports","text":"In digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#36-afu-instantiation","title":"3.6. AFU Instantiation","text":"Instantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#4-afu","title":"4. AFU","text":"The AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#42-afu-interfaces","title":"4.2. AFU Interfaces","text":"The AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\n
The Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#43-csr-interface","title":"4.3. CSR Interface","text":"The MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#46-systemverilog-packages","title":"4.6. SystemVerilog Packages","text":"The AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
- Host Channel Package: ofs_plat_host_chan_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv - Local Memory Package: local_mem_cfg_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.sv
The HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
- HSSI Channel Package: ofs_fim_eth_if_pkg
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#5-host-software-development","title":"5. Host Software Development","text":"The host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\n
- power - Accelerator Function power consumption, in watts. Set to 0 for Intel ADP platforms.
- clock-frequency-high - Clock frequency for uclk_usr in MHz. (optional)
- clock-frequency-low - Clock frequency for uclk_usr_div2 in MHz. (optional)
- afu-top-interface:
- class : Set to \"ofs_plat_afu\" for PIM based AFU, \"afu_main\" for native/hybrid AFU's.
- accelerator-clusters:
- name : name of AFU
- total-contexts : Set to '1'
- accelerator-type-uuid : 128-bit Universally Unique Identifier (UUID) used to identify the AFU.
The ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\n
The Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#62-source-list-file","title":"6.2. Source List File","text":"The source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#63-directory-structure","title":"6.3. Directory Structure","text":"Below is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\n
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/ug_dev_pim_based_afu/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/afu_dev/ug_dev_pim_based_afu/images/readme/","title":"Readme","text":"images
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/","title":"oneAPI Accelerator Support Package (ASP): Getting Started User Guide","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#10-introduction","title":"1.0 Introduction","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#11-about-this-document","title":"1.1 About This Document","text":"This document serves as a quick start guide for setting up Intel\u00ae oneAPI Base Toolkit (Base Kit) on Open FPGA Stack (OFS) using oneapi-asp repository. Please see Table 1-1 for OFS reference platforms targeted in this guide.
Table 1-1 HLD Tools
Target Device for the oneAPI ASP Target Platform for the oneAPI ASP Agilex\u00ae 7 FPGA Intel\u00ae FPGA SmartNIC N6001-PL Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) Agilex\u00ae 7 FPGA Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) Stratix\u00ae 10 FPGA Intel\u00ae FPGA PAC D5005 Attention: Intel is discontinuing the Intel FPGA SDK for OpenCL software product. Refer to the Product Discontinuation Notice PDN2219. Alternatively, Intel recommends using the Intel\u00ae oneAPI Base Toolkit (Base Kit), which provides core tools and libraries for developing high-performance data-centric applications across diverse architectures. It features an industry-leading C++ compiler that implements SYCL*, an evolution of C++ for heterogeneous computing. For more information, refer to the Intel\u00ae oneAPI Base Toolkit (Base Kit) web page. To migrate your OpenCL FPGA designs to SYCL, review Migrating OpenCL\u2122 FPGA Designs to SYCL* that demonstrates important differences between OpenCL and SYCL for FPGA and provides steps to migrate your OpenCL designs.
After reviewing the document you will be able to:
- Setup your host machine to develop HLD AFUs
- Compile and run sample HLD applications on OFS
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#12-terminology","title":"1.2 Terminology","text":"This table defines some of the common terms used when discussing OFS.
Table 1-2: Terminology
Term Abbreviation Description Open FPGA Stack OFS A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. Accelerator Functional Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. FPGA Interface Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. High Level Design HLD For the purpose of this guide, this term refers to designing with Intel\u00ae oneAPI Base Toolkit (Base Kit). oneAPI Accelerator Support Package oneAPI ASP OR oneapi-asp A collection of hardware and software components that enable oneAPI kernels to communicate with oneAPI runtime as well as with OFS components. The hardware components of the oneAPI ASP along with the kernels lie in the AFU region. Open Programmable Acceleration Engine Software Development Kit OPAE SDK A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. Platform Interface Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Device Feature List DFL A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. Best Known Configuration BKC The exact hardware configuration Intel has optimized and validated the solution against. SYCL - SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL\u2122) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Installable Client Driver ICD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports the OpenCL ICD extension from the Khronos Group\u2122. The OpenCL ICD extension allows you to have multiple OpenCL implementations on your system. With the OpenCL ICD Loader Library, you may choose from a list of installed platforms and execute OpenCL API calls that are specific to your OpenCL implementation of choice. FPGA Client Driver FCD Intel\u00ae FPGA Runtime for OpenCL\u2122 Software Technology supports FPGA Client Driver(FCD) extension. FCD allows the runtime to automatically find and load the oneAPI ASP libraries at host run time"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#13-introduction-to-high-level-design-on-ofs","title":"1.3 Introduction to High Level Design on OFS","text":"Intel currently provides Intel\u00ae oneAPI Base Toolkit (Base Kit) for FPGA application development using high level languages like Data Parallel C++(DPC++).
Figure 1-1 shows how OFS components can be used with Intel HLD tool.
Figure 1-1 HLD Tool on OFS Platforms
For high level description and setup details for OFS components shown in figure above, please refer to the Getting Started guide for your target device.
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL).
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)).
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)).
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs.
For a more detailed diagram and more information about the FPGA Interface Manager(FIM) shown in figure above, please refer to the FIM developer guides for your target device.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs.
- Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
The oneAPI ASP is required for compiling and running HLD application kernel on OFS platforms using Intel oneAPI. It is a collection of hardware and software components that enable oneAPI kernels to communicate with oneAPI runtime as well as with other OFS components. The hardware components of the oneAPI ASP along with the kernel lie in the AFU region shown in the figure above. For more details about the components of the oneAPI ASP, please refer to oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
Figure 1-2 shows the setup steps to use oneAPI base toolkit on OFS platforms.
Figure 1-2 Setup Steps for oneAPI base toolkit on OFS Platforms
The next section covers the setup steps in detail.
Note: Administrative privileges are needed for multiple setup steps, ensure you have administrative/sudo privileges before proceeding.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#20-setup-flow-for-using-hld-tool-on-ofs","title":"2.0 Setup Flow for Using HLD Tool on OFS","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#21-setup-server-for-ofs","title":"2.1 Setup Server for OFS","text":"As a first step, the server or host machine being used for developing HLD application needs to be setup for OFS. This involves setting up the FPGA card as well as installing OFS software stack including OPAE SDK and OFS DFL kernel driver.
Please follow steps in Software Installation Guide: OFS for PCIe Attach FPGAs to setup Linux DFL kernel driver and install OPAE SDK.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#22-clone-and-compile-fim","title":"2.2 Clone and Compile FIM","text":"As shown in Figure 1-1, OFS components in the FPGA include the FIM and Accelerator Functional Unit(AFU). The oneAPI ASP is in the Partial Reconfiguration(PR) region of the AFU and relies on the compiled database of the static region(FIM) to interface with the host and board peripherals(e.g. on-board memory).
Once the server is setup with OPAE SDK and DFL kernel driver, the next step is to clone and compile the static region of the design, i.e. FIM. You can use the default configuration of the FIM for all target platforms. Additionaly for Agilex\u00ae 7 FPGA you have the option to create 2 different types of FIM with different PCIe configurations, 1PF/1VF and 2PF (built using an ofss file provided, see table below for more information). Furthermore, for Intel\u00ae FPGA SmartNIC N6001-PL and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) platforms you can create a minimal FIM which removes the HSSI subsystem and host exercisers in the design and expand the AFU PR region with the area previously used by the removed components, follow the corresponding Shell Developer Guide for more information. The difference between this two PCIe configurations, are the amount of VFs that must be created, in case of 1PF/1VF , only 1 VF is needed. In case of 2PF FIM no VF creation is required. 2PF FIM could be used in FPGA development in virtual machines, see Section 3.0 OneAPI on OFS Running in a Virtual Machine. Please see Table 2-1 below for more information.
Table 2-1 Agilex\u00ae 7 FPGA FIM Configurations
Target Platform FIM 1PF/1VF configuration FIM 2PF configuration Intel\u00ae FPGA SmartNIC N6001-PL pcie_host_1pf_1vf.ofss pcie_host_2pf.ofss Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) pcie_host_1pf_1vf.ofss pcie_host_2pf.ofss Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) pcie_host_2link_1pf_1vf.ofss no ofss provided Note: For steps to build the 2PF FIM refer to Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs Please follow steps in the Shell Developer Guides for your target device to compile FIM supporting PR release.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs.
- Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
A pr_build_template directory will be generated in the work directory specified as part of the FIM compile command (using OFS/ofs-common/scripts/common/syn/build_top.sh script with the '-p' option enable to create an out-of-tree PR release). The pr_build_template directory is required for successful setup of the oneAPI ASP.
Once the FIM compile is complete, please program FIM using fpgasupdate and Remote System Update(rsu) command for Intel\u00ae FPGA SmartNIC N6001-PL and Intel\u00ae FPGA PAC D5005. Use of these commands has been demonstrated in 4.6.3 Updating with fpgasupdate and 4.6.5 Loading Images with rsu sections in Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL), refer to sections 5.2.1 fpgasupdate and 5.2.3 rsu in Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs for Stratix\u00ae 10 FPGA. In case of Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) platforms, program the FIM with a SOF image via JTAG, refer to Chapter 5 in Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs and Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs guides for more information.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#23-prerequisites","title":"2.3 Prerequisites","text":"In addition to server setup and FIM compilation, a few linux packages are needed to setup the oneAPI ASP and develop HLD applications.
1) Install the following packages:
sudo dnf install numactl-devel ncurses-compat-libs\n
2) Ensure that IOMMU is turned on as explained in section 3.2 Building and Installing the OFS DFL Kernel Drivers from Source in Software Installation Guide: OFS for PCIe Attach FPGAs.
You can verify this setting using cat /proc/cmdline command. The output must have intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200.
$ cat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-6.1.78-dfl root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\n
3) Install HLD development software. Please see Table 2-2 below for download link. Install the latest version. Use sudo privileges to do the installation.
Table 2-2 Intel HLD Tool and Download Information
HLD Tool Target Platform for the oneAPI ASP Tool Download Information Intel\u00ae oneAPI Base Toolkit (Base Kit) - Intel\u00ae FPGA SmartNIC N6001-PL
- Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
- Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
- Intel\u00ae FPGA PAC D5005
Download here Note: Install the oneAPI compiler patches required, according to the version you are using, you can download them in the patch history section from the FPGA Support Package for Intel\u00ae oneAPI DPC++/C++ Compiler page.
Tool installation guide for your reference:
- Intel\u00ae oneAPI Toolkits Installation Guide for Linux* OS
4) Ensure you have all the Quartus patches installed, refer to Table 2-4 and 2-5 for required Quartus version.
Note: For Agilex\u00ae 7 FPGA ensure Quartus patches 0.18, 0.26 and 0.02iofs are installed. You can find patches 0.18 and 0.26 in a tar file under assets and Quartus Prime Pro License File 0.02iofs under the Summary section in the following link patch-agx7-2024-2-1.tar.gz. For Stratix\u00ae 10 FPGA ensure Quartus patch 0.01iofs is installed. You can find it in the following link 0.01iofs. For quartus patches installation to work properly, you must have Git Large File Storage (LFS) installed when cloning the ofs-fim repository.
Use following command to check Quartus version and installed patches.
quartus_sh -v\n
5) After completing the tool installation, set the following environment variables required to execute build scripts successfully:
# Adding Quartus to PATH\nexport PATH=$PATH:path-to-quartus-installation-dir/bin\n export QUARTUS_ROOTDIR=path-to-quartus-installation-dir/quartus\n export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n# Other OFS environment variables\nexport OFS_ROOTDIR=path-to-directory-containing-cloned-ofs-fim-repo/#ofs-agx7-pcie-attach for Agilex\u00ae 7 FPGA or ofs-d5005 for Stratix\u00ae 10 FPGA\n export WORKDIR=$OFS_ROOTDIR\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n export OPAE_SDK_REPO_BRANCH=release/branch-tag # Refer to Table 2-4 and 2-5, for OPAE SDK branch tag\nexport OFS_PLATFORM_AFU_BBB=path-to-cloned-ofs-platform-afu-bbb-repo\n export OPAE_PLATFORM_ROOT=path-to-ofs-fim-pr_build_template-directory # (see section 2.2 for more details)\nexport OFS_ASP_ROOT=path-to-directory-containing-oneapi-asp/oneapi-asp/platform-name #platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005. \nexport LIBOPAE_C_ROOT=/usr # (OPAE libraries are installed in /usr/lib64 by default if you followed the OPAE SDK steps covered in section 2.1 as is and installed OPAE rpm packages. If you have a different OPAE installation path, please point LIBOPAE_C_ROOT to your OPAE installation location that you specified using -DCMAKE_INSTALL_PREFIX=installation-path in cmake command for building OPAE)\n
Note: To re-use this environment setting, you can copy the above export statements to a shell script, update the paths to match your tool installations and source this script each time a new shell is started.
6) Source initialization script for oneAPI, path is shown in table below.
Table 2-3 Initialization Script for HLD tool
Tool Command to source initialization script Intel\u00ae oneAPI Base Toolkit (Base Kit) source path-to-intel-oneapi-toolkit-installation-directory/setvars.sh Once the environment variables are set, you can check the tool version using the following commands:
# Printing all (Quartus, Intel\u00ae oneAPI Base Toolkit, GCC) versions for user info\nquartus_sh -v\nicpx --version (for oneAPI Base Toolkit)\ngcc --version\n
Table 2-4 and 2-5 summarize the tool version/Best Known Configurations(BKC).
Table 2-4 Best Known Configuration(BKC) for Agilex\u00ae 7 FPGA OFS
Component/Tool Version FPGA Platform Intel\u00ae FPGA SmartNIC N6001-PL Operating System RHEL 8.8 , Kernel : 4.18.0-dfl linux-dfl (DFL Kernel Driver) Tag: intel-1.11.0-2 opae-sdk Branch: release/2.13.0, Tag: 2.13.0-3 ofs-fim Tag: ofs-2024.2-1 oneapi-asp Tag: ofs-2024.2-2 > Note: Cloning and build of this repo is discussed in the section 2.4 Quartus Prime Pro Edition Version 24.1 Pro Edition with patches (0.18, 0.26 and 0.02iofs) under assets and summary sections in this link patch-agx7-2024-2-1.tar.gz Intel\u00ae oneAPI Base Toolkit (Base Kit) Latest version > Note: Ensure to install the oneAPI compiler patches required, according to the version you are using, you can download them in the patch history section from the FPGA Support Package for Intel\u00ae oneAPI DPC++/C++ Compiler page GCC 8.5.0 cmake 3.15 Table 2-5 Best Known Configuration(BKC) for Stratix\u00ae 10 FPGA
Component/Tool Version FPGA Platform Intel\u00ae FPGA PAC D5005 Operating System RHEL 8.6 , Kernel : 6.1.78-dfl linux-dfl (DFL Kernel Driver) Tag: ofs-2024.1-6.1-2 opae-sdk Branch: release/2.12.0, Tag: 2.12.0-5 ofs-fim Tag: ofs-2024.1-1 oneapi-asp Tag: ofs-2024.2-2 > Note: Cloning and build of this repo is discussed in the section 2.4 Quartus Prime Pro Edition Version 23.4 Pro Edition with patch 0.01iofs Intel\u00ae oneAPI Base Toolkit (Base Kit) Latest version > Note: Ensure to install the oneAPI compiler patches required, according to the version you are using, you can download them in the patch history section from the FPGA Support Package for Intel\u00ae oneAPI DPC++/C++ Compiler page GCC 8.5.0 cmake 3.15"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#24-build-and-install-oneapi-asp","title":"2.4 Build and Install oneapi-asp","text":"Once all pre-requisites are installed and the environment variables are set, next step is to clone and build oneapi-asp.From ofs-2024.2-2 tag you have the option to use an editor tool for an easy parametrization of the oneAPI Accelerator Support Pacakges in OFS based platforms using the IP Parameter Editor GUI in Quartus Prime, you can generate the ASP and select only the board variants you want to include using the preset files provided, see appendix A for more information about the oneAPI ASP editor. This section covers the build of the oneAPI ASP with the default values, including all the board variants for the corresponding OFS platform.
1) Clone oneapi-asp repository and checkout tag matching the BKC for your target platform (see Tables 2-4 and 2-5 for the BKCs).
Note: You will need a personal access token (use the classic mode) to be used as the password to clone successfully. Please see more information about token authentication requirements for Git operations here.
git clone https://github.com/OFS/oneapi-asp.git\n cd oneapi-asp\n git checkout tags/ofs-2024.2-2\n
Ensure the correct tag has ben checked out:
git describe --tags\n
- Output:
ofs-2024.2-2\n
2) Ensure that OPAE_PLATFORM_ROOT and LIBOPAE_C_ROOT have been set as described in section 2.3. Generate the oneAPI ASP hardware and software using provided build-asp.sh script. This script clones required repositories and builds the oneAPI ASP libraries required by HLD host application to run successfully.
cd path-to-directory-containing-cloned-oneapi-asp-repo/oneapi-asp/platform-name\n ./scripts/build-asp.sh\n
The generated directory structure is shown below. For more details refer to the oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
\noneapi-asp/platform-name\n|--ase/\n| |--base/\n| |--hack_ip_files/\n| |--compile-kernel.sh\n| |--run-ase.sh\n| |--setup.sh\n| |--simulate-aocx.sh\n|--bringup\n| |--binaries/\n| |--source/simple-add-buffers/\n| | |--simple-add-buffers.cpp\n|--build/\n| |--bringup/\n| |--release/\n|--hardware/\n| |--common/build/\n| |--ofs_platform-name/\n| |--ofs_platform-name_iopipes/\n| |--ofs_platform-name_usm/\n| |--ofs_platform-name_usm_iopipes/\n|--linux64/\n| |--doc/\n| |--include/\n| |--lib/\n| | |--libintel_opae_mmd.so\n| | |--libMPF-cxx.so\n| | |--libMPF.so\n| |--libexec/\n| | |--diagnose\n| | |--flash\n| | |--initialize\n| | |--install\n| | |--program\n| | |--setup_permissions.sh\n| | |--uninstall\n|--scripts/\n| |--build-asp.sh\n| |--build-asp-sw.sh\n| |--build-default-binaries.sh\n| |--build-mmd.sh\n| |--build-opae.sh\n| |--create-tarball.sh \n| |--dedup-hardware.sh\n| |--README.txt\n| |--setup-asp.py\n|--board_env.xml\n|--README.md\n
3) Once the oneAPI ASP is generated, add the following to LD_LIBRARY_PATH. You can add it to your script for setting environment variables (if you created one as noted in step 5 in section 2.3)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:path-to-oneapi-asp/platform-name/linux64/lib\n
4) Check if FPGA Client Driver(FCD) exists for any other version of oneAPI ASP or for any other board. You can check with aocl list-devices command. It is recommended to run aocl list-devices as root user (login with sudo su) to see all installed ASPs on the host.
A sample output when a oneAPI ASP FCD is installed is shown below:
--------------------------------------------------------------------\nDevice Name:\nacl0\n\nBSP Install Location:\n/home/ofsuser/oneapi-asp/platform-name\n\nVendor: Intel Corp\n\nPhysical Dev Name Status Information\n\nofs_ee00000 Uninitialized PR slot function not configured \n Need to follow instructions to bind vfio-pci driver to PR slot function\n\nASP DIAGNOSTIC_PASSED\n--------------------------------------------------------------------\n
If a oneAPI ASP/BSP is installed, uninstall using aocl uninstall path-to-oneapi-asp-install-location, where path-to-oneapi-asp-install-location is provided under BSP Install Location: in the output of aocl list-devices. If you are prompted with a question to unset the FCD, type Y. If you are prompted with a question to remove oneAPI-ASP configuration settings, type Y.
Sample output for aocl uninstall command:
$ aocl uninstall /home/ofsuser/oneapi-asp/platform-name\naocl uninstall: Removing the FPGA Client Driver (FCD) from the system\n[sudo] password for ofsuser:\naocl uninstall: Removing the board package /home/ofsuser/oneapi-asp/platform-name from the list of installed packages. This process may require admin privilege\naocl uninstall: Running uninstall from /home/ofsuser/oneapi-asp/platform-name/linux64/libexec\nDo you want to remove oneAPI-ASP configuration settings [Y/n] Y\nDeleting OPAE config files\nRemoving configuration files\nOFS oneAPI-ASP uninstall complete\n
5) Install FPGA Client Driver(FCD) file for the oneAPI ASP using aocl install path-to-oneapi-asp/platform-name command as shown below. The host program uses FCD to find and link to the platform Memory Mapped Device (MMD) library. For more information about MMD library, refer to oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
aocl install path-to-directory-containing-oneapi-asp/oneapi-asp/platform-name\n
Notes: 1. Type Y when prompted to setup FCD at /opt/Intel/OpenCLFPGA/oneAPI/Boards (default location for Intel oneAPI).
Sample output aocl install command in Intel\u00ae oneAPI Base Toolkit (Base Kit) environment is shown below.
aocl install: Setting up the FPGA Client Driver (FCD) to the system. This process may require admin privilege\nInstall the FCD file to /opt/Intel/OpenCLFPGA/oneAPI/Boards\n[sudo] password for ofsuser:\naocl install: Adding the board package path-to-oneapi-asp/platform-name to the list of installed packages\nInstalling the board package driver to the system.\naocl install: Running install from path-to-oneapi-asp/platform-name/linux64/libexec\nConfiguring locked memory setting\nConfiguring udev rules for DFL FPGA device permission\nConfiguring system with 1024 2M hugepages\nSetting access permisions of /dev/uio to 666\nFinished setup_permissions.sh script. All configuration settings are persistent.\nIntel OFS oneAPI ASP install complete.\nRun 'aocl diagnose' to list devices or 'aocl initialize <dev_name> <board_variant> to load default image\n
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#25-board-initialization","title":"2.5 Board Initialization","text":"OFS software stack expects boards to be initialized with a bitstream for the board variant intended to be used for development. An oneAPI sample application, named simple-add-buffers, has been provided in the oneapi-asp repository for generating initialization bitstreams for included board variants. The sample is located in oneapi-asp/platform-name/bringup/source.
oneapi-asp has four board variants for Intel\u00ae FPGA SmartNIC N6001-PL and two board variants for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and Intel\u00ae FPGA PAC D5005 (oneapi-asp/platform-name/hardware has the hardware design files for these). For more details on the architecture of the board variants, please refer to the oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
Table 2-6 oneAPI Sample Applications
Board Variants Sample Application - ofs_platform-name
- ofs_platform-name_usm
- ofs_n6001_iopipes
- ofs_n6001_usm_iopipes
simple-add-buffers Note: platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005.
All samples are located in oneapi-asp/platform-name/bringup/source.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#251-compile-initialization-bitstreams","title":"2.5.1 Compile Initialization Bitstreams","text":"A script is provided in repo to compile simple-add-buffers oneAPI sample application. The script is oneapi-asp/platform-name/scripts/build-default-binaries.sh.
Script usage is as follows:
./build-default-binaries.sh -b name-of-board-variant\n
Note: name-of-board-variant can be ofs_platform-name, ofs_platform-name_usm, ofs_n6001_iopipes or ofs_n6001_usm_iopipes where platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005. Refer to Table 2-6 for board variants available for each target platform. Compilation will take a few hours to complete.
The output directory of the sample application is written to oneapi-asp/platform-name/build/bringup. The generated bitstreams are also copied to oneapi-asp/platform-name/bringup/binaries/. These are used in the initialization of the platform.
Once the bitstreams are generated, create a VF and initialize the board as explained in following section. Ensure that the FIM has been programmed on the board as explained in section 2.2 Clone and Compile FIM
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#252-pfvf-mapping","title":"2.5.2 PF/VF mapping","text":"The oneAPI ASP is located in the PR region of the FIM and is accessed through PF/VF Mux. Refer to the FIM Reference Manual for your target platforms for more details about PF/VF mapping.
- Reference FIM for Agilex\u00ae 7 FPGA OFS:
For Base_x16 FIM (default) and 1PF/1VF FIM (refer to table 2-1 for the ofss file needed to build this FIM configuration), VF0 is mapped to PR region and you can create 1 VF when using this FIM. Base_x16 FIM has 5 PF's.
For 2PF FIM (built using pcie_host_2pf.ofss) PF1 is mapped to PR region and no VF creation is required.
See Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for diagram showing PF/VF mapping.
Note: See Table 2-1 for available FIM configurations for Intel\u00ae FPGA SmartNIC N6001-PL, Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) platforms.
- Reference FIM for Stratix\u00ae 10 FPGA OFS:
VF1 is mapped to PR region and you must create 2 VFs when using this FIM. This FIM has 1 PF. See Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs for diagram showing PF/VF mapping.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#2521-create-vf","title":"2.5.2.1 Create VF","text":"Note: This section only applies for Base_x16 FIM and 1PF/1VF FIM for Agilex\u00ae 7 FPGA and default FIM for Stratix\u00ae 10 FPGA.
- Create a VF using PCIe ID obtained from the output of
fpgainfo fme (PCIe s\\:b\\:d.f output)
sudo pci_device s:b:d.f vf num_vf #num_vf is 1 for Agilex\u00ae 7 FPGA and 2 for Stratix\u00ae 10 FPGA\n
- Check that the VF is created using
sudo opae.io ls command and note the PCIe ID for the VF(s) (the function number in s\\:b\\:d.f will be different for the VF). Sample output for Agilex\u00ae 7 FPGA 1PF/1VF FIM is shown below. Output for base_x16 FIM should display 5 PF's and the PCIe ID for VF0 will be s:b:d.5.
$ sudo opae.io ls\n [0000:b1:00.0] (0x8086, 0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086, 0xbccf) Intel Acceleration Development Platform N6001 (Driver: None)\n
Sample output for Stratix\u00ae 10 FPGA is shown below.
$ sudo opae.io ls\n [0000:d8:00.0] (0x8086, 0xbcce) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\n [0000:d8:00.1] (0x8086, 0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\n [0000:d8:00.2] (0x8086, 0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\n
Note:sudo opae.io ls will list the accelerators, respective PCIe ID as well as the driver it is currently bound to.
Note: For more information about pci_device and opae.io utilities, refer to the OPAE FPGA tools page here.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#2522-bind-pf-and-vf","title":"2.5.2.2 Bind PF and VF","text":"For Base_x16 FIM and 1PF/1VF FIM for Agilex\u00ae 7 FPGA and default FIM for Stratix\u00ae 10 FPGA :
- Bind the created VF(s) to vfio-pci driver, use the PCIe ID for the VF(s) for this step. Verify you are using the PCIe ID of the VFs you have created. For example:
- From sample output for Agilex OFS target platform having 1PF/1VF FIM programmed shown in Section 2.5.2.1 Create VF,
s:b:d.vf will be 0000:b1:00.1 in command below. For base_x16 FIM should be s:b:d.5.
sudo opae.io init -d s:b:d.vf $USER\n
- Sample output for Agilex\u00ae 7 FPGA OFS target platform 1PF/1VF FIM. Output for base_x16 FIM should be similar.
$ sudo opae.io init -d 0000:b1:00.1 $USER\n Unbinding (0x8086,0xbccf) at 0000:b1:00.1 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:b1:00.1 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:b1:00.1 is 319\n Assigning /dev/vfio/319 to ofsuser\n Changing permissions for /dev/vfio/319 to rw-rw----\n\n $ sudo opae.io ls\n [0000:b1:00.0] (0x8086, 0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086, 0xbccf) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n\n $ ls -lt /dev/vfio\n total 0\n crw-rw----. 1 ofsuser root 509, 0 Dec 3 20:41 319\n crw-rw-rw-. 1 root root 10, 196 Dec 3 20:41 vfio\n
- Sample output for Stratix\u00ae 10 FPGA OFS target platform:
$sudo opae.io init -d 0000:12:00.1 $USER\n Unbinding (0x8086,0xbccf) at 0000:12:00.1 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:12:00.1 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:12:00.1 is 149\n Assigning /dev/vfio/149 to ofsuser\n Changing permissions for /dev/vfio/149 to rw-rw----\n\n $sudo opae.io init -d 0000:12:00.2 $USER\n Unbinding (0x8086,0xbccf) at 0000:12:00.2 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:12:00.2 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:12:00.2 is 152\n Assigning /dev/vfio/152 to ofsuser\n Changing permissions for /dev/vfio/152 to rw-rw----\n\n $ sudo opae.io ls\n [0000:12:00.0] (0x8086:0xbcce) Intel FPGA Programmable Acceleration Card D5005 (Driver: dfl-pci)\n [0000:12:00.1] (0x8086:0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: vfio-pci)\n [0000:12:00.2] (0x8086:0xbccf) Intel FPGA Programmable Acceleration Card D5005 (Driver: vfio-pci)\n\n $ls -lt /dev/vfio\n total 0\n crw-rw----. 1 ofsuser root 235, 3 Dec 3 16:25 149\n crw-rw----. 1 ofsuser root 235, 0 Dec 3 16:22 152\n crw-rw-rw-. 1 root root 10, 196 Dec 1 07:28 vfio\n
For 2PF FIM for Agilex\u00ae 7 FPGA :
Bind the PF1 to vfio-pci driver, use sudo opae.io ls command and note the PCIe ID (s\\:b\\:d.f) for the PF(s). Verify you are using the PCIe ID of the PF1.
$ sudo opae.io ls\n [0000:b1:00.0] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
Output below shows the command to bind PF1, in this case s:b:d.f will be 0000:b1:00.1.
sudo opae.io init -d s:b:d.f $USER\n
- Sample output for Agilex\u00ae 7 FPGA OFS target platform 2PF FIM.
$ sudo opae.io init -d 0000:b1:00.1 $USER\n Unbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\n Binding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\n iommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 13\n Assigning /dev/vfio/13 to ofsuser\n Changing permissions for /dev/vfio/13 to rw-rw----\n\n $ sudo opae.io ls\n [0000:b1:00.0] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n [0000:b1:00.1] (0x8086:0xbcce) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n\n $ ls -lt /dev/vfio\n total 0\n crw-rw----. 1 ofsuser root 511, 0 Feb 2 22:47 13\n crw-rw-rw-. 1 root root 10, 196 Feb 2 16:56 vfio\n
If the driver fails to bind due to an error related to iommu_group (e.g. `No such file or directory: '/sys/bus/pci/devices/0000:b1:00.5/iommu_group'), ensure IOMMU is turned on as explained in step 2 in Section 2.3 Prerequisites.
Note: For more information about opae.io utilities, refer to the OPAE FPGA tools page here.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#253-initialize-board-and-run-diagnostic-test","title":"2.5.3 Initialize Board and Run Diagnostic Test","text":"Before initializing the board set the value of 2048 to OFS_ASP_ENV_NUM_HUGEPAGES env variable to configure 2048 2M hugepages:
export OFS_ASP_ENV_NUM_HUGEPAGES=2048\n
- Initialize the board with default bitstream using
aocl initialize command
aocl initialize device-name name-of-board-variant\n
Note: device-name in aocl initialize command is from the ASP Diagnostics section in the output of aocl list-devices (sample output is shown in Section 2.4). device-name is acl0 if there is only 1 board connected to the server. name-of-the-board-variant can be one of the supported board variants listed in Table 2-6 provided the sample application bitstream using steps in section above.
Sample output for aocl initialize acl0 ofs_n6001 is shown below. Output for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and Intel\u00ae FPGA PAC D5005 platforms and other board variants should be similar.
$ aocl initialize acl0 ofs_n6001\naocl initialize: Running initialize from path-to-oneapi-asp/n6001/linux64/libexec\nInitializing with default ASP binary ofs_n6001.fpga\nSaving target image to \"ofs_n6001.aocx.0\"\nConfiguring locked memory setting\n[sudo] password for $USER:\nConfiguring udev rules for DFL FPGA device permission\nConfiguring system with 1024 2M hugepages\nSetting access permisions of /dev/uio to 666\nFinished setup_permissions.sh script. All configuration settings are persistent.\nProgram succeed.\n
Notes: 1. aocl initialize command needs to be executed only one time unless you reboot the server.
Run aocl diagnose to check that the board Status under BSP Diagnostics is equal to Passed.
platform-name in command output below is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005.
$ aocl diagnose\n--------------------------------------------------------------------\nICD System Diagnostics\n--------------------------------------------------------------------\n\nUsing the following location for ICD installation:\n /etc/OpenCL/vendors\n\nFound 3 icd entry at that location:\n /etc/OpenCL/vendors/intel64.icd\n /etc/OpenCL/vendors/Altera.icd\n /etc/OpenCL/vendors/Intel_FPGA_SSG_Emulator.icd\n\n\nThe following OpenCL libraries are referenced in the icd files:\n /opt/intel/oneapi/compiler/latest/lib/libintelocl.so\n\nChecking LD_LIBRARY_PATH for registered libraries:\n /opt/intel/oneapi/compiler/latest/lib/libintelocl.so was registered on the system.\n /opt/intel/oneapi/compiler/latest/opt/oclfpga/host/linux64/lib/libalteracl.so\n\n\nChecking LD_LIBRARY_PATH for registered libraries:\n /opt/intel/oneapi/compiler/latest/linux/lib/oclfpga/host/linux64/lib/libalteracl.so was registered on the system.\n /opt/intel/oneapi/compiler/latest/linux/lib/x64/libintelocl_emu.so\n\nChecking LD_LIBRARY_PATH for registered libraries:\n /opt/intel/oneapi/compiler/latest/linux/lib/x64/libintelocl_emu.so was registered on the system.\n\nUsing OCL_ICD_FILENAMES to search for ICD clients, it is set to libintelocl_emu.so:libalteracl.so:/opt/intel/oneapi/compiler/2024.0/lib/libintelocl.so\n\nChecking LD_LIBRARY_PATH for registered libraries specified by OCL_ICD_FILENAMES\n libintelocl_emu.so was registered on the system at \n /opt/intel/oneapi/compiler/2024.0/lib\n libalteracl.so was registered on the system at \n /opt/intel/oneapi/compiler/2024.0/opt/oclfpga/host/linux64/lib\n /opt/intel/oneapi/compiler/2024.0/lib/libintelocl.so was registered on the system.\n\nUsing the following location for fcd installations:\n /opt/Intel/OpenCLFPGA/oneAPI/Boards\n\nFound 1 fcd entry at that location:\n /opt/Intel/OpenCLFPGA/oneAPI/Boards/platform-name.fcd\n\nThe following OpenCL libraries are referenced in the fcd files:\n libopae-c.so\n/home/ofsuser/oneapi-asp/platform-name/linux64/lib/libMPF.so\n/home/ofsuser/oneapi-asp/platform-name/linux64/lib/libintel_opae_mmd.so\n\nChecking LD_LIBRARY_PATH for registered libraries:\n libopae-c.so was registered on the system at /usr/lib64\n /home/ofsuser/oneapi-asp/platform-name/linux64/lib/libMPF.so was registered on the system.\n /home/ofsuser/oneapi-asp/platform-name/linux64/lib/libintel_opae_mmd.so was registered on the system.\n\nNumber of Platforms = 4\n 1. Intel(R) FPGA Emulation Platform for OpenCL(TM) | Intel(R) Corporation | OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3\n 2. Intel(R) FPGA SDK for OpenCL(TM) | Intel(R) Corporation | OpenCL 1.0 Intel(R) FPGA SDK for OpenCL(TM), Version 2024.0\n 3. Intel(R) OpenCL | Intel(R) Corporation | OpenCL 3.0 LINUX\n 4. Intel(R) FPGA SDK for OpenCL(TM) | Intel(R) Corporation | OpenCL 1.0 Intel(R) FPGA SDK for OpenCL(TM), Version 2024.0\n--------------------------------------------------------------------\nICD diagnostics PASSED\n--------------------------------------------------------------------\n--------------------------------------------------------------------\nBSP Diagnostics\n--------------------------------------------------------------------\n--------------------------------------------------------------------\nDevice Name:\nacl0\n\nBSP Install Location:\n/home/ofsuser/oneapi-asp/platform-name\n\nVendor: Intel Corp\n\nPhysical Dev Name Status Information\n\nofs_ee00001 Passed Intel OFS Platform (ofs_ee00001)\n PCIe b1:00.0\n FPGA temperature = 59 degrees C.\n\nASP DIAGNOSTIC_PASSED\n--------------------------------------------------------------------\n\nCall \"aocl diagnose <device-names>\" to run diagnose for specified devices\nCall \"aocl diagnose all\" to run diagnose for all devices\n
Run complete diagnostic using aocl diagnose device-name command. device-name is acl0 if you have only 1 board in the server. The test reads and write data to the board to check the interfaces function correctly and report the measured bandwidth. The test must show ASP DIAGNOSTIC_PASSED message at the end.
aocl diagnose acl0\n
Next section you will build and run oneAPI host applications.
Once you are done with your application testing, you can release the device from vfio-pci driver, the steps for this are provided in Section 2.7 Release VF.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#26-compile-and-run-oneapi-sample-applications","title":"2.6 Compile and Run oneAPI Sample Applications","text":"Different code samples are provided for testing different board interfaces. Please refer to table below for more information.
Table 2-7 DPC++ Sample Application
Board Variants Sample Application Description Link to Samples Repository Location of the Sample Samples Repository Tag - ofs_platform-name
- ofs_platform-name_usm
board_test This sample measures kernel clock frequency, kernel launch latency and tests the different interfaces required by the oneAPI kernel to function correctly (e.g. host to kernel interface, kernel to EMIF as well as host to EMIF). oneAPI-samples path-to-oneAPI-samples/oneAPI-samples/DirectProgramming/C++SYCL_FPGA/ReferenceDesigns/board_test oneAPI compiler version you are using, this sample supports usm board variant from 2024.0 oneAPI version. - ofs_n6001_iopipes
- ofs_n6001_usm_iopipes
io_streaming_one_pipe An FPGA code sample to perform a loopback test using SYCL* input/output (I/O) pipes to stream data through the FPGA I/O. examples-afu path-to-examples-afu/examples-afu/oneapi-samples/io_streaming_one_pipe ofs-2024.2-1 First clone the repository and then checkout to the corresponding tag.
git clone link-to-samples-repository\n cd oneAPI-samples # (oneapi-samples for examples-afu repository)\n git checkout tags/samples-repository-tag\n
Note: For link-to-samples-repository and samples-repository-tag refer to the table 2-7. To check your oneAPI compiler version use the command:
\n icpx --version\n
Follow steps below to compile and run oneAPI board_test and io_streaming_one_pipe binaries. Use -DFPGA_DEVICE in cmake command to provide path to the oneAPI ASP and name of board variant being compiled. For board_test when targeting USM board variant add -DSUPPORTS_USM=1 flag in cmake command, for more information about this flag see the README file in the location of the sample, for this location refer to the table 2-7.
Ensure you have sourced setvars.sh script located in the root of your oneAPI installation as explained in Table 2-3 Initialization Script for HLD tool.
cd path-to-sample-location\n mkdir build\n cd build\n
Cmake for compiling board_test when targeting USM board variant.
cmake -DFPGA_DEVICE=full-path-to-oneapi-asp/platform-name:board_variant -DSUPPORTS_USM=1 ..\n
Cmake for compiling the rest of the board variants.
cmake -DFPGA_DEVICE=full-path-to-oneapi-asp/platform-name:board_variant ..\n
Compile the design using the generated Makefile. The following build targets are provided:
- Generate the optimization report:
make report\n
- Compile for FPGA hardware (takes longer to compile, targets FPGA device):
make fpga\n
Hardware compilation takes several hours to complete. Once complete, you should see sample-name.fpga executable generated, where sample-name could be board_test or io_streaming_one_pipe.
For more information on additional environment settings required for running io_streaming_one_pipe sample see the README file in the location of the sample, for this location refer to the table 2-7 .
Run the generated hardware executable as follows:
./sample-name.fpga\n
Note: If your FPGA compile fails to meet timing requirements, the Intel oneAPI compiler prints an error message, returns an error code and deletes the generated binary. In case of timing failure, *.failing_clocks.rpt and *.failing_paths.rpt files are generated in compiled output directory sample-name.fpga.prj, where sample-name could be board_test or io_streaming_one_pipe. You can recompile with a different seed using -Xsseed option. You can pass this option using USER_HARDWARE_FLAGS=-Xsseed=seed_value in the cmake command above and recompile hardware image.
To view test details and usage information using the binary, use the -help option.
./sample-name.fpga -help # sample-name could be board_test or io_streaming_one_pipe.\n
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#27-release-vf","title":"2.7 Release VF","text":"Once you are done with your application testing, you can release the device from vfio-pci driver using following command.
$ sudo opae.io release -d s:b:d.vf\n
Sample output for Agilex\u00ae 7 FPGA OFS target platform having programmed 1PF/1VF FIM is shown below. The output for 2PF FIM, base_x16 FIM for Agilex\u00ae 7 FPGA and base_x16 FIM for Stratix\u00ae 10 FPGA should be similar.
Note: For Stratix\u00ae 10 FPGA you will need to release an extra VF as for this target 2 Vfs were created.
$ sudo opae.io release -d 0000:b1:00.1\nReleasing (0x8086,0xbccf) at 0000:b1:00.1 from vfio-pci\nRebinding (0x8086,0xbccf) at 0000:b1:00.1 to dfl-pci\n\n$ sudo opae.io ls\n[0000:b1:00.0] (0x8086, 0xbcce) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n[0000:b1:00.1] (0x8086, 0xbccf) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#30-oneapi-on-ofs-running-in-a-virtual-machine","title":"3.0 OneAPI on OFS Running in a Virtual Machine","text":"Virtual machines (VM's) can be used for FPGA development, 2PF FIM (built using pcie_host_2pf.ofss) is provided to use oneAPI on OFS Agilex\u00ae 7 FPGA device in a VM. For Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) refer to Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs to get the steps to build the 2PF FIM. For more information about 2PF FIM configuration refer to the FIM developer guides for your target device. - Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs. - Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs. - Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs.
The setup flow for your virtual machine could be found in the KVM User Guide: Open FPGA Stack, there are additional things for oneAPI flow listed below which you should ensure while configuring your VM:
- Assign the enough memory to the VM for running your oneAPI workloads.
- In section
5.1 Passing Devices to the VM when adding the PCIe Host Devices to the VM, ensure to have PF0 and PF1 BDF adjacent (s\\:b\\:d.f, s\\:b\\:d.f+1). The following example shows the address element of the PCIe Host Device XML file of PF0 and PF1, keeping the same value for domain, bus and slot attributes and only changing the function attribute (increasing its value by one):
- Install libnsl.so.1 library with the following command:
$sudo yum install libnsl.so.1\n
Once this setup is done, follow Section 2.0 Setup Flow for Using HLD Tool on OFS to finish the configuration of the VM for oneAPI .
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#appendix","title":"Appendix","text":""},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#appendix-a-oneapi-accelerator-support-packageasp-editor","title":"Appendix A: oneAPI Accelerator Support Package(ASP) Editor","text":"The oneapi-asp tag ofs-2024.2-2 offers a new option for the users to generate and customize the ASP in OFS based platforms using an editor tool. The aim of this tool is to enable easy parametrization of the oneAPI Accelerator Support Packages in OFS based platforms using the IP Parameter Editor GUI in Quartus Prime. This section covers all the steps to use the editor tool to generate ASP for your board variant using the OFS reference platform presets files.
Tables A-1 to A-4 show the reference configurations of the editor tool for all OFS reference platforms. For more information about the functionality, parametization of the oneAPI ASP editor tool refer to the Appendix B of the oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
Table A-1 Reference configurations of the oneAPI ASP editor for Intel\u00ae FPGA SmartNIC N6001-PL
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 2 PF Minimal FIM (pcie_host_2pf.ofss) ofs_n6001 ofs_n6001.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 2 PF Minimal FIM (pcie_host_2pf.ofss) ofs_n6001_usm ofs_n6001_usm.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_n6001_iopipes ofs_n6001_iopipes.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 8 Data Width 64 Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_n6001_usm_iopipes ofs_n6001_usm_iopipes.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^32 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 8 Data Width 64 Table A-2 Reference configurations of the oneAPI ASP editor for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_fseries-dk ofs_fseries-dk.qprs Number of Banks in each global memory 2 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 Default FIM ofs_fseries-dk_usm ofs_fseries-dk_usm.qprs Number of Banks in each global memory 2 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Table A-3 Reference configurations of the oneAPI ASP editor for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 1PF/1VF minimal FIM (pcie_host_2link_1pf_1vf.ofss) ofs_iseries-dk ofs_iseries-dk.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 1PF/1VF minimal FIM (pcie_host_2link_1pf_1vf.ofss) ofs_iseries-dk_usm ofs_iseries-dk_usm.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Table A-4 Reference configurations of the oneAPI ASP editor for Intel\u00ae FPGA PAC D5005
ASP Tab ASP Parameter Value FIM Configuration Board Variant ASP Preset File Global Memory (On-board) Number of Global Memories (different memory types) 1 default FIM ofs_d5005 ofs_d5005.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size Disabled Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A Global Memory (On-board) Number of Global Memories (different memory types) 1 default FIM ofs_d5005_usm ofs_d5005_usm.qprs Number of Banks in each global memory 4 Data Width 512 Memory Size (1 bank) 2^33 bytes Unified Shared Memory Size 2^48 Direct Memory Access (DMA) Number of DMA Channels 1 Channels Number of I/O Channels 0 Data Width N/A"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#generate-the-oneapi-accelerator-support-package-asp-using-the-presets-files-with-the-editor","title":"Generate the oneAPI Accelerator Support Package (ASP) Using the Presets Files with the Editor","text":"Sections 2.1 to 2.3 (until step 4, the environment variables are quite different) must be covered before generating the ASP .
1) Clone oneapi-asp repository and checkout tag matching the BKC for your target platform (see Tables 2-4 and 2-5 for the BKCs).
Note: 1) You will need a personal access token (use the classic mode) to be used as the password to clone successfully. Please see more information about token authentication requirements for Git operations here. 2) Editor tool is an option enabled from ofs-2024.2-2 tag onwards.
git clone https://github.com/OFS/oneapi-asp.git\n cd oneapi-asp\n git checkout tags/ofs-2024.2-2\n
Ensure the correct tag has ben checked out:
git describe --tags\n
- Output:
ofs-2024.2-2\n
2) Set the following environment variables required to execute build scripts successfully:
# Adding Quartus to PATH\nexport PATH=$PATH:path-to-quartus-installation-dir/quartus/bin\n export PATH=$PATH:path-to-quartus-installation-dir/qsys/bin\n export QUARTUS_ROOTDIR=path-to-quartus-installation-dir/quartus\n export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n # Other OFS environment variables\nexport OPAE_PLATFORM_ROOT=path-to-ofs-fim-pr_build_template-directory # (see section 2.2 for more details)\nexport LIBOPAE_C_ROOT=/usr # (OPAE libraries are installed in /usr/lib64 by default if you followed the OPAE SDK steps covered in section 2.1 as is and installed OPAE rpm packages. If you have a different OPAE installation path, please point LIBOPAE_C_ROOT to your OPAE installation location that you specified using -DCMAKE_INSTALL_PREFIX=installation-path in cmake command for building OPAE)\n
Note: To re-use this environment setting, you can copy the above export statements to a shell script, update the paths to match your tool installations and source this script each time a new shell is started.
3) Source initialization script for oneAPI as is demonstrated in section 2.3 Prerequisites.
source path-to-intel-oneapi-toolkit-installation-directory/setvars.sh\n
4) Go to the oneapi_asp_editor folder
cd path-to-directory-containing-oneapi-asp/oneapi-asp/oneapi_asp_editor\n
5) Run the Platform Designer editor from command-line and open the oneapi_asp_editor.ip file selecting the oneapi_asp_editor as project file. Figure A-1 shows the oneAPI ASP Editor GUI.
qsys-edit ip/oneapi_asp_editor.ip --qpf=oneapi_asp_editor\n
Figure A-1 oneAPI ASP Editor GUI
6) At the right bottom of the page are displayed all the OFS reference platform presets provided in the oneapi-asp. Select the board variant you want to include in your ASP, click the Apply botton, you will see how the parameters will update according to the board variant settings. Then click the generate HLD botton, select the options suitable for you, click generate and save the changes, repeat the process for the rest of the board variants you want to add in your ASP.
Figure A-2 Editor GUI Presets
Figure A-3 Generate HLD window
7) Once you have added all the board variants of the OFS reference platform you are using, exit platform designer. A folder with the name of the platform you are using will be created inside the oneapi_asp_editor folder, the presets files you have generated will be already included in the hardware subfolder.
8) Go to the folder of the reference platform you are using, export two environment variables and run the build-asp.sh script.
cd platform-name/\n
export AOCL_BOARD_PACKAGE_ROOT=$PWD\nexport OFS_ASP_ROOT=$PWD\n
./scripts/build-asp.sh\n
Note: Where platform-name is n6001 for Intel\u00ae FPGA SmartNIC N6001-PL, fseries-dk for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), iseries-dk for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) and d5005 for Intel\u00ae FPGA PAC D5005
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#31-further-development","title":"3.1 Further Development","text":"Once you have completed running the oneAPI sample application, you can start developing your own applications.
For more information about developing FPGA applications with Intel oneAPI, refer to Intel\u00ae oneAPI DPC++/C++ Compiler Handbook for Intel\u00ae FPGAs.
If you want to customize the oneAPI ASP, you can refer to oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack.
"},{"location":"hw/common/user_guides/oneapi_asp/ug_oneapi_asp/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
- Set up the Intel\u00ae Quartus\u2122 Prime Pro Edition Software in a host server
- Set up the Docker engine
- Build and load your Docker image to the Docker engine
- Run a Docker container with OFS preloaded
The Open FPGA Stack (OFS) Docker image has two main personas:
- Development: You can develop, simulate, and build any component of the OFS. The Docker image enables you to use your laptop or server without having drivers, FPGA Platform, or specific Linux* distribution installed in your host computer. You can follow the development flow provided to run Docker on Linux.
- Deployment: You can program, load binaries, or execute real-time testing using the OPAE and OFS. To do so, the host computer must have the specified software distribution and drivers installed.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#12-background-information","title":"1.2 Background Information","text":"A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":" - What is a container?
- Docker vs. Virtual Machines
- Does the Docker container have its own Kernel?
- No, Docker image or Container uses the application layer of the host computer; this functionality is the main reason for docker having lightweight and fast applications.
- Does Docker run on Linux, macOS, and Windows?
- Intel Docker Image can use the PCIe card from the host server?
- Yes, The drivers and additional information could be shared, but this could create potential security concerns (ensure your system is secure).
- Docker security
- Docker subscription
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#20-prerequisites-and-scope","title":"2.0 Prerequisites and Scope","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"hw/common/user_guides/ug_docker/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\n
-
Add the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\n
-
Install the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#ubuntu-2204","title":"Ubuntu 22.04","text":"The Docker installation steps for Ubuntu are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\n
-
Install packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\n
-
Add Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\n
-
The following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\n
-
Update the package manager index again:
sudo apt-get update\n
-
Install Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#33-load-docker-image-installation","title":"3.3 Load Docker Image installation","text":"The Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#build-the-image","title":"Build the image","text":" -
You can download the Dockefile from OFS GitHub Docker.
-
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\n
Save the file
-
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\n
Note: Never remove --no-cache this could cause issues with your environmental variables inside of the container
-
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\n
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#volumen-creation","title":"Volumen creation","text":" -
Docker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\n
For more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
-
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\n
-
Inside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\n
The docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#34-create-a-container","title":"3.4 Create a container","text":"Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
- Using the previous example now, you can execute the docker run command.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
-
Now the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\n
Your Container ID is bdc1289fb081.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\n
The container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
-
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the evaluation script.
-
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\n
-
The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n
Now you can control and compile the design. You can use the interactive or de-attach mode. -
If you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\n
all the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#40-deployment","title":"4.0 Deployment","text":"The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
- Follow the steps listed in sections 2.1 to 2.3
- 2.1 Quartus installation
- 2.2 Docker Engine installation
- 2.3 Load Docker Image installation
- The steps required for DFL driver installation are documented Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
-
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Example, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
-
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\n
-
Now, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n
\u200b Your Container ID is 25b41eb4d232.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\n
The container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
-
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the eval script.
-
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n
- The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\n
Now you can control and compile the design using the interactive or de-attach mode.
"},{"location":"hw/common/user_guides/ug_docker/ug_docker/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/","title":"Evaluation User Guide: OFS for Agilex\u00ae 7 PCIe Attach","text":""},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#1-overview","title":"1 Overview","text":""},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the checkout and evaluation of an Intel\u00ae FPGA SmartNIC N6001-PL, Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) development platform using Open FPGA Stack (OFS). After reviewing the document, you will be able to:
- Use the script to run through the most common build and simulation flows when using OFS
- Run hardware and software tests to evaluate the complete OFS flow
- Modify and leverage the script to your environment and design
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#table-1-2-software-version-summary","title":"Table 1-2: Software Version Summary","text":"Component Version Description FPGA Platform Intel\u00ae FPGA SmartNIC N6001-PL, Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile), Agilex platform you can use for your custom board development OFS FIM Source Code Branch: https://github.com/OFS/ofs-agx7-pcie-attach, Tag: ofs-2024.2-1 OFS Shell RTL for Agilex FPGA (targeting Intel\u00ae FPGA SmartNIC N6001-PL) AFU Examples Branch: examples-afu , Tag:ofs-2024.2-1 Tutorials and simple examples for the Accelerator Functional Unit region (workload region) OPAE SDK Branch: 2.13.0-3, Tag: 2.13.0-3 Open Programmable Acceleration Engine Software Development Kit Kernel Drivers Branch: intel-1.11.0-2, Tag: intel-1.11.0-2 OFS specific kernel drivers OPAE Simulation Branch: opae-sim, Tag: 2.13.0-2 Accelerator Simulation Environment for hardware/software co-simulation of your AFU (workload) Quartus Prime Pro Edition Design Software 24.1 [Quartus\u00ae Prime Pro Edition Linux] Software tool for Altera FPGA Development Operating System RHEL 8.10 Operating system on which this script has been tested A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae FPGA SmartNIC N6001-PL can be found on the OFS ofs-2024.2-1 official release drop on GitHub.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#2-introduction-to-ofs-evaluation-script","title":"2 Introduction to OFS Evaluation Script","text":"Two scripts have been developed to allow the user to clone, build, compile and test the OFS platform hardware.
1) ofs-clone_repositories.sh. This script clones the repositories from GitHub needed to build and test any OFS platform. 2) ofs-agx7-pcie-attach_eval.sh. This script evaluates compiles, builds and tests hardware features that the OFS framework provides. This script can also be leveraged for your own development.
NOTE:
This guide uses the Intel\u00ae FPGA SmartNIC N6001-PL as the platform for all example steps. Additionally, this guide and the example steps can be used with other platforms, such as the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile).
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#21-pre-requisites","title":"2.1 Pre-Requisites","text":"This script uses the following set of software tools which should be installed using the directory structure below. Tool versions can vary.
- Quartus\u00ae Prime Pro Software : The software can be downloaded here. For details on installation of required patches and quartus installation, refer to section 1.3.4 of the Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs.
- Synopsys\u00ae VCS Simulator : This simulator can be downloaded from the Synopsys web page here
- Siemens\u00ae Questa\u00ae Simulator : This simulator can be downloaded from the Siemens web page here
Figure 2-1 Folder Hierarchy for Software Tools
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#22-agilex-7-pcie-attach-clone-and-evaluation-script-download-and-modification","title":"2.2 Agilex\u00ae 7 PCIe Attach Clone and Evaluation Script download and modification","text":" - Download the follwing tar file (scripts_ofs-2024.2_external.tar.gz) from the assets tab of the release page
-
Untar to a user folder using the following command
tar -xvf scripts_ofs-2024.1-1_external.tar.gz\n````\n\n3. The folder structure containing the clone script (ofs-clone_repositories.sh) and evaluation script (ofs-agx7-pcie-attach_eval.sh) is as shown below in Figure 2-2\n\n**Figure 2-2 Directory Structure for the clone script**\n\n``` sh\n## ofs_scripts\n## -> host_chan_mmio.stp\n## -> ofs-agx7-pcie-attach_eval.sh\n## -> README_ofs-agx7-pcie-attach_eval.txt\n## ofs-clone_repositories.sh\n
-
Open the clone script ofs-clone_repositories.sh in any text editor
- Please enter the location of your proxy server to allow access to external internet to build software packages. (lines 6-8)
Note: Failing to add proxy server will prevent cloning of repositories and the user will be unable to build the OFS framework.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
-
Please enter the license file locations (lines 11-13) for the following tool variables
export LM_LICENSE_FILE=<user_license>\n export DW_LICENSE_FILE=<user_license>\n export SNPSLMD_LICENSE_FILE=<user_license>\n
-
Add the name of the directory where you want the platform repositories to be cloned (line 19)
export OFS_RELEASE=ofs-2024.2-1\n
-
After setting the required variables, source the clone script with the following command
source ofs-clone_repositories.sh\n
- The ofs-clone_repositories.sh script has different platforms to choose from the menu as shown below in Figure 2-3. For the Intel\u00ae FPGA SmartNIC N6001-PL the user will choose option 3
Figure 2-3 Directory Structure for OFS Project
-
Once the repositories are cloned, the user can navigate to the following directory
cd ofs-2024.2-1\n
-
After cloning, the OFS repositories will be shown below in Figure 2-4.
Figure 2-4 Directory Structure for OFS Project
## ofs-2024.2-1\n## N6001(OFS platform)\n## -> examples-afu\n## -> ofs-agx7-pcie-attach\n## -> oneapi-asp\n## -> oneAPI-samples\n## -> opae-sim\n## -> opae-sdk\n## -> linux-dfl-backport\n## -> ofs-agx7-pcie-attach_eval.sh\n## -> README_ofs-agx7-pcie-attach_eval.txt\n## -> host_chan_mmio.stp\n
- Once the repositories are cloned, open the platform specific evaluation script, e.g., in ofs-agx7-pcie-attach_eval.sh and follow the instructions below which explains which line numbers to change to adapt this script to the user environment.
a) Tools Location (line 87, 88, 89, 90)
Set Location of Quartus, Synopsys, Questasim and oneAPI Tools
export QUARTUS_TOOLS_LOCATION=/home\nexport SYNOPSYS_TOOLS_LOCATION=/home\nexport QUESTASIM_TOOLS_LOCATION=/home\nexport ONEAPI_TOOLS_LOCATION=/opt\n
In the example above /home is used as the base location of Quartus, Synopsys and Questasim tools, /opt is used for the oneAPI tools
b) PCIe (Bus Number)
The Bus number must be entered by the user after installing the hardware in the chosen server, in the example below \"b1\" is the Bus Number for a single card as defined in the evaluation script.
export OFS_CARD0_BUS_NUMBER=b1\n
The evaluation script uses the bus number as an identifier to interrogate the card. The command below will identify the accelerator card plugged into a server.
lspci | grep acc\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\nb1:00.1 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.2 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\nb1:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
The result identifies the card as being assigned \"b1\" as the bus number so the entry in the script changes to
export OFS_CARD0_BUS_NUMBER=b1\n
The user can also run the following command on the ofs-agx7-pcie-attach_eval.sh script to automatically change the bus number to b1 in the ofs-agx7-pcie-attach_eval.sh script.
grep -rli 'b1' * | xargs -i@ sed -i 'b1' @
if the bus number is 85 for example
85:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\n85:00.1 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.2 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\n85:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
the command to change to 85 in the evaluation script would be
grep -rli 'b1' * | xargs -i@ sed -i '85' @
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#ofs-platform-example-n6000-n6001-fseries-dk-iseries-dk-custom_board-choice-line-173","title":"OFS Platform (Example:= n6000, n6001, fseries-dk, iseries-dk, custom_board) choice (line 173)","text":"The ofs-agx7-pcie-attach_eval.sh script has now been modified to the server set-up and the user can proceed to build, compile and simulate the OFS stack
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#3-ofs-agx7-pcie-attach-evaluation-script","title":"3 ofs-agx7-pcie-attach Evaluation Script","text":""},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#31-overview","title":"3.1 Overview","text":"The figure below shows a snapshot of the full evaluation script menu showing all 62 options and each one of 11 sub-menus which focus on different areas of evaluation. Each of these menu options are described in the next section.
Figure 3-1 ofs-agx7-pcie-attach_eval.sh Evaluation Menu
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#311-ofs-tools-menu","title":"3.1.1 OFS TOOLS MENU","text":"By selecting \"List of Documentation for OFS Project,\" a list of links to the latest OFS documentation appears. Note that these links will take you to documentation for the most recent release which may not correspond to the release version you are evaluating. To find the documentation specific to your release, ensure you clone the OFS tag that corresponds to your OFS version.
By selecting \"Check Versions of Operating System and Quartus Premier Design Suite\", the tool verifies correct Operating System, Quartus version, kernel parameters, license files and paths to installed software tools.
Menu Option Example Output 1 - List of Documentation for OFS PCI Attach Project Project Open FPGA Stack Overview Guides you through the setup and build steps to evaluate the OFS solution https://ofs.github.io 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS) Checking Linux release Linux version 5.15.52-dfl (guest@hw-rae-svr4-l) (gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4), GNU ld version 2.30-79.el8) #1 SMP Fri Sep 23 17:19:37 BST 2022 Checking RedHat release CentOS Linux release 8.3.2011 Checking Ubuntu release cat: /etc/lsb-release: No such file or directory Checking Kernel parameters BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.15.52-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 Checking Licenses LM_LICENSE_FILE is set to port@socket number:port@socket number DW_LICENSE_FILE is set to port@socket number:port@socket number SNPSLMD_LICENSE_FILE is set to port@socket number:port@socket number Checking Tool versions QUARTUS_HOME is set to /home/intelFPGA_pro/24.1/quartus QUARTUS_ROOTDIR is set to /home/intelFPGA_pro/24.1/quartus IMPORT_IP_ROOTDIR is set to /home/intelFPGA_pro/24.1/quartus/../ip QSYS_ROOTDIR is set to /home/intelFPGA_pro/24.1/quartus/../qsys/bin Checking QPDS Patches Quartus Prime Shell Version 24.1 Build XXX XX/XX/XXXX Patches X.XX SC Pro Edition Copyright (C) XXXX Intel Corporation. All rights reserved."},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#312-ofs-hardware-menu","title":"3.1.2 OFS HARDWARE MENU","text":"Identifies card by PCIe number, checks power, temperature and current firmware configuration.
Menu Option Example Output 3 - Identify Platform Hardware via PCIe PCIe card detected as b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.4 Processing accelerators: Intel Corporation Device bcce Host Server is connected to SINGLE card configuration 4 - Identify the Board Management Controller (BMC) Version and check BMC sensors Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** BMC SENSORS ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 5 - Identify the FPGA Management Engine (FME) Version Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Management Controller Build version: 3.2.0 //****** FME ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 Boot Page : user1 Factory Image Info : a7c6879683182ce61084c420e51f50b6 User1 Image Info : 8a7440ddff52e0e27dbb989d5eb954f4 User2 Image Info : a7c6879683182ce61084c420e51f50b6 6 - Check Board Power and Temperature Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** POWER ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 ( 1) VCCRT_GXER_0V9 Voltage : 0.91 Volts etc ...................... Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** TEMP ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 ( 1) FPGA E-Tile Temperature [Remote] : 33.50 Celsius etc ...................... Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 7 - Check Accelerator Port status //****** PORT ******// Object Id : 0xED00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 8 - Check MAC and PHY status Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** MAC ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 Number of MACs : 255 mac info is not supported Intel Acceleration Development Platform N6001 Board Management Controller NIOS FW version: 3.2.0 Board Management Controller Build version: 3.2.0 //****** PHY ******// Object Id : 0xEE00000 PCIe s:b:d.f : 0000:B1:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x1771 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x50102027135A894 Bitstream Version : 5.0.1 Pr Interface Id : 7dbb989d-5eb9-54f4-8a74-40ddff52e0e2 //****** HSSI information ******// HSSI version : 1.0 Number of ports : 8 Port0 :25GbE DOWN Port1 :25GbE DOWN Port2 :25GbE DOWN Port3 :25GbE DOWN Port4 :25GbE DOWN Port5 :25GbE DOWN Port6 :25GbE DOWN Port7 :25GbE DOWN Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#313-ofs-pfvf-mux-menu","title":"3.1.3 OFS PF/VF MUX MENU","text":"This menu reports the number of PF/VF functions in the reference example and also allows you to reduce the number to 1PF and 1VF to reduce resource utilisation and create a larger area for your workload development. This selection is optional and if the user wants to implement the default number of PF's and VF's then option 9, 10 and 11 should not be used. Additionally the code used to make the PF/VF modification can be leveraged to increase or modify the number of PF/VFs in the existing design within the limits that the PCIe Subsystem supports (8PF/2K VFs)
Menu Option Description 9 - Check PF/VF Mux Configuration This menu selection displays the current configuration of all n6001 ofss files located in the following directory $OFS_ROOTDIR/tools/ofss_config Check n6001 base config OFSS set up [ip] type = ofs [settings] platform = n6001 family = agilex fim = base_x16 part = AGFB014R24A2E2V device_id = 6001 Check n6001 hssi_2x100 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GCAUI-4 Check n6001 hssi_2x100_caui2 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GAUI-2 Check n6001 hssi_8x10 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 10GbE Check n6001 hssi_8x25 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 25GbE Check n6001 IOPLL OFSS set up [ip] type = iopll [settings] output_name = sys_pll instance_name = iopll_0 [p_clk] freq = 470 Check n6001 Memory OFSS set up [ip] type = memory [settings] output_name = mem_ss_fm preset = n6001 Check n6001 PCIe Host OFSS set up [ip] type = pcie [settings] output_name = pcie_ss [pf0] num_vfs = 3 bar0_address_width = 20 vf_bar0_address_width = 20 [pf1] [pf2] bar0_address_width = 18 [pf3] [pf4] Check n6001 PCIe 1pf_1vf OFSS set up [ip] type = pcie [settings] output_name = pcie_ss [pf0] num_vfs = 1 bar0_address_width = 20 vf_bar0_address_width = 20 10 - Modify PF/VF Mux Configuration As an example this menu selection modifies the pcie_host.ofss file to 1 PF located in the following directory $OFS_ROOTDIR/tools/ofss_config This option also displays the modified pcie_host.ofss file 11 - Build PF/VF Mux Configuration If option 10 is not used then then the default number of PF's and VF's is used to build the FIM, if option 10 is selected then only 1 VF is built to reduce logic utilisation"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#314-ofs-fimpr-build-menu","title":"3.1.4 OFS FIM/PR BUILD MENU","text":"Builds FIM, Partial Reconfiguration Region and Remote Signal Tap
Menu Option Description 12 - Check OFS software version for OFS Project OFS_ROOTDIR is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach OPAE_SDK_REPO_BRANCH is set to release/X.X.X OPAE_SDK_ROOT is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/../opae-sdk LD_LIBRARY_PATH is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/../opae-sdk/lib64: 13 - Build FIM for Hardware This option builds the FIM based on the setting for the $OFS_PLATFORM, $FIM_SHELL environment variable. Check this variable in the following file ofs-agx7-pcie-attach_eval.sh 14 - Check FIM Identification of FIM for Hardware The FIM is identified by the following file fme-ifc-id.txt located at $OFS_ROOTDIR/$FIM_WORKDIR/syn/board/$OFS_PLATFORM/syn_top/ 15 - Build Partial Reconfiguration Tree for Hardware This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the oneAPI build flow 16 - Build Base FIM Identification(ID) into PR Build Tree template This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and oneAPI workloads 17 - Build Partial Reconfiguration Tree for Hardware with Remote Signal Tap This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the oneAPI build flow for the Remote Signal Tap flow 18 - Build Base FIM Identification(ID) into PR Build Tree template with Remote Signal Tap This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree for Remote Signal Tap to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and oneAPI workloads"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#315-ofs-hardware-programmingdiagnostic-menu","title":"3.1.5 OFS HARDWARE PROGRAMMING/DIAGNOSTIC MENU","text":"The following submenu allows you to: * Program and check flash * Perform a remote system update (RSU) of the FPGA image into the FPGA * Bind virtual functions to VFIO PCIe driver * Run host exerciser (HE) commands such as loopback to test interfaces VFIO PCI driver binding * Read the control and status registers (CSRs) for bound modules that are part of the OFS reference design.
Menu Option Description 19 - Program BMC Image into Hardware The user must place a new BMC flash file in the following directory $OFS_ROOTDIR/bmc_flash_files. Once the user executes this option a new BMC image will be programmed. A remote system upgrade command is initiated to store the new BMC image 20 - Check Boot Area Flash Image from Hardware This option checks which location area in FLASH the image will boot from, the default is user1 Boot Page : user1 Note: This feature is not supported on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 21 - Program FIM Image into user1 area for Hardware This option programs the FIM image \"ofs_top_page1_unsigned_user1.bin\" into user1 area in flash Note: Please refer to the Getting Started Guide for details on flashing images for the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 22 - Initiate Remote System Upgrade (RSU) from user1 Flash Image into Hardware This option initiates a Remote System Upgrade and soft reboots the server and re-scans the PCIe bus for the new image to be loaded 2022-11-10 11:26:24,307 - [[pci_address(0000:b1:00.0), pci_id(0x8086, 0xbcce)]] performing RSU operation 2022-11-10 11:26:24,310 - [[pci_address(0000:b0:02.0), pci_id(0x8086, 0x347a)]] removing device from PCIe bus 2022-11-10 11:26:24,357 - waiting 10 seconds for boot 2022-11-10 11:26:34,368 - rescanning PCIe bus: /sys/devices/pci0000:b0/pci_bus/0000:b0 2022-11-10 11:26:35,965 - RSU operation complete Note: Please refer to the Getting Started Guide for details on flashing images for the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 23 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 24 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 25 - Create Virtual Functions (VF) and bind driver to vfio-pci Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 23 to check that the new drivers are bound 26 - Verify FME Interrupts with hello_events The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors 27 - Run HE-LB Test This option runs 5 tests 1) checks and generates traffic with the intention of exercising the path from the AFU to the Host at full bandwidth 2) run a loopback throughput test using one cacheline per request 3) run a loopback read test using four cachelines per request 4) run a loopback write test using four cachelines per request 5) run a loopback throughput test using four cachelines per request 28 - Run HE-MEM Test This option runs 2 tests 1) Checking and generating traffic with the intention of exercising the path from FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host 2) run a loopback throughput test using one cacheline per request 29 - Run HE-HSSI Test This option runs 1 test HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic 1) Send traffic through the 10G AFU 30 - Run Traffic Generator AFU Test This option runs 3 tests TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank 1) Run the preconfigured write/read traffic test on channel 0 2) Target channel 1 with a 1MB single-word write only test for 1000 iterations 3) Target channel 2 with 4MB write/read test of max burst length for 10 iterations 31 - Read from CSR (Command and Status Registers) for Hardware This option reads from the following CSR's HE-LB Command and Status Register Default Definitions HE-MEM Command and Status Register Default Definitions HE-HSSI Command and Status Register Default Definitions"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#316-ofs-hardware-afu-testing-menu","title":"3.1.6 OFS HARDWARE AFU TESTING MENU","text":"This submenu tests partial reconfiguration by building and loading an memory-mapped I/O example AFU/workload, executes software from host, and tests remote signal tap.
Menu Option Description 32 - Build and Compile host_chan_mmio example This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) ready for hardware programming 33 - Execute host_chan_mmio example This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test 34 - Modify host_chan_mmio example to insert Remote Signal Tap This option inserts a pre-defined host_chan_mmio.stp Signal Tap file into the OFS code to allow a user to debug the host_chan_mmio AFU example 35 - Build and Compile host_chan_mmio example with Remote Signal Tap This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS(Green Bit Stream) ready for hardware programming with Remote Signal tap enabled 36 - Execute host_chan_mmio example with Remote Signal Tap This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test. The user must open the Signal Tap window when running the host code to see the transactions in the Signal Tap window"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#317-ofs-hardware-afu-bbb-testing-menu","title":"3.1.7 OFS HARDWARE AFU BBB TESTING MENU","text":"This submenu tests partial reconfiguration using a hello_world example AFU/workload, executes sw from host
Menu Option Description 37 - Build and Compile hello_world example This option builds the hello_ world example from the following repo $FPGA_BBB_CCI_SRC/samples/tutorial/afu_types/01_pim_ifc/$AFU_BBB_TEST_NAME, where AFU_BBB_NAME=hello_world. This produces a GBS(Green Bit Stream) ready for hardware programming 38 - Execute hello_world example This option builds the host code for hello_world example and programs the GBS file and then executes the test"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#318-ofs-oneapi-project-menu","title":"3.1.8 OFS ONEAPI PROJECT MENU","text":"Builds oneAPI kernel, executes sw from host and runs diagnostic tests
Menu Option Result 39 - Check oneAPI software versions for Project This option checks the setup of the oneAPI software and adds the relevant oneAPI environment variables to the terminal. This option also informs the user to match the oneAPI software version to the oneAPI-samples version 40 - Build and clone shim libraries required by oneAPI host This option builds the oneAPI directory structure 41 - Install oneAPI Host Driver This option Installs the oneAPI Host driver at the following location /opt/Intel/OpenCLFPGA/oneAPI/Boards/, and requires sudo permission 42 - Uninstall oneAPI Host Driver This option Uninstall's the oneAPI Host driver, and requires sudo permissions 43 - Diagnose oneAPI Hardware This option Checks ICD (Intel Client Driver) and FCD (FPGA Client Driver), oneAPI library locations and detects whether oneAPI BSP is loaded into the FPGA 44 - Build oneAPI BSP Default Kernel (hello_world) This option Builds the oneAPI BSP using simple-add_buffers kernel 45 - Build oneAPI MakeFile Environment This option Builds the oneAPI environment using a Makefile for kernel insertion 46 - Compile oneAPI Sample Application (board_test) for Emulation This option compiles the board_test kernel for Emulation 47 - Run oneAPI Sample Application (board_test) for Emulation This option executes the board_test kernel for Emulation 48 - Generate oneAPI Optimization report for (board_test) This option generates an optimization report for the board_test kernel 49 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 50 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 51 - Create Virtual Function (VF) and bind driver to vfio-pci Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 45 to check that the new drivers are bound 52 - Program OpenCL BSP Default Kernel (hello_world) This option programs the FPGA with a aocf file based on the hello_world kernel 53 - Compile oneAPI Sample Application (board_test) for Hardware This option compiles the board_test kernel for Hardware 54 - Run oneAPI Sample Application (board_test) for Hardware This option builds the host code for board_test kernel and executes the program running through kernel and host bandwidth tests"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#319-ofs-unit-test-project-menu","title":"3.1.9 OFS UNIT TEST PROJECT MENU","text":"Builds, compiles and runs standalone simulation block tests. More unit test examples are found at the following location ofs_n6001/sim/unit_test
Menu Option Result 55 - Generate Simulation files for Unit Test This option builds the simulation file set for running a unit test simulation 56 - Simulate Unit Test dfh_walker and log waveform This option runs the dfh_walker based on the environment variable \"UNIT_TEST_NAME=dfh_walker\" in the evaluation script. A user can change the test being run by modifying this variable"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#3110-ofs-uvm-project-menu","title":"3.1.10 OFS UVM PROJECT MENU","text":"Builds, compiles and runs full chip simulation tests. The user should execute the options sequentially ie 68,69, 70 and 71
Menu Option Description 57 - Check UVM software versions for Project DESIGNWARE_HOME is set to /home/synopsys/vip_common/vip_Q-2020.03A UVM_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm VCS_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel VERDIR is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/verification VIPDIR is set to /home/user_area/ofs-X.X.X/ofs-agx7-pcie-attach/verification 58 - Compile UVM IP This option compiles the UVM IP 59 - Compile UVM RTL and Testbench This option compiles the UVM RTL and Testbench 60 - Simulate UVM dfh_walking_test and log waveform This option runs the dfh_walking test based on the environment variable \"UVM_TEST_NAME=dfh_walking_test\" in the evaluation script. A user can change the test being run by modifying this variable 61 - Simulate all UVM test cases (Regression Mode) This option runs the n6001 regression mode, cycling through all UVM tests defined in verification/tests/test_pkg.svh file"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#3111-ofs-build-all-project-menu","title":"3.1.11 OFS BUILD ALL PROJECT MENU","text":"Builds the complete OFS flow, good for regression testing and overnight builds
For this menu a user can run a sequence of tests (compilation, build and simulation) and executes them sequentially. After the script is successfully executed, a set of binary files is produced which a you can use to evaluate your hardware. Log files are also produced which checks whether the tests passed.
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 62 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 44, 45, 53, 55, 56, 57, 58, 59 and 60. These 24 menu options are chosen to build the complete OFS flow covering build, compile and simulation.
Menu Option Result 62 - Build and Simulate Complete Project Generating Log File with date and timestamp Log file written to /home/guest/ofs-2.3.1/log_files/ofs-agx7-pcie-attach_log_2022_11_10-093649/ofs-agx7-pcie-attach_eval.log"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#definition-of-multi-test-set-up","title":"Definition of Multi-Test Set-up","text":"Menu Option 62 above in the evaluation script can be refined to tailor it to the users need and is principally defined by the variable below
MULTI_TEST[A,B]=C
where
A= Total Number of menu options in script B= Can be changed to a number to select the test order C= Menu Option in Script
Example 1 MULTI_TEST[62,0]=2
A= 62 is the total number of options in the script B= 0 indicates that this is the first test to be run in the script C= Menu option in Script ie 2- List of Documentation for OFS n6001 Project
Example 2 MULTI_TEST[62,0]=2 MULTI_TEST[62,1]=9
In the example above two tests are run in order ie 0, and 1 and the following menu options are executed ie 2- List of Documentation for OFS n6001 Project and 9 - Check OFS software versions for OFS n6001 Project
The user can also modify the build time by de-selecting options they do not wish to use, see below for a couple of use-case scenarios.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#default-user-case","title":"Default User Case","text":"A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 62 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 44, 45, 53, 55, 56, 57, 58, 59 and 60. All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#user-case-for-ofs-fimpr-build-menu","title":"User Case for OFS FIM/PR BUILD MENU","text":"In the example below when the user selects option 62 from the main menu the script will only run options from the OFS FIM/PR BUILD MENU (7 options, main menu options 12, 13, 14, 15, 16, 17 and 18). All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#4-n6001-common-test-scenarios","title":"4 n6001 Common Test Scenarios","text":"This section will describe the most common compile build scenarios if a user wanted to evaluate an acceleration card on their server. The Pre-requisite column indcates the menu comamnds that must be run befere executing the test eg To run Test 5 then a user needs to have run option 13, 15 and 16 before running options 23, 24, 25, 32 and 33.
Test Test Scenario Pre-Requisite Menu Option Menu Option Test 1 FIM Build - 13 Test 2 Partial Reconfiguration Build 13 15, 16 Test 3 Program FIM and perform Remote System Upgrade 13 21, 22 Test 4 Bind PF and VF to vfio-pci drivers - 23, 24, 25 Test 5 Build, compile and test AFU on hardware 13, 15, 16 23, 24, 25, 32, 33 Test 6 Build, compile and test AFU Basic Building Blocks on hardware 13, 15, 16 23, 24, 25, 37, 38 Test 7 Build, compile and test oneAPI on hardware 13, 15, 16 39, 40, 41, 44, 45, 49, 50, 51, 52, 53, 54 Test 8 Build and Simulate Unit Tests - 55, 56 Test 9 Build and Simulate UVM Tests - 57, 58, 59, 60 Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/user_guides/ug_eval_script_ofs_agx7_pcie_attach/ug_eval_script_ofs_agx7_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/","title":"FPGA Developer Journey Guide: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#1-introduction","title":"1 Introduction","text":"This document is intended to help you understand the FPGA developer flow using OFS as well as considerations you should take when creating your custom platform.
After reviewing the document, you shall be able to:
- Understand how to evaluate the OFS framework for your platform needs.
- Select a starting shell or FIM to begin your work.
- Understand what resources are available for guiding you through making modifications for custom design as well as simulation and debug.
The general development flow is depicted in the diagram below and discussed in more detail in each section of this document.
Figure 1-1: FPGA Developer Flow
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#2-evaluate-ofs","title":"2 Evaluate OFS","text":"The repositories in the OFS site are tagged based on year and release number in the format <yyyy>.<release>-<iteration>. For example, a tag of 2024.2-1 indicates the 2nd release of the year. If there are any updates to a particular release package, the last number will be incremented by 1, for example, as 2024.2-2.
By clicking on the release link to the right of the RTL repositories, you will find the latest release, the tag number and release notes.
Figure 2-1: OFS Repository Release Page Link
By scrolling to the end of the release page, you will find assets attached to the release that you can leverage for quick evaluation of OFS, such as FPGA binary files, POF and SOF as well as a sample AFU and a partial reconfiguration directory for building your own custom workload.
There are two ways to evaluate OFS depending on your needs:
Option 1: Setup your card and software in a server using the steps provided in one of the corresponding Getting Started Guides and leverage the appended binaries in the FIM RTL repository release page \"Assets\" tab to preview the software and design functionality the OFS framework provides you out of the box. This step will give you a good high-level overview of OFS. Getting Started Guides are available for the following FIM(shell) designs:
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs
Option 2: After your card and software are installed using the steps provided in one of the corresponding Getting Started Guides listed above, use a corresponding Evaluation Guide and provided evaluation script to run through all the capabilities of the OFS framework by selecting one of the choices in the evaluation menu. The evaluation script gives you familiarity of the entire design, build, simulation, programming and test flow for OFS, including a OneAPI flow.
- Automated Evaluation User Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs
- Automated Evaluation User Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Automated Evaluation User Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs
The scripts that go with each user guide are found in the assets tab of the corresponding FIM RTL repository's latest tag release page.
To use the full functionality of the script you will want to ensure you clone all of the appropriate repositories below at the same level just under an OFS directory you create, such as ofs-2024.2, similar to the figure below.
Figure 2-2: Directory Structure of Cloned Repositories
##|-- ofs-2024.2\n##| |--examples-afu\n##| |--linux-dfl-backport\n##| |--ofs-agx7-pcie-attach\n##| |--ofs-oneapi-asp\n##| |--opae-sdk\n##| |--opae-sim\n##| |--ofs-agx7-pcie-attach_eval.sh\n
You can access the repositories from ofs.github.io by clicking on the GitHub icon in the right corner of the site.
Figure 2-3: OFS Site Link from ofs.github.io
After making your top level directory and initializing the repository, clone one of the FIM RTL repositories you intend to use:
#Make top level directory\nmkdir OFS\ncd OFS\n\n#Initialize repository\ngit init\n\n#Select a FIM RTL repository to clone\n#To clone Intel Agilex 7 PCIe Attach FIM RTL repository\ngit clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\ncd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n\n#To clone Intel Agilex 7 SoC Attach FIM RTL Repository\ngit clone --recurse-submodules https://github.com/OFS/ofs-f2000x-pl.git\ncd ofs-f2000x-pl\ngit checkout --recurse-submodules tags/ofs-2024.1-1\n\n#To clone Intel Stratix 10 PCIe Attach FIM RTL Repository\ngit clone --recurse-submodules https://github.com/OFS/ofs-d5005.git\ncd ofs-d5005\ngit checkout --recurse-submodules tags/ofs-2024.1-1\n
After cloning your FIM RTL repository, clone the other necessary repositories:
#All other repositories below should be cloned to use the evaluation script:\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n\n#Use this OPAE clone command for all PCIe Attach Cards\ngit clone https://github.com/OFS/opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\n\n#Use this OPAE clone command for the SoC Attach Card\ngit clone https://github.com/OFS/opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.12.0-5\n\ngit clone https://github.com/OFS/oneapi-asp.git cd /home/OFS/oneapi-asp\ngit checkout tags/ofs-2024.2-2\n\ngit clone https://github.com/OFS/ofs-platform-afu-bbb.git\ncd /home/OFS/ofs-platform-afu-bbb\ngit checkout tags/ofs-2024.1-1\n\ngit clone https://github.com/OFS/opae-sim.git\ncd /home/OFS/opae-sim\ngit checkout tags/2.13.0-2\n\ngit clone https://github.com/OFS/examples-afu.git\ncd /home/OFS/examples-afu\ngit checkout tags/ofs-2024.1-1\n
You will also want to ensure you install the correct version of Intel Quartus Prime Pro as directed in the release notes in addition to any Quartus patches. Note that Quartus Prime Pro software can be downloaded from the downloads tab on intel.com. Quartus Prime Pro patches required are attached to the assets tab at the bottom of the tagged RTL repository release page. Simulator tools as listed corresponding UVM Simulation User Guides:
- UVM Simulation User Guide: OFS for Agilex\u00ae 7 PCIe Attach
- UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach
- UVM Simulation User Guide: OFS for Stratix\u00ae 10 PCIe Attach
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#3-select-a-starting-shellfim","title":"3 Select a Starting Shell/FIM","text":"To begin your development, start with an existing shell that most closely matches your device and end solution. The OFS site has both Intel Intel Agilex 7 and Stratix 10 FPGA reference designs you can use as a starting point for your own design. These designs can be ported to different device OPNs if a different resource utilization is required.
To begin you may want to learn more about Intel Stratix 10 and Intel Agilex 7 family offerings. Please refer to the following links:
- Intel Agilex 7 Product Page
- Intel Stratix 10 Product Page
Note that each reference design provides an integrated shell, called the FPGA Interface Manager (FIM), which is encompasses in blue in the diagrams below. This shell provides standard AXI interfaces to the Accelerator Functional Unit (AFU) region, shown in green, which depicts a region where a workload or partial reconfiguration slot resides. The regions are not drawn to scale. The figures and tables below give an overview of the available starting points for your design.
Figure 3-1: OFS FIM Targeting Agilex\u00ae7 PCIe Attach (P-tile, E-tile)
Key Feature Description Target OPN AGFB014R24A2E2V PCIe P-tile PCIe* Gen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 5 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each* Four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB Ethernet 2x4x25GbE, 2x4x10GbE or 2x100GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration)* Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Intel\u00ae FPGA SmartNIC N6001-PL Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
Figure 3-2: OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xF-tile)
Key Feature Description Target OPN AGFB027R24C2E2VR2 PCIe P-tile PCIe* Gen4x16 (currently downtrains to Gen4x8 in the ES version of the development kit) Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 3 DDR Channels:* One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 2400 MHz, 1GB each* Two Fabric DDR4 banks, x64 (no ECC), 2400 MHz, 8GB Ethernet 2x4x25GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
Figure 3-3: OFS FIM Targeting Agilex\u00ae7 PCIe Attach (2xR-tile, F-tile)
Key Feature Description Target OPN AGIB027R29A1E2VR3 PCIe PR-tile PCIe 1xGen5x16R-tile PCIe 2xGen5x8R-tile PCIe 1xGen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand Memory 4 Fabric DDR4 channels, x64 (no ECC), 2666 MHz, 8GB Ethernet 2x4x25GbE, 2x200GbE, 2x400GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support * Linux DFL drivers targeting OFS FIMs* OPAE Software Development Kit* OPAE Tools Target Board Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) Click here for the OFS Collateral for Agilex\u00ae 7 FPGA PCIe Attach Reference FIM documentation collection.
Figure 3-4: OFS FIM Features Targeting Agilex\u00ae 7 SoC Attach
Key Feature Description Device OPN AGFC023R25A2E2VR0 PCIe P-tile PCIe* Gen4x16 to the HostP-tile PCIe* Gen4x16 to the SoC (IceLake-D) Virtualization Host: 2 physical functions SoC: 1 physical function and 3 Virtual functions Memory Four Fabric DDR4 banks, x40 (optional ECC, be configured as x32 and ECC x8 ), 1200 MHz, 4GB Ethernet 2x4x25GbE Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) * Platform Controller Management Interface (PMCI) Module for Board Management Controller Partial Reconfiguration Supported Software Support * Linux DFL drivers targeting OFS FIMs * OPAE Software Development Kit * OPAE Tools Target Board Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL Note: Source code for BMC RTL/Firmware that works with this reference FIM can be obtained by contacting your Intel Sales Representative.
Click here for the OFS Collateral for Agilex\u00ae SoC Attach Reference FIM documentation collection.
Figure 3-5: OFS FIM Targeting Stratix\u00ae 10 FPGA PCIe Attach
Key Feature Intel Stratix 10 Reference FIM Device OPN 1SX280HN2F43E2VG Ethernet Configuration 1x10GbE example with 2x100GbE capability PCIe Gen3x16 EMIF Up to four DDR channels PF/VF 1 PF/3 VFs Management FPGA Management Engine (FME) with FIM management registers Interface Arm\u00ae AMBA\u00ae4 AXI Interface HLD support oneAPI Software Kernel code upstreamed to Linux.org Target Board Intel\u00ae FPGA Programmable Acceleration Card D5005 Click here for the OFS Collateral for Stratix\u00ae 10 FPGA PCIe Attach Reference FIM documentation.
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#4-review-release-notes","title":"4 Review Release Notes","text":"Before beginning your design, read the release notes for each repository you are using by selecting the appropriate release tag found by clicking on the \"Release\" link to the right of the corresponding repository. The release page may also have assets appended such as useful binaries, patches or scripts.
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#5-setup-your-environment-for-development","title":"5 Setup Your Environment For Development","text":"When you are ready to begin development you will want to ensure you have any other setup requirements satisfied by reviewing instructions in the corresponding FIM Developer Guide and if you are implementing a OneAPI Board Support Package, the oneAPI ASP Getting Started User Guide as well.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs * Can be used with Agilex\u00ae 7 FPGA R-Series Development Kit (2xR-Tile, 1xF-Tile)
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs * Can be used with Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs * Can be used with Intel\u00ae FPGA SmartNIC N6001-PL
- Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs * Can be used with Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL
- Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs * Can be used with Intel\u00ae FPGA PAC D5005
For oneAPI setup:
- oneAPI Accelerator Support Package (ASP): Getting Started User Guide * Can be used with: - Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) - Intel\u00ae FPGA SmartNIC N6001-PL
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#6-customize-your-shell-build-and-debug","title":"6 Customize Your Shell, Build and Debug","text":"For your own custom design, you may need to:
- Target the OFS reference design to a different device
- Modify the pin assignments
- Modify or remove existing peripheral IP
- Add new interface IP
- Repartition the partial reconfiguration region
The FIM Developer Guides for each reference FIM show you how to make these changes and provide steps on how to run unit simulation or add SignalTap to your design for debugging.
If you are also interested in testing different examples for the Acceleration Functional Unit (AFU) or workload region of your design or creating your own AFU design, you can refer to the corresponding AFU Developer Guides:
- Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs
- Workload Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#7-simulate-and-debug","title":"7 Simulate and Debug","text":"Setup and test files to perform system-level Universal Verification Methodology (UVM) testing are provided in each FIM RTL repository. Please refer to the corresponding UVM Simulation User Guide for details on test bench architecture, setup and testing.
- UVM Simulation User Guide: OFS for Agilex\u00ae 7 PCIe Attach
- UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach
- UVM Simulation User Guide: OFS for Stratix\u00ae 10 PCIe Attach
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#8-optional-build-oneapi-accelerator-support-package-asp","title":"8 Optional: Build OneAPI Accelerator Support Package (ASP)","text":"If you are considering providing oneAPI support for your custom board design, you must integrate the oneAPI ASP hardware and software components into your design. Reference the following documents to learn about the architecture and implementation flow for the oneAPI ASP with an OFS design.
- oneAPI Accelerator Support Package (ASP): Getting Started User Guide
- oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#9-optional-driver-or-ofs-software-tool-modifications","title":"9 Optional: Driver or OFS Software Tool Modifications","text":"As you add or remove interfaces to your custom design, you may need to modify or enhance existing drivers that accompany the OFS reference design. Additionally, you may decide you want to create additional utilities or plugins leveraging the OFS software infrastructure. In this case, refer to the Software Reference Manual: Open FPGA Stack to learn more about the underlying driver and software architecture and how to make modifications.
Additionally, for guidance on using a Kernel-based Virtual Machine with OFS, refer to our KVM User Guide.
KVM User Guide: Open FPGA Stack
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#10-test-and-deploy","title":"10 Test and Deploy","text":"When testing and deploying your platform you are encouraged to modify and tailor the evaluation scripts you used in Section 2 to assist in automating your continuous integration flow. You may also want to refer to our Docker User Guide to understand how to use Docker for Development and Deployment.
- Docker User Guide: Open FPGA Stack
"},{"location":"hw/common/user_guides/ug_fpga_developer/ug_fpga_developer/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
- Install the necessary tools, such as virt-manager, on the host machine. This may involve downloading and installing the software from the internet.
- Enable the virtualization feature on the host machine. This may involve going into the BIOS settings and enabling hardware-assisted virtualization or using a command-line tool to enable it in the operating system.
- Use virt-manager to create a new virtual machine and configure its settings. This may involve choosing a name and operating system for the virtual machine and setting the amount of memory and storage it will use.
- Install the OPAE (Open Programmable Acceleration Engine) tool on the virtual machine. This may involve downloading and installing the OPAE software.
- Install the DFL (Data Field Level) drivers on the virtual machine. These drivers allow the virtual machine to access and use the PCIe devices on the host machine. This may involve downloading and installing the drivers from the internet.
- Once all of the steps have been completed, you should be able to use the virtual machine to access and use the PCIe devices on the host machine. You may need to configure the virtual machine's settings to enable it to use the PCIe devices, such as by assigning a specific device to the virtual machine.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#1-modes-of-operation","title":"1. Modes of Operation","text":"Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
-
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
-
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#2-enable-virtualization","title":"2. Enable Virtualization","text":"To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\n
This command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\n
In this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":" - Open a terminal window and log in as a user with sudo privileges.
- Check if the virtualization kernel modules are loaded by running the following command:
lsmod | grep kvm\n
-
If the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
-
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\n
- If the kernel modules are not loaded, and you cannot load them manually, it may be because virtualization is not supported or enabled in your system's BIOS or UEFI settings. You must reboot your system and enter the BIOS or UEFI settings menu to enable virtualization. The exact steps for doing this may vary depending on your system's hardware and BIOS/UEFI version, so consult your motherboard or system documentation for specific instructions.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#4-install-virtual-machine-manager","title":"4. Install Virtual Machine Manager","text":"Virtual Machine Manager (also known as libvirt) can be installed by following the below steps:
- Open a terminal window and log in as a user with sudo privileges.
-
Update your system package index by running the following command:
- Redhat
sudo dnf update\n
* Ubuntu sudo apt update\n
-
Install the libvirt package and any required dependencies by running the following command:
- Redhat
sudo dnf install @virtualization\n
- Ubuntu
sudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\n
-
Start the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\n
- Optional: Install the virt-manager package, which provides a GUI application for managing virtual machines, by running the following command:
sudo dnf install virt-manager\n
- Optional: If you want to be able to run virtual machines as a non-root user, add your user to the libvirt group by running the following command, replacing \"USERNAME\" with your username:
sudo usermod -a -G libvirt USERNAME\n
- You can now launch virt-manager by running the command
virt-manager as the non-root user.
Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\n
You will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\n
- Reboot your server to apply the changes
reboot\n
After completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
- Start the
virt-manager GUI by running the following command:
sudo virt-manager&\n
-
Create a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
-
In the virt-manager window, click the \"New virtual machine\" button.
-
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
-
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
-
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
-
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
-
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
-
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
-
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
-
Click on the \"+\" icon (Bottom left) to create the storage pool.
-
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
-
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
-
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
-
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
-
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#51-passing-devices-to-the-vm","title":"5.1 Passing Devices to the VM","text":"In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n
\u200b
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
The following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Ensure you add the desired VF in your PCIE devices list.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#513-2-pf-mode","title":"5.1.3 2 PF Mode","text":"For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
-
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n
-
Pass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
-
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\n
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#52-virt-manager-configuration-changes","title":"5.2 Virt-Manager Configuration Changes","text":" -
Edit the XML file for your machine and include the following
-
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\n
-
Inside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\n
-
Ensure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\n
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Save the changes \"Apply\"
-
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\n
-
Ensure your devices are enumerated properly.
-
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n
2. Deployment Mode:
B1:00.5\n
-
Under the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n
2. Deployment Mode:
177:00.0\n
-
Click on \"Begin Installation.\" and follow the wizard installation of the OS.
-
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
-
Under your virtual machine, configure your VM proxy:
- Redhat How to apply a system-wide proxy?
- Ubuntu Define proxy settings
- Configure Git to use a proxy
-
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
-
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\n
-
Use the Virtual function (Not supported at management mode)
-
Ensure the DFL kernel drivers is install in your VM system
-
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\n
-
Verify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n
-
Test the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\n
After the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
"},{"location":"hw/common/user_guides/ug_kvm/ug_kvm/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/","title":"UVM Simulation User Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel Max10 or Intel Cyclone10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implemented on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Management Component Transport Protocol MCTP A standardized model for communication with management controllers. Defines the transport protocol carrying PLDM messages through the BMC. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE-SDK The OPAE-SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Level Data Model PLDM A specification for reporting telemetry data to the host, such as board temperature, voltage, and current. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#1-overview","title":"1 Overview","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the UVM simulation tool using Open FPGA Stack (OFS) for Agilex\u00ae 7 FPGAs PCIe Attach and the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). After reviewing the document, you will be able to:
- Set-up the UVM verification tool suite
- Run pre-existing UVM unit tests and also create new UVM tests for your design
NOTE:
This guide uses the Intel\u00ae FPGA SmartNIC N6001-PL as the platform for all example steps. Additionally, this guide and the example steps can be used with other platforms, such as the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile).
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#2-introduction-to-uvm","title":"2 Introduction to UVM","text":"The Functional Verification Environment for OFS is UVM (Universal Verification Methodology) compliant and provides configurable setup for verifying various FIM features in simulation.
The purpose of this document is to demonstrate a full chip level and unit level UVM based verification environment for the current base shell FIM architecture as well as providing examples to extend the setup for different OFS variants by reusing the existing architecture and infrastructure for UVM based verification.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#3-universal-testbench-architecture","title":"3 Universal Testbench Architecture","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#31-overview","title":"3.1 Overview","text":"The main idea behind UVM is to develop modular, reusable, and a scalable testbench structure by providing an API framework that can be deployed across multiple projects.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#32-core-verification-concepts","title":"3.2 Core Verification Concepts","text":"The following is the list of verification components that will be used to design a UVM testbench architecture:
\u2022 Sequencer \u2022 Driver \u2022 Monitor \u2022 Scoreboard \u2022 Virtual Sequencer \u2022 Interface \u2022 Verification Environment \u2022 TOP Testbench
Figure 1 provides the general UVM Testbench and the verification components involved in the top-level architecture.
Figure 1 Typical UVM Testbench
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4-ofs-testbench-architecture","title":"4 OFS Testbench Architecture","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#41-overview","title":"4.1 Overview","text":"OFS (Open FPGA Stack) provides a UVM (Universal Verification Methodology) environment for the FIM with a modular, reusable, and scalable testbench structure via an API framework.
The framework consists of a FIM Testbench which is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this Testbench. UVM RAL(Register Abstraction Level) is used for CSR (Command and Status Registers) verification.
The qualified verification IPs will help to detect incorrect protocol behavior, help to focus on FIM features and accelerate the verification process.
Verification components include:
\u2022 FIM monitor to detect correct design behavior\n\u2022 FIM assertions for signal level integrity testing\n\u2022 Arm AMBA Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity\n\u2022 FIM coverage to collect functional data\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#42-base-fim-dut","title":"4.2 Base FIM DUT","text":"The hardware architecture of an Agilex FIM is based on the OFS hardware architecture. The following is the list of features and subsystems supported in the base shell.
\u2022 PCIe Subsystem\n\u2022 HSSI Subsystem\n\u2022 Memory Subsystem\n\u2022 HPS Subsystem\n\u2022 FME\n\u2022 AFU with PR support\n\u2022 QSFP Controllers\n\u2022 PMCI Controller, MCTP\n
Figure 2 DUT Base Shell Diagram
Figure 2 shows the high level architecture of an Agilex Base Shell. It has a Gen4x16, 100G Ethernet Datapath in a 2x4x25G configuration. The Agilex Base Shell is a shell that will enable a user to build other shell variants for a custom configuration. For the N6001 board there is one shell variant
base_x16
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43-full-chip-level-verification-archiecture-for-fim","title":"4.3 Full Chip Level Verification Archiecture for FIM","text":"Figure 3 shows a graphical representation a full chip testbench that includes major RTL blocks depicted in a OFS Agilex based UVM environment
Figure 3 OFS FIM Testbench
The major connection is the interface between the Xeon CPU and the FPGA where the PCIe Verification IP is connected to PCIe Subsystem. Therefore, as a full chip simulation environment, PCIe host VIP is the sole VIP/BFM used. PCIe host VIP connects to PCIe device which resides in FPGA in serial mode.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#431-testbench-components","title":"4.3.1 Testbench components","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4311-tb-top","title":"4.3.1.1 TB TOP","text":"TOP is the top level testbench and consists of a FIM DUT instance and top-level UVM Verification Environment instance. It binds RTL inputs with the verification environmnet interfaces to drive stimulus. It also has clock generation and reset driving logic.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4312-fim-verification-environment","title":"4.3.1.2 FIM Verification Environment","text":"This is the top most verification environment class and consists of the protocol specific PCI Express and AXI UVM environment VIP instances, Monitors, Scoreboards, Assertions, Functional coverage Modules and other verification components. It instantiates Virtual sequencers to control stimuli for FIM traffic from different sequencers of the VIPs.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4313-synopsys-vips","title":"4.3.1.3 Synopsys VIPs","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43131-pci-vip-as-host","title":"4.3.1.3.1 PCI VIP as Host","text":"This is Synopsys Licensed PCI Express Gen4 VIP and acts as Root Port. The PCI Express link is connected to the DUT using TX-RX lanes. Agent is an active component and includes a sequencer to generate TLPs, Driver to drive it on a PCI Express link and Monitor to check the protocol correctness.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43132-axi-streaming-vip-monitors","title":"4.3.1.3.2 AXI-Streaming VIP Monitors","text":"This is Synopsys Licensed AXI streaming interface Verification IP used as a Passive Agent to monitor AXI-ST links at various points. Please refer to Figure 3 to see all the AXI-ST monitors connected to different modules.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43133-axi4-memory-mapped-vip-monitors","title":"4.3.1.3.3 AXI4-Memory Mapped VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 memory mapped interface Verification IP used in passive mode to observe memory requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and performing data integrity checks.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43134-axi4-lite-vip-monitors","title":"4.3.1.3.4 AXI4-Lite VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used in passive mode to observe MMIO requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and perfoming data integrity checks. Please refer to Figure 3 to see all the Arm\u00ae AMBA\u00ae 4 AXI4-Lite monitors connected to different modules.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#43135-axi4-lite-vip-as-pmci-master","title":"4.3.1.3.5 AXI4-Lite VIP as PMCI Master","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used to drive and observe MMIO requests as PMCI master. For each master-slave pair, the verification environment has a VIP instantiated in active mode and includes a sequencer to generate MMIO requests, driver to drive these requests on AXI-4 Lite interface to BPF and a monitor for observing protocol violations and data integrity checks.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4314-axi4-s-scoreboard","title":"4.3.1.4 AXI4-S Scoreboard","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-S scoreboard checks data integrity of source and sink components. It has input transactions from VIP monitors and a TLP to AXI converter for PCIe TLP packets. It makes sure the valid TLPs or AXI transactions are not missed while traversing from Host to AFU and reverse. The scoreboard will be disabled for error testing especially for invalid TLP requests and UR responses.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4315-virtual-sequencer","title":"4.3.1.5 Virtual Sequencer","text":"The virtual sequencer is the top-level sequencer which controls Enumeration, MMIO Requests, downstream and Upstream traffic as well as HSSI and Memory transactions. It makes sure that the transactions are ordered correctly as per the design specification while running a UVM test simulation.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4316-fim-monitor","title":"4.3.1.6 FIM Monitor","text":"The FIM monitor is used for checking the correctness of a specific FIM feature. As an example, a user can add interrupt related checks, error generation and clearing checks, Protocol checker impacts etc. in this component.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4317-fim-assertions","title":"4.3.1.7 FIM Assertions","text":"The assertion component is used for checking the signal level integrity of a specific FIM feature or behavior. As an example, we can add interrupt signal triggering and clear assertions, Error detection to error register bit assertions, protocol checker and clearing logic, FLR behavior assertion etc. in this top-level module. There are alos assertions to make sure the inputs are driven correctly to catch errors in a users simulation.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#4318-fim-functional-coverage","title":"4.3.1.8 FIM Functional Coverage","text":"The FIM functional coverage component will have the functional coverage of each feature like interrupts, CSR's, MMIO requests, Byte align transactions, error reporting etc. to demonstrate a variety of FIM features. This will help us to find holes in the design as well as wide verification coverage to make sure thorough testing of a design. It will also provide a user what the FIM is capable of and how much code coverage they are testing.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#5-uvm-verification-set-up","title":"5 UVM Verification set-up","text":"To run the tutorial steps in this guide requires the following development environment:
Item Version Intel Quartus Prime Pro Intel Quartus Prime Pro 24.1 Simulator Synopsys VCS S-2023.03-SP2-1 or newer for UVM simulation of top level FIM"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#51-uvm-prerequisite","title":"5.1 UVM Prerequisite","text":"Retrieve OFS repositories.
The OFS FIM source code is included in the OTCShare GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories. Cloning the repo using the HTTPS method requires a personal access token. Please see this blog post for information about obtaining a personal access token Token authentication requirements for Git operations.
Navigate to location for storage of OFS source, create the top-level source directory and clone OFS repositories.
$ mkdir ofs-2024.2-1\n$ cd ofs-2024.2-1\n$ export OFS_BUILD_ROOT=$PWD\n$ git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n\nCloning into 'ofs-agx7-pcie-attach'...'\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\n\n$ cd ofs-agx7-pcie-attach\n$ git checkout tags/ofs-2024.2-1\n
Verify that the correct tag/branch have been checked out
$ git describe --tags\n\n$ ofs-2024.2-1\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#52-license-requirements","title":"5.2 License Requirements","text":"The FIM Testbench is UVM compliant and integrates third party VIP's from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this TB. UVM RAL (Register Abstraction Layer) is used for CSR Verification.
The Qualified Verification IPs will help to detect incorrect protocol behavior easily, help to focus on FIM features and accelerate the verification process.
\u2022 VCS & DVE\n\u2022 SNPS-Assertions\n\u2022 Verdi\n\u2022 VerdiCoverage\n\u2022 VerdiSimDB\n\u2022 VerdiTransactionDebugUltra\n\u2022 VIP-AMBA-AXI-SVT\n\u2022 VIP-AMBA-STREAM-SVT\n\u2022 VIP-PCIE-SVT\n\u2022 VIP-PCIE-TS-SVT\n\u2022 VIP-PCIE-G3-OPT-SVT\n\u2022 VIP-Ethernet-SVT\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#53-software-tools-requirements","title":"5.3 Software Tools Requirements","text":"The following tools are required for successful UVM set-up
- Python 3.6.8
- Synopsys PCIE and AMBA AXI UVM VIP Q-2020.03A License
- Synopsys Verdi S-2023.03-SP2-1 License Note: Makefile can be modified to use DVE instead of Verdi
- VCS S-2023.03-SP2-1 License
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#54-creating-a-software-tools-script","title":"5.4 Creating a Software Tools Script","text":"The UVM tool set-up is best done by creating a simple set-up script so all applicable tools are sourced before running the tests.
The following environment variables can be pasted into a script and used prior to running the UVM verification environment
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#license-files","title":"License Files","text":"export LM_LICENSE_FILE=\nexport SNPSLMD_LICENSE_FILE=\n
The license environment variables LM_LICENSE_FILE and SNPSLMD_LICENSE_FILE can point to a server license on your system.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#general-environment-variables","title":"General Environment Variables","text":"export IOFS_BUILD_ROOT=$PWD\nexport OFS_ROOTDIR=<user_path>/ofs-agx7-pcie-attach\nexport WORKDIR=$OFS_ROOTDIR\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#quartus-tools","title":"Quartus Tools","text":"export QUARTUS_HOME=<user_path>/intelFPGA_pro/24.1/quartus\nexport QUARTUS_ROOTDIR=$QUARTUS_HOME\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#synopsys-verification-tools","title":"Synopsys Verification Tools","text":"export DESIGNWARE_HOME=<user_path>/synopsys/vip_common/vip_Q-2020.03A\nexport PATH=$DESIGNWARE_HOME/bin:$PATH\nexport UVM_HOME=<user_path>/synopsys/vcsmx/S-2023.03-SP2-1/linux64/rhel/etc/uvm\nexport VCS_HOME=<user_path>/synopsys/vcsmx/S-2023.03-SP2-1/linux64/rhel\nexport PATH=$VCS_HOME/bin:$PATH\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#6-running-a-uvm-simulation-test-and-analysing-results","title":"6 Running a UVM Simulation Test and Analysing Results","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#61-simulation","title":"6.1 Simulation","text":"The default simulator used in the simulation script is Synopsys VCS-MX. Users can refer to the options and adopt the options for other simulators. The script is a makefile that calls vlogan, vcs and simv for compilation, elaboration and simulation, respectively.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#62-file-structure","title":"6.2 File Structure","text":"After cloning the repo, the verification and ofs-common directories contain all UVM verification related files. The directory structure is shown in Figure 4 below.
Figure 4 UVM Verification Directory File Structure
ofs-agx7-pcie-attach/verification/testbench has a testbench, uvm env, virtual sequencer, RAL etc.
ofs-agx7-pcie-attach/tests contains all uvm tests and sequences.
Users can run the simulation under \"ofs-agx7-pcie-attach/verification/scripts\" directory and the simulation result is outputted to a \"sim\" directory for Synopsys VCS.
The simulation result folder is named after the test name with increasing suffix number. If user runs the same test multiple times, the suffix is incremented by 1 each time.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#63-uvm-test-suite","title":"6.3 UVM Test Suite","text":"The UVM environment contains a variety of tests that have been developed to test out the FIM portion of OFS.
The table below has four columns which describe the \"Test Name\", \"DUT Scope\", \"Test Scenario\" and the \"Checking Criteria\".
Tests are located at ofs-agx7-pcie-attach/verification/tests
Test Name DUT Scope Test Scenario Checking Criteria afu_mmio_flr_pf0_test PF0 FLR Reset Apply FLR Reset for PF0 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF0 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf0_test PF0_VF0_FLR Reset Apply FLR Reset for PF0_VF0 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf1_test PF0_VF1_FLR Reset Apply FLR Reset for PF0_VF1 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf2_test PF0_VF2_FLR Reset Apply FLR Reset for PF0_VF2 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf2_test PF2 FLR Reset Apply FLR Reset for PF2 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF2 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf3_test PF3 FLR Reset Apply FLR Reset for PF3 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF3 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf4_test PF4 FLR Reset Apply FLR Reset for PF4 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF4 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_stress_5bit_tag_test AFU-Stress To check the AFU Stress by sending traffic from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_8bit_tag_test AFU-Stress To check the AFU Stress by sending traffic from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_test AFU-Stress 1. Initiate transactions to all the supported PF/VF from PCIE VIP and ensure that traffic is sent to all blocks of the AFU. 2. Ensure that CE/HE-LB/HE-MEM/HSSI/BPF/FME are seeing traffic. 3. Ensure that HE-LB/HE-MEM/CE sends DMWR/DMRD requests to PCIE VIP. 4. Ensure the Mux/DeMux blocks is able to handle the traffic based on the PF's/VF's and proper muxing/demuxing happens. Data checking bar_32b_test PCIe MMIO Path Set the BAR values to 32bit and test mmio access BAR address, Register Base Offset bar_64b_test PCIe MMIO Path Set the BAR values to 64bit and test mmio access BAR address, Register Base Offset dfh_walking_test DFH DFH walking offset checking, eol checking -> tb emif_csr_test EMIF CSR access data checking fme_csr_test FME CSR CSR accesses data checking fme_err_intr_test Interrupt FME Interrupt request using FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_intr_test Interrupt FME interrupt request using RAS ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_ras_cat_fat_err_test FME Error FME This test verifies the RAS fatal error register in FME. RAS error is generated by forcing/writing into the error register. MSI Generation is verified via PBA mechanism by masking and unmasking interrupt vector Error Checking fme_ras_no_fat_err_test FME Error This test verifies the RAS no-fatal error register in FME. RAS error is generated by forcing/writing into the error register. MSI Generation is verified via PBA mechanism by masking and unmasking interrupt vector Error Checking fme_multi_err_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_hssi_csr_test HE-HSSI CSR accesses data checking he_hssi_rx_lpbk_25G_10G_test HE-HSSI Sending back to back ethernet data traffic with 25G speed on HSSI RX Port0-7 lanes using Ethernet VIPs Enable the loopback mode in HE-HSSI and compare the pkts recived on HSSI TX Port(DATA CHECKING) he_hssi_tx_lpbk_P0_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port0 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P1_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port1 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P2_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port2 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P3_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port3 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P4_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port4 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P5_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port5 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P6_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port6 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P7_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port7 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_lpbk_cont_test HE-LPBK Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_lpbk_csr_test HE-LPBK CSR accesses data checking he_lpbk_flr_rst_test HE-LPBK Set HE_LPK in all the modes randomly and iterate the transactions in loop. At the end of every loop assert the Soft reset in the middle of the transactions data checking, counter checking he_lpbk_long_rst_test HE-LPBK Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_lpbk_long_test HE-LPBK Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_lpbk_multi_user_intr_test HE-LPBK generate user HE_LB interrupt interrupt checking he_lpbk_rd_cont_test HE-LPBK Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_rd_test HE-LPBK Read only mode. Randomize num_lines, addresses, req_len counter checking he_lpbk_reqlen16_test HE-LPBK To check the behavior of HE_LPK block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_lpbk_reqlen1_test HE-LPBK Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_lpbk_reqlen2_test HE-LPBK Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_lpbk_reqlen4_test HE-LPBK Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_lpbk_reqlen8_test HE-LPBK Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_lpbk_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_thruput_contmode_test HE-LPBK Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking he_lpbk_thruput_test HE-LPBK generate user HE_LB interrupt counter checking he_lpbk_user_intr_test HE-LPBK Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_lpbk_wr_cont_test HE-LPBK Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_wr_test HE-LPBK Write only mode. Randomize num_lines, addresses, req_len counter checking he_mem_cont_test HE-MEM Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking, counter checking he_mem_csr_test HE-MEM CSR accesses data checking he_mem_lpbk_long_rst_test HE-MEM Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_mem_lpbk_long_test HE-MEM Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_mem_lpbk_reqlen16_test HE-MEM To check the behavior of HE_MEM block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_mem_lpbk_reqlen1_test HE-MEM Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_mem_lpbk_reqlen2_test HE-MEM Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_mem_lpbk_reqlen4_test HE-MEM Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_mem_lpbk_reqlen8_test HE-MEM Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_mem_lpbk_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_multi_user_intr_test Interrupt Back to back multiple User interrupt request from HE MEM Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read for multiple back to back request he_mem_rd_cont_test HE-MEM Read only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_test HE-MEM Read only mode. Randomize num_lines, addresses, req_len counter checking he_mem_thruput_contmode_directed_test HE-MEM Set HE_LPK in thruput mode and send traffic with req len 1 and num_lines set to 40 data checking, counter checking he_mem_thruput_contmode_test HE-MEM Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_thruput_test HE-MEM Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_mem_user_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_mem_wr_cont_test HE-MEM Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_wr_test HE-MEM Write only mode. Randomize num_lines, addresses, req_len counter checkingt he_mem_flr_rst_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len Read the Control and Status registers before FLR Reset. Apply PF0VF0 FLR reset and check again the control and status registers. Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from HE-MEM he_random_test All HEs Enable all HEs and randomize modes data checking if in lpbk mode, counter checking helb_rd_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Read only mode Measure the performance helb_rd_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Read only mode Measure the performance helb_rd_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Read only mode Measure the performance helb_thruput_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_4cl_5bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_8bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Thruput mode Measure the performance helb_wr_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Write only mode Measure the performance helb_wr_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Write only mode Measure the performance helb_wr_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Write only mode Measure the performance hssi_ss_test HSSI CSR accesses data checking malformedtlp_pcie_rst_test Protocol Checker generate malformed TLP protocol error and initiate pcie reset to clear the error Check for error malformedtlp_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MaxTagError_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transactions are completing. mem_tg_csr_test MEM-TG CSR access data checking mem_tg_traffic_gen_test MEM-TG This test checks the MEM_TG traffic generator flow for 1 bank data checking mini_smoke_test All HEs shorter simpler version of random test for turn-in sanity check data checking if in lpbk mode, counter checking mix_intr_test Interrupt Mix interrupt testcase to send multiple FME and User interrupt request simultaneously Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests through different sources - FME and HE-MEM modules mmio_pcie_mrrs_128B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_128B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Read Checking valid possible combination of MPS & MRRS mmio_stress_nonblocking_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux with non-blocking MMIO reads data checking mmio_stress_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux data checking mmio_test PCIe MMIO Path MMIO targeting PF0(ST2MM, FME, PMCI, QSFP, HSSI SS), PF1, PF2,PF3, PF4, PF1.VF1, PF1.VF2 data checking mmio_unimp_test PCIe MMIO Path MMIO acccess to unimplemented addresses data checking MMIODataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOTimedout_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. pcie_csr_test PCIESS Earlier msix registers were in fme block but now it has moved from fme to pciess. Hence coded a seperate test for msix data checking pcie_pmci_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pcie_pmci_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM (Vendor Defined Message) single packet received from PCIe HIA subsystem to PMCI controller over APF and BPF via AXI4-lite memory write request pmci_csr_test PMCI CSR CSR accesses data checking pmci_fme_csr_test PMCI FME CSR CSR accesses data checking pmci_pcie_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pcie_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM single packet received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pciess_csr_test PMCI PCIESS CSR CSR accesses data checking pmci_qsfp_csr_test PMCI QSFP CSR CSR accesses data checking port_gasket_csr_test PORT GASKET CSR accesses data checking qsfp_csr_test QSFP CSR accesses data checking TxMWrDataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. TxMWrInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. uart_intr_test UART Checking Generates UART interrupt Check interrupt UnexpMMIORspErr_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. protocol_checker_csr_test Protocol Checker CSR access to Protocol Checker data checking vdm_err_vid_test Vendor ID check generate a packet with undefined Vendor-ID from Host to PMCI_SS ID check The next section describes how to compile and build the UVM environment prior to running each UVM test and analyinsg the results in the log files
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#64-ip-compile","title":"6.4 IP Compile","text":"To compile all IPs for the Synopsys VCS simulater targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk cmplib_adp\n
To compile all IPs for the Synopsys VCS simulater targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk cmplib_adp FTILE_SIM=1\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#65-rtl-test-bench-compile","title":"6.5 RTL & Test Bench Compile","text":"The RTL file list for compilation is located here: verification/scripts/rtl_comb.f
The TB file list for compilation is located here: verification/scripts/ver_list.f
To compile RTL and Testbench for the Synopsys VCS simulater targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_adp DUMP=1\n
To compile RTL and Testbench for the Synopsys VCS simulater targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_adp FTILE_SIM=1\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#66-ip-and-rtl-test-bench-compile","title":"6.6 IP and RTL & Test Bench Compile","text":"If the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS targetting the Intel\u00ae FPGA SmartNIC N6001-PL then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_all\n
If the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_all FTILE_SIM=1\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#67-running-individual-testcases","title":"6.7 Running Individual Testcases","text":"To run a simulation for Synopsys VCS targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk run TESTNAME=ofs_mmio_test\n
To run a simulation for Synopsys VCS targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk run TESTNAME=ofs_mmio_test FTILE_SIM=1\n
To dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation targetting the Intel\u00ae FPGA SmartNIC N6001-PL:
gmake -f Makefile_VCS.mk build_adp DUMP=1\n\n gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> DUMP=1\n
Or
gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> DUMP=1\n
To dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile):
gmake -f Makefile_VCS.mk build_adp FTILE_SIM=1 DUMP=1\n\n gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> FTILE_SIM=1 DUMP=1\n
Or
gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> FTILE_SIM=1 DUMP=1\n
There are some optimizations in the Table below for convenience if you want to bypass some commands for both Synopsys VCS:
Command (Synopsys VCS) for n6001 * Command (Synopsys VCS) for fseries-dk Details gmake -f Makefile_VCS.mk build_all DUMP=1 gmake -f Makefile_VCS.mk build_all FTILE_SIM=1 DUMP=1 compile IP + compile RTL gmake -f Makefile_VCS.mk build_run TESTNAME= DUMP=1 gmake -f Makefile_VCS.mk build_run TESTNAME= FTILE_SIM=1 DUMP=1 compile RTL + run test gmake -f Makefile_VCS.mk do_it_all TESTNAME= DUMP=1 gmake -f Makefile_VCS.mk do_it_all TESTNAME= FTILE_SIM=1 DUMP=1 compile IP, RTL and run test gmake -f Makefile_VCS.mk rundb TESTNAME= DUMP=1 gmake -f Makefile_VCS.mk rundb TESTNAME= FTILE_SIM=1 DUMP=1 run test in sim dir + over-writes content Note: * For N6000, use flag DISABLE_EMIF=1 for the tests to run successfully
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#67-uvm-regression-test","title":"6.7 UVM Regression Test","text":"If the user wants to run the complete set of UVM tests in one command for VCS targetting the Intel\u00ae FPGA SmartNIC N6001-PL then follow the procedure below
cd $VERDIR/scripts\n\npython uvm_regress.py -l -n 8 -p adp -k <pkg_name> -s vcs -c none\n
If the user wants to run the complete set of UVM tests in one command for VCS targetting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) then follow the procedure below
python uvm_regress.py -l -n 8 -p adp -k <pkg_name> -s vcs -c none -e -t ftile\n
Test Packages for Intel\u00ae FPGA SmartNIC N6001-PL and Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) are listed below:
<pkg_name> :top_pkg (it contains two packages test_pkg + test_long_pkg)\n :test_pkg\n :test_long_pkg\n :tx_pkg\n :tx_pkg_100G_200G\n :tx_pkg_400G\n :rx_pkg\n :rx_pkg_100G\n :rx_pkg_400G\n
Test Package files are located in tests directory ($VERDIR/tests) please go through the list of testcases that package file contains before running the regression test
Results are created in a sim directory ($VERDIR/sim) with individual testcase log dir
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#68-uvm-waveform-and-transcript-analysis","title":"6.8 UVM Waveform and Transcript Analysis","text":"Running Synopsys VCS UVM tests will generate a ofs-agx7-pcie-attach/verification/sim directory
\u2022 All build time logs are located at ofs-agx7-pcie-attach/verification/sim\n\n\u2022 Each testcase will have separate directory inside sim ofs-agx7-pcie-attach/verification/sim/<test_case_name>\n
There are two tracker or log files that are available: runsim.log and trans.log.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#runsimlog","title":"runsim.log","text":"runsim.log is the simulation log file generated from Synopsys VCS. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 5
Figure 5 runsim.log
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#translog","title":"trans.log","text":"trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 6
Figure 6 trans.log
The waveform generated is named as \"inter.vpd\". To open the waveform, go to simulation result directory and run
dve -full64 -vpd inter.vpd &\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#69-uvm-coverage-analysis","title":"6.9 UVM Coverage Analysis","text":"For Intel\u00ae FPGA SmartNIC N6001-PL
The following command allows to run a single testcase with coverage enabled
gmake -f Makefile_VCS.mk cmplib_adp && gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1 COV_FUNCTIONAL=1&& gmake -f Makefile_VCS.mk run TESTNAME=<TESTCASE-NAME> DUMP=1 DEBUG=1 COV_FUNCTIONAL=1 &\n
The following command shows how to merge and generate the coverage report
urg -dir <$VERDIR/sim/simv.vdb> <$VERDIR/sim/regression.vdb> -format both -dbname <regression_database_name>\n
This will generate both urgreport directory and .vdb file Multiple regression.vdb from different regressions can be merged with the same command.
e.g \"urg -dir <path1_till_simv.vdb> <path1_till_regression.vdb> <path2_till_regression.vdb> -report <dir> -format both -dbname <dirname>\"\n
The following commands shows how to launch DVE and check the coverage reports
To open DVE of a single regression or testcase, execute:\n\n dve -full64 -cov -covdir simv.vdb regression.vdb &\n\nTo open DVE of a merged regression test, execute:\n\n dve -full64 -cov -covdir <dirname.vdb> &\n
Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
The following command allows to run a single testcase with coverage enabled
gmake -f Makefile_VCS.mk cmplib_adp FTILE_SIM=1 && gmake -f Makefile_VCS.mk build_adp FTILE_SIM=1 DUMP=1 DEBUG=1 COV_FUNCTIONAL=1&& gmake -f Makefile_VCS.mk run TESTNAME=<TESTCASE-NAME> FTILE_SIM=1 DUMP=1 DEBUG=1 COV_FUNCTIONAL=1 &\n
The following command shows how to merge and generate the coverage report
urg -dir <$VERDIR/sim/simv.vdb> <$VERDIR/sim/regression.vdb> -format both -dbname <regression_database_name>\n
This will generate both urgreport directory and .vdb file Multiple regression.vdb from different regressions can be merged with the same command.
e.g \"urg -dir <path1_till_simv.vdb> <path1_till_regression.vdb> <path2_till_regression.vdb> -report <dir> -format both -dbname <dirname>\"\n
The following commands shows how to launch DVE and check the coverage reports
To open DVE of a single regression or testcase, execute:\n\n dve -full64 -cov -covdir simv.vdb regression.vdb &\n\nTo open DVE of a merged regression test, execute:\n\n dve -full64 -cov -covdir <dirname.vdb> &\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#7-csr-verification-using-uvm-ral","title":"7 CSR Verification using UVM RAL","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#71-overview","title":"7.1 Overview","text":"The UVM Register Layer provides a standard base class library that enable users to implement the object-oriented model to access the DUT registers and memories. The UVM Register Layer is also referred to as UVM Register Abstraction Layer (UVM RAL). Design registers can be accessed independently of the physical bus interface. i.e. by calling read/write methods. This is shown in Figure 9 below.
Figure 9 RAL UVM Testbench
The RAL register models for different CSR's mimics the design registers. All RAL files are located here.
ofs-agx7-pcie-attach/verification/testbench/ral\n
The RAL model is generated through the Synopsys RALGEN tool and is used for CSR verification.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#72-ral-integration","title":"7.2 RAL Integration","text":"For UVM RAL model integration to the environment, adapters for each CSR is implemented and integrated into the Testbench Environment. It is used to convert the PCIe bus sequence items into uvm_reg_bus_op and vice versa. The CSR test cases pick up all the registers from the respective CSR blocks and perform a default value, wr/rd check on them.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#73-ral-model-generation","title":"7.3 RAL Model Generation","text":"Steps for RAL model generation
Excel(xls) file containing the registers is required. Make sure there are separate xls files for each CSR and the worksheet name must contain the name \"reg_fields\".
Excel sheet snapshot example below for EMIF_CSR.xls located at /ipss/mem/rtl/adp
\u2022 Navigate to ofs-agx7-pcie-attach/ofs-common/verification/common/scripts/ral\n\u2022 Copy the excel file (xls) to the above area\n\u2022 In the bash terminal run mk_ral.sh <Excel sheet name without extension > <output *.sv file name without ral_ prepended >\n\u2022 The above steps generate two ral *.sv files. File with _cov suffix is a coverage enabled ral model. \n\u2022 Copy *.sv files to ofs-agx7-pcie-attach/verification/testbench/ral\n
\u2022 As an example to generate ral_ac_ce.sv from AC_CE_CSR.xls file the command is\n\n mk_ral.sh AC_CE_CSR ac_ce\n
This generates two ral models (ral_ac_ce.sv and ral_ac_ce_cov.sv)
To add new registers
\u2022 To create new registers, copy existing ones and modify the cells in the xls. Make sure the last line is also a copied blank line\n\u2022 Follow all the steps of RAL model generation\n
To Generate a RAL model when a new xls sheet is created for a new component
\u2022 Copy the relevant xls sheet to ofs-agx7-pcie-attach/ofs-common/verification/common/scripts/ral\n\u2022 Follow all the steps of RAL model generation\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#74-top-level-verification-architecture-for-csr-testing","title":"7.4 Top Level Verification Architecture for CSR testing","text":""},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#741-testbench-components","title":"7.4.1 Testbench components","text":"The testbench components for RAL are defined below
\u2022 ral_reg_iofs_* (uvm_reg) generated by the steps as mentioned in section 5.3\n
The uvm register class is written by extending the uvm_reg. A register represents a set of fields that are accessible as a single entity Each register contains any number of fields, which mirror the values of the corresponding elements in hardware
\u2022 ral_block_iofs_* (uvm_block) generated in the same register file\n
A register model is an instance of a register block, which may contain any number of registers, register files, memories, and other blocks
\u2022 ral_block_ofs (uvm_block) \u2013 Contains all the CSR block instantiations\n\u2022 Reg2vip_*_adapter (uvm_reg_adapter) \u2013 This class defines an interface for converting between uvm_reg_bus_op and a specific bus transaction. For each CSR a respective adapter is present\n
All the components are defined in ofs-agx7-pcie-attach/ofs-common/verification/testbench
Integration of components in testbench
\u2022 The RAL blocks and adapters for each CSR is instantiated in tb_env\n\u2022 The bar range for each CSR is also set in the tb_env\n
Sample Environment Integration snippets
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#8-modifying-uvm-testbench","title":"8 Modifying UVM Testbench","text":"The next sections describe what needs to be considered when modifying the UVM, adding a new interface to the testbench and creating a new UVM test for a customised OFS Accelerator platform.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#81-modifying-uvm-environment-for-new-shell-variant","title":"8.1 Modifying UVM environment for new Shell Variant","text":"OFS n6001 comprises a shell based on PCIe Gen4x16 and is named base_x16
This base_x16 shell is described by an RTL file list, IP File lists and setup scripts to complete the build flow
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#82-modifying-uvm-environment-and-setting-up-compile-and-run-flow","title":"8.2 Modifying UVM environment and setting up Compile and Run flow","text":"All the variants can mostly reuse the existing UVM infrastructure to setup the build and run flow
\u2022 Create directory under $OFS_BUILD_ROOT new variant e.g ofs-n9000\n\u2022 Change directory to $OFS_BUILD_ROOT/ofs-n9000/verification/scripts\n\u2022 modify Makefile it to point to the new RTL, IP and script files.\n
Following these three steps above will enable the build and sim flow to run the existing UVM TB and tests with new IOFS n9000 variant.
Adding a new interface requires signal connections in the testbench. An additional BFM or verification IP is needed to drive the new interface. The main testbench file tb_top.sv is found at the following location
ofs-agx7-pcie-attach/verification/testbench\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#83-adding-a-new-ral-directory","title":"8.3 Adding a new RAL directory","text":"In case the design has many register changes and the user decides to generate all the new RAL models instead of reusing from existing base RAL models, the following steps will help to create and integrate a new RALDIR in the UVM environment.
\u2022 Generate the new RAL files in desired directory. Preferably under the \"ofs-agx7-pcie-attach/verification/common/testbench\" \n\u2022 By default, the Makefile points to base FIM RAL so set the RALDIR path in the Makefile to the new generated RAL file directory\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#84-modifying-tbtest-files-for-new-variant","title":"8.4 Modifying TB/Test files for new Variant","text":"Create a define for the new variant. e.g 'define FIM_NEW. If you are modifying common files then add new logic under this define so other projects will not get affected with variant specific change.
If there are more changes, please create separate \"testbench\" and \"test\" folders under this new variant.
\u2022 Extend variant specific env file from base env file to add variant specific changes.\n\u2022 Create new test/seq lib files in \"tests\" folder.\n\u2022 Create new variant package, add new TB/Tests/Sequence lib files and also import the base package files.\n
If you are adding new files then make sure it's included in Makefile for the build+run flow.
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#85-uvm-pcie-drivers","title":"8.5 UVM PCIe Drivers","text":"The \"svt_pcie_driver_app_transaction_base_sequence\" is part of Synopsys PCIe VIP library. You can find the sequence definition in the following two directories
\u2022 Navigate to \"$DESIGNWARE_HOME/vip/svt/pcie_svt/Q-2020.03/sverilog/src/vcs/svt_pcie_driver_app_transaction_sequence_collection.svp\" file. All the base and PCIe sequences are defined in this file.\n\n\u2022 When the OFS UVM build command is executed, it creates \"vip\" directory under \"$OFS_BUILD_ROOT/ofs-agx7-pcie-attach/verification\". You can also find the same sequence file at \"$IOFS_BUILD_ROOT/ofs-agx7-pcie-attach/verification/vip/pcie_vip/src/sverilog/vcs/svt_pcie_driver_app_transaction_sequence_collection.sv\"\n
"},{"location":"hw/common/user_guides/ug_sim_ofs_agx7_pcie_attach/ug_sim_ofs_agx7_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/","title":"Software Developer Journey Guide: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#1-introduction","title":"1 Introduction","text":"This document is intended to help you understand the FPGA Software developer flow using OFS as well as considerations you should take when creating custom software.
After reviewing the document, you shall be able to:
- Understand how to evaluate the OFS framework for your platform needs.
- Know where to find installation and build instructions for the OFS software stack.
- Understand what resources are available for guiding you through making modifications and for debug.
The general development flow is depicted in the diagram below and discussed in more detail in each section of this document.
Figure 1-1: SW Developer Flow
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#2-evaluate-ofs","title":"2 Evaluate OFS","text":"The repositories in the OFS site are tagged based on year and release number. For example, a tag of 2024.2-1 indicates the first release package of the quarter. If there are updates to this release package, the last number will be incremented by 1, such as 2024.2-2. Not all repositories follow the exact same naming scheme; both the OPAE SDK user space software and Linux DFL Backport kernel driver repository tags differ from their hardware counterparts. See the below table for a summary of the most recent release versions for each software component:
Table 1: Combined OPAE and Platform Version Summary
The OPAE SDK repository is located at https://github.com/OFS/opae-sdk.
Component Platform Most Recent Release OPAE SDK Intel\u00ae FPGA SmartNIC N6000-PL 2.13.0-3 OPAE SDK Intel\u00ae FPGA SmartNIC N6001-PL 2.13.0-3 OPAE SDK Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) 2.13.0-3 OPAE SDK Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) 2.13.0-3 OPAE SDK Intel\u00ae FPGA PAC D5005 2.12.0-5 OPAE SDK IPU Platform F2000X-PL 2.12.0-5 Table 2: Combined DFL and Platform Version Summary
The Linux DFL repository is located at https://github.com/OFS/linux-dfl, the backport repository is located at https://github.com/OFS/linux-dfl-backport.
Component Platform Most Recent Release Linux DFL Intel\u00ae FPGA SmartNIC N6000-PL intel-1.11.0-2 Linux DFL Intel\u00ae FPGA SmartNIC N6001-PL intel-1.11.0-2 Linux DFL Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) intel-1.11.0-2 Linux DFL Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) intel-1.11.0-2 Linux DFL Intel\u00ae FPGA PAC D5005 ofs-2024.1-6.1-2 Linux DFL IPU Platform F2000X-PL ofs-2024.1-6.1-2 Not all OFS releases will support all OFS enabled hardware platforms. Review the official OFS release pages at https://github.com/OFS/ofs-agx7-pcie-attach/releases for PCIe Attach and https://github.com/OFS/ofs-f2000x-pl/releases for SoC Attach.
By clicking on the release link to the right side of the GitHub web page in either of the software repositories, you will find the latest release, the tag number and release notes.
Figure 2-1: OFS Repository Release Page Link
By scrolling to the end of the release page, you will find assets attached to the release that you can leverage for quick evaluation of OFS, such as FPGA binary files, POFs and SOFs, sample AFUs, and pre-compiled binaries for the OFS software stack.
Before evaluating an OFS product, you will need to choose a platform and associated software release that supports it. Each OFS enabled platform has an associated Getting Started Guide, alongside the Software Installation Guidelines and Board Installation Guidelines. The Getting Started Guide will provide a high level overview of the entire setup process, and guide you to the other documents as needed.
Set up your card and software in a server using the steps provided in one of the corresponding Getting Started Guides and leverage the appended binaries in the official OFS release repository release page under the \"Assets\" tab to preview the software and design functionality the OFS framework out of the box. Getting Started Guides are available for the following FIM(shell) designs:
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs
Every OFS release has been validated against a specific OS release. This information, alongside any recommended BIOS settings changes, can be found in the Getting Started Guides.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#3-select-a-starting-shellfim","title":"3 Select a Starting Shell/FIM","text":"To begin your development, start with an existing shell that most closely matches your device and end solution. The OFS site has both Intel Intel Agilex 7 and Stratix 10 FPGA reference designs you can use as a starting point for your own design. These designs can be ported to different device OPNs if a different resource utilization is required, although that will not be covered here.
To begin you may want to learn more about Intel Stratix 10 and Intel Agilex 7 family offerings. Please refer to the following links:
- Intel Agilex 7 Product Page
- Intel Stratix 10 Product Page
In this document, we assume you have downloaded and configured your device with the appropriate pre-compiled binary image for your chosen OFS release and platform without any further customization. For more information on your OFS Shell design and its available key features go to https://ofs.github.io/latest, choose your shell design on the top navigation bar, and select the Shell Technical Reference Manual and Shell Developer Guide. You can also review the FPGA Developer Journey Guide: Open FPGA Stack document.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#4-software-documentation","title":"4 Software Documentation","text":"The Software Reference Manual: Open FPGA Stack provides a complete picture of the OPAE SDK code API alongside concepts you will need to know when developing user space application and driver code. OPAE provides API bindings in C, C++, and Python, although not all functions are supported across languages. Section 13.0 Building a Sample Application provides example application code and build steps that can be used as a quick reference.
You also have the OPAE FPGA Tools available to use for deployment and debug, provided you have set up your environment to work with OFS platforms. These tools can monitor device status, program new PR or statically compiled images, probe CSR space and run basic peripheral exercisers to validate functionality. They can also be used to quickly bind / unbind your device from specific drivers. A full description of the capabilities of these commands can be found on the OFS Site, with shorter descriptions provided in the Getting Started User Guides.
The OPAE SDK repository contains all of the source code for plugins needed for each supported OFS platform. Review section 2.3 Plugin Manager of the Software Reference Manual for further reading on the topic of plugins and where they fit into the OFS stack architecture.
An older documentation repository that stopped being the primary API resource after OPAE 2.3.0 can be found at https://opae.github.io/latest/index.html. The information in this repo should be considered out of date unless using a legacy version of the OPAE SDK software.
The Linux DFL Wiki page provides and introduction to the topics and design philosophy that goes into DFL enabled driver creation. DFL driver documentation has also been upstreamed into the Linux kernel and can be found under Documentation/fpga/dfl.rst.
The below picture demonstrates the structure of the OFS stack stack at a high level:
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#5-setup-your-environment-for-development","title":"5 Setup Your Environment For Development","text":"Before beginning development you will need to ensure you have OFS setup requirements satisfied by reviewing instructions in the corresponding Software Installation Guidelines.
For any additional software you plan on using in conjunction with OPAE and DFL - for example, DPDK and IPDK - refer to that software solution's installation guidelines, and make sure your environment has been properly initialized.
All OFS repositories are open source and do not require permissions to clone.
The primary component of the OFS software stack that must be installed before developing any application is the OPAE SDK. Additionally, if you plan on testing your applications on hardware then the DFL driver set must also be installed. All Getting Started User Guides will point you to the correct location for install steps relevant to a given platform.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#6-pick-a-reference","title":"6 Pick a Reference","text":"Before beginning development of a new driver or user space software application, it can help to find an already existing piece of code that can help guide your development.
If looking to write user space application code that uses the OPAE SDK API, we provide a series of sample applications under the /opae/samples directory. These cover a wide range of topics, including taking advantage of CXL, built-in host exercisers, HSSI, and AFU and FIM event checking. If you have never written an application before you should review hello_afu and its included README.
Driver code samples vary widely depending on their implementation and the needs of the associated IP, sometimes being split into multiple source files. The 8250 DFL UART IP Driver represents a minimal driver implementation, and clearly demonstrates where IP specific implementation ends and DFL framework integration begins. This driver calls into the existing UART driver framework, while providing the bus \"glue\" logic for core DFL functionality (specifically, the line dfluart->line = serial8250_register_8250_port(&uart); is where the driver calls into the existing Linux kernel 8250 driver framework).
A second driver example you can review is the DFL specific implementation of a generic SPI driver. Where again the line err = devm_spi_register_controller(dev, host); is used to register this device with the SPI framework provided by the kernel, while also calling upon DFL framework specific constructs such as the FEATURE_ID and FME_ID to properly identify the device based on its DFH.
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#7-opae-sdk-application-build-and-debug","title":"7 OPAE SDK Application Build and Debug","text":"Building an OPAE application involves linking your source code with the OPAE libraries. A minimal build command using gcc is provided below as a reference for an application using the C API. The API uses some features from the C99 language standard. The -std=c99 switch is required if the compiler does not support C99 by default.
This library internally uses libuuid and libjson-c. But they are not distributed by it. Make sure you have these libraries properly installed. The -c flag may not be necessary depending on the platform being used.
gcc -std=c99 hello_fpga.c -I/usr/local/include -L/usr/local/lib -lopae-c -luuid -ljson-c -lpthread -o hello_fpga\n
The OPAE SDK has a built-in debug logging facility. To enable it, set the cmake flag -DCMAKE_BUILD_TYPE=Debug and then use the following environment variables:
Variable Description LIBOPAE_LOG=1 Enable debug logging output. When not set, only critical error messages are displayed. LIBOPAE_LOGFILE=file.log Capture debug log output to file.log. When not set, the logging appears on stdout and stderr. The file must appear in a relative path or it can be rooted at /tmp. To enable gdb-based debugging, the cmake configuration step must specify a value for -DCMAKE_BUILD_TYPE of either Debug or RelWithDebInfo so that debug symbols are included in the output binaries. The OPAE SDK makes use of dynamically-loaded library modules. When debugging with gdb, the best practice is to remove all OPAE SDK libraries from the system installation paths to ensure that library modules are only loaded from the local build tree:
$ cd opae-sdk/build\n$ LD_LIBRARY_PATH=$PWD/lib gdb --args <some_opae_executable> <args>\n
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#8-test-and-deploy","title":"8 Test and Deploy","text":"Deployment of an AFU or statically compiled FIM+AFU image is detailed in the associated Getting Started User Guide for a given platform. If you wish to deploy your application into a VM or docker image you can review the Virtual machine User Guide: Open FPGA Stack + KVM and the Docker user guide, located under the Virtualization header of your chosen platform.
When testing a custom written user space application, you can use the following procedure:
- Build and compile application with debug/GDB flags
- Configure FPGA with AFU or static image using
fpgasupdate, or JTAG flow - Verify the hardware has been properly configured with
fpgainfo - Double check that your FPGA is attached to the correct driver with
opae.io ls 4.a. If not, you can swap between dfl-pci and vfio-pci with opae.io bind/unbind - Run your application
- Customize, rebuild and redeploy as needed
"},{"location":"hw/common/user_guides/ug_sw_developer/ug_sw_developer/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/","title":"AFU Developer Guide: OFS for Stratix\u00ae 10 FPGA PCIe Attach FPGAs","text":""},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#1-introduction","title":"1. Introduction","text":"This document is a design guide for creating an Accelerator Functional Unit (AFU) using Open FPGA Stack (OFS) for Stratix\u00ae 10 FPGA. The AFU concept consists of separating the FPGA design development process into two parts, the FIM and AFU, as shown in the diagram below:
This diagram shows the FPGA board interface development separation from the internal FPGA workload creation. This separation starts with the FPGA Interface Manager (FIM), which consists of the external interfaces and board management functions. The FIM is the base system layer typically provided by board vendors. The FIM interface is specific to a particular physical platform. The AFU uses the external interfaces with user-defined logic to perform a specific application. Separating the lengthy and complicated process of developing and integrating external interfaces for an FPGA into a board allows the AFU developer to focus on their workload needs. OFS for Stratix\u00ae 10 FPGA provides the following tools for rapid AFU development:
- Scripts for both compilation setup
- Integration with Open Programmable Acceleration Engine (OPAE) SDK for rapid software development for your AFU application
Please notice that the AFU region consists of both static and PR logic in the above block diagram. Creating AFU logic for the static region is described in Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs. This guide covers logic in the AFU Main (PR) region.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#11-document-organization","title":"1.1 Document Organization","text":"This document is organized as follows:
- Description of design flow
- Interfaces and functionality provided in the Intel\u00ae FPGA PAC D5005 FIM
- Downloading and installing OFS and OPAE SDK
- Hardware/Software co-simulation using ASE
- Testing the AFU example in Intel\u00ae FPGA PAC D5005
- Debugging an AFU with Remote Signal Tap
This guide provides theory followed by tutorial steps to solidify your AFU development knowledge.
This guide uses the Intel\u00ae FPGA PAC D5005 as the platform for all tutorial steps. Additionally, this guide and the tutorial steps can be used with other platforms; However, please consult the board and FIM supplier of other platforms for specific instructions on the use of custom FIM to develop AFU design.
If you have worked with previous Programmable Acceleration products, you will find OFS for Stratix\u00ae 10 FPGA is similar; however, there are differences, and you are advised to carefully read and follow the tutorial steps to understand the design tools and flow fully.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#12-prerequisite","title":"1.2 Prerequisite","text":"This guide assumes you understand the following FPGA logic design-related knowledge and skills:
- FPGA compilation flows, including the Quartus\u00ae Prime Pro Edition design flow.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition software, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on critical timing paths.
- RTL and coding practices to create synthesized logic.
- High-level synthesis (HLS) and Platform Designer design entry tools are supported.
- RTL simulation tools.
- Signal Tap Logic Analyzer tool in the Quartus\u00ae Prime Pro Edition software.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#121-development-environment","title":"1.2.1 Development Environment","text":"To run the tutorial steps in this guide requires this development environment:
Item Version Operating System RHEL 8.6 Python 3.6.8 cmake 3.15 GCC 8.5.0 git 1.8.3.1 perl 5.8.8 Verify your development has the above tools installed.
The following server and PAC card are required to run the examples in this guide:
- Intel\u00ae FPGA PAC D5005 with root entry hash erased (Please contact Altera\u00ae for root entry hash erase instructions). The standard Intel\u00ae FPGA PAC D5005 card is programmed only to allow the FIM binary files signed by Altera\u00ae to be loaded. The root entry hash erases process will allow unsigned FIM binary files to be loaded.
- Intel\u00ae FPGA PAC D5005 installed in a qualified server following instructions in Board Installation Guide: OFS for Acceleration Development Platforms.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#13-acceleration-functional-unit-afu-development-flow","title":"1.3 Acceleration Functional Unit (AFU) Development Flow","text":"OFS Stack provides a rapid design methodology for creating complex FPGA applications. In addition, you are provided with the following:
- Hardware shell layer, known as FIM
- Software stack including tools for debug/diagnostics
- FPGA design flow with full-stack simulation support
- AFU code samples demonstrating all interfaces
For any non-Altera\u00ae platform, contact your board vendor for the above components specific to the platform.
To start with AFU development, the first step should be to understand your platform capabilities. For example, what interface is the FPGA connected to the Host machine over PCI-E, if it is AXI like the Stratix\u00ae 10 FPGA Platform, or CCIP or CXL. Does the platform provide an External Memory Interface or the HSSI interface? Once you know what the platform offers, you can develop your AFU requirements and architecture as the next step.
This document will cover example AFU architecture and things that will help build AFU for Stratix\u00ae 10 FPGA reference platform and others coming in the future. In addition, this knowledge can be relatively applied for AFU development on other vendor-provided platforms.
The figure below shows a typical AFU development process independent of the platform used.
flowchart TB;\n A[Understand platform capabilities with OFS]-->B[Review AFU requirements and code samples provided];\n B[Review AFU requirements and code samples provided]-->C[Define AFU architecture];\n C[Define AFU architecture]-->D[Design AFU hardware];\n D[Design AFU hardware]-->E[Develop AFU software to control hardware];\n E[Develop AFU software to control hardware]-->F{\"Simulate in AFU Simulation Enviroment (ASE)\"};\n F:::if -- Pass --> H[\"Compile AFU for synthesis, place & route and timing (uses Quartus)\"];\n H[\"Compile AFU for synthesis, place & route and timing (uses Quartus)\"] --> I[\"Analyze Quartus Compile reports\"];\n I --> J{\"Quartus reports clean? (e.g. timing closed)\"};\n J:::if -- No --> P2;\n J -- Yes --> K[\"Run/Validate design on OFS Platform\"];\n K --> L{\"Hardware validation pass?\"};\n L == Yes ==> M[\"AFU ready to deploy\"];\n L -- No --> N[\"Debug on hardware using traditional FPGA tools (e.g. SignalTab\"];\n N --> P2[\"Fix AFU design (e.g Design changes, timing closure constraints)\"];\n P2 --> O{\"Need functional validation?\"};\n O:::if -- Yes -->P[\"Fix AFU design (e.g Functional design changes, bug fixes)\"];\n O -- No -->H; \n F -- Fail --> P;\n P -->D; \n\n classDef default color:#fff,fill:#0071c5,stroke:#71c5,stroke-width:1px\n classDef if color:#0071c5,fill:#fff,stroke:#0071c5,stroke-width:2px
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#131-high-level-data-flow","title":"1.3.1. High Level Data Flow","text":"The OFShigh-level data flow is shown below: The control and data are composed of the following:
- Host Interface Adapter (PCIe)
- Low-Performance Peripherals
- Slow speed peripherals (I2C, Smbus, etc)
- Management peripherals (FME)
- High-Performance Peripherals
- Memory peripherals
- Acceleration Function peripherals
- HPS Peripheral
- Fabrics
- Peripheral Fabric (multi-drop)
- AFU Streaming fabric (point to point)
Peripherals are connected using AXI or Avalon:
- Via the peripheral fabric (AXI4-Lite, multi-drop)
- Via the AFU streaming fabric (AXI-S, point to point)
Peripherals are presented to software as:
- OFS managed peripherals that implement DFH CSR structure.
- Native driver managed peripherals (i.e., Exposed via an independent PF, VF)
The peripherals connected to the peripheral fabric are primarily OPAE managed resources, whereas the peripherals connected to the AFU are \"primarily\" driven by native OS drivers. The word \"primarily\" is used since the AFU is not mandated to expose all its peripherals to OPAE. Instead, it can be connected to the peripheral fabric but can choose to expose only a subset of its capability to OPAE.
OFS uses a defined set of CSRs to expose the functionality of the FPGA to the host software. These registers are described in Open FPGA Stack Reference Manual - MMIO Regions section.
If you make changes to the FIM that affect the software operation, OFS provides a mechanism to communicate that information to the proper software driver. The Device Feature Header (DFH) structure provides a mechanism to maintain compatibility with OPAE software. Please see FPGA Device Feature List (DFL) Framework Overview for an excellent description of DFL operation from the driver perspective.
When planning your address space for your FIM updates, please be aware OFS FIM targeting Intel\u00ae FPGA PAC D5005, 256KB of MMIO region is allocated for external FME features, and 128kB of MMIO region is given for external port features. Each external feature must implement a feature DFH, and the DFH needs to be placed at the 4KB boundary. The last feature in the external feature list must have the EOL bit in its DFH set to 1 to mark the end of the external feature list. Since the FPGA address space is limited, consider using an indirect addressing scheme to conserve address space.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#132-considerations-for-pim-usage","title":"1.3.2. Considerations for PIM Usage","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
The following resources are available to assist in creating an AFU:
PIM Core Concepts provides details on using the PIM and its capabilities.
PIM Based AFU Developer Guide provides details on interfacing your AFU to the FIM using the PIM.
The examples-afu repo provides several AFU examples:
Example Description PIM-based Hybrid Native clocks Example AFU using user configurable clocks. X copy_engine Example AFU moving data between host channel and a data engine. X dma Example AFU moving data between host channel and local memory with a DMA. X hello_world Example AFU sending \"Hello World!\" to host channel. X X X local_memory Example AFU reading and writing local memory. X X These examples can be run with the current OFS FIM package. There are three AFU types of examples provided (PIM based, hybrid and native). Each example provides the following:
- RTL, which includes the following interfaces:
- Host Channel:
- Host memory, providing a DMA interface.
- MMIO, providing a CSR interface.
- Local Memory
- Host software example interfacing to the CSR interface and host memory interface, using the OPAE C API.
- Accelerator Description File .json file
- Source file list
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#133-afu-interfaces-included-with-intel-fpga-pac-d5005","title":"1.3.3 AFU Interfaces Included with Intel\u00ae FPGA PAC D5005","text":"The figure below shows the interfaces available to the AFU in this architecture. It also shows the design hierarchy with module names from the FIM (top.sv) to the PR region AFU (afu_main.sv). One of the main differences from the previous Stratix\u00ae 10 FPGA OFS architecture is a static port gasket region (port_gasket.sv) that has components to facilitate the AFU and also consists of the GBS region (afu_main.sv) via the PR slot. The Port Gasket contains all the PR -specific modules and logic, e.g., PR slot reset/freeze control, user clock, remote STP etc. Architecturally, a Port Gasket can have multiple PR slots to which user workload can be programmed. However, only one PR slot is supported for OFS Release for Stratix\u00ae 10 FPGA. Therefore, everything in the Port Gasket until the PR slot should be provided by the FIM developer. The task of the AFU developer is to add their desired application in the afu_main.sv module by stripping out unwanted logic and instantiating the target accelerator. As shown in the figure below, here are the interfaces connected to the AFU (highlighted in green) via Intel\u00ae FPGA PAC D5005 FIM:
- AXI Streaming (AXI-S) interface to the Host via PCIe Gen3x16
- Avalon Memory-Mapped Channels (4) to the DDR4 EMIF interface
- AXI Streaming (AXI-S) interface to the HSSI 10G Ethernet
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#134-platform-capabilities","title":"1.3.4. Platform Capabilities","text":"The FIM targets operation in the Intel\u00ae FPGA PAC D5005 card. The block diagram of the Intel\u00ae FPGA PAC D5005 is shown below:
The key Intel\u00ae FPGA PAC D5005 FPGA interfaces are:
- Host interface - PCIe Gen3 x 16
- Network interface
- 2 - QSFP28 cages
- Current FIM supports 1 x 10 GbE, other interfaces can be created
- External Memory
- 2 or 4 channels of DDR4-2400 to RDIMM modules
- RDIMM modules = 8GB organized as 1 Gb X 72
- Board Management
- SPI interface
- FPGA configuration
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#135-top-level-fpga","title":"1.3.5. Top Level FPGA","text":"The internal FPGA architecture is shown below:
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2-set-up-afu-development-environment","title":"2. Set Up AFU Development Environment","text":"This section covers:
- Development environment set up
- Retrieving and installing OFS, OPAE SDK
- Building theIntel\u00ae FPGA PAC D5005 FIM
- Building a relocatable AFU tree
- Compiling the host_chan_mmio example AFU
Additionally, this section includes steps to demonstrate loading and running the host_chan_mmio example AFU in an Intel\u00ae FPGA PAC D5005 equipped Linux server.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#21-prepare-afu-development-environment","title":"2.1. Prepare AFU development environment","text":"Typical development and hardware test environments consist of a development server or workstation with installed FPGA development tools and a separate server installed with the target OFS-compatible FPGA PCIe card. The typical usage and flow of data between these two servers are shown below:
Please refer to Unit Level Simulation if you would like to make any simulation Unit Level Simulation.
Note that both development and hardware testing can be performed on the same server if desired.
This guide uses Intel\u00ae FPGA PAC D5005 as the target OFS-compatible FPGA PCIe card platform for demonstration steps. The Intel\u00ae FPGA PAC D5005 must be fully installed following Board Installation Guide: OFS for Acceleration Development Platforms. If using a different OFS FPGA PCIe card, contact your supplier for instructions on how to install and operate a user-developed AFU.
NOTE:
The following chapters assume you use the same server for development and Deployment (Run the FIM/AFU/SW over the Intel\u00ae FPGA PAC D5005):
Development: Modify the FIM/AFU/SW run simulation and compile the design (Generate the binaries). Deployment: Program the binaries under the Intel\u00ae FPGA PAC D5005 and exercise the Hardware and Sw with real hardware
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#211-installation-of-quartus-and-ofs","title":"2.1.1. Installation of Quartus and OFS","text":"Building AFU with OFS forStratix\u00ae 10 FPGA requires the build machine to have at least 64 GB of RAM.
The following is a summary of the steps to set up for AFU development:
- Install Quartus\u00ae Prime Pro Edition 23.4 Linux with Stratix\u00ae 10 FPGA device support.
- Make sure support tools are installed and meet version requirements.
- Clone the repository.
- Review the files provided in the repository.
- Build a relocatable PR tree - this will be the base FIM for your AFU.
Quartus\u00ae Prime Pro Edition version 23.4 is the currently verified version of Quartus\u00ae Prime Pro Edition 23.4 used for building the AFU images. The recommended Best Known Configuration (BKC) OFS Version 2024.1-1:
Item Version Quartus\u00ae Prime Pro Edition 23.4 Operating System RHEL 8.6 OPAE SDK 2.12.0-5 OFS Release ofs-2024.1-1 Python 3.6.8 cmake 3.15 GCC 8.5.0 git 1.8.3.1 perl 5.8.8"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2111-installation-of-quartus","title":"2.1.1.1 Installation of Quartus","text":"Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.6 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are No patches for this release.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2112-install-ofs","title":"2.1.1.2. Install OFS","text":" -
Retrieve OFS repositories:
The OFS FIM source code is included in the OFS GitHub repository. First, create a new directory to store the retrieved files as a clean starting point. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories.
-
Navigate to the location for storage of OFS source, create the top-level source directory, and clone OFS repositories.
mkdir ofs_fim_build_root\ncd ofs_fim_build_root\n
export OFS_BUILD_ROOT=$PWD\n
git clone --recurse-submodules https://github.com/OFS/ofs-d5005.git\n
Console Output:
Cloning into 'ofs-d5005' ...\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\n
Edit your bashrc file ~/.bashrc to add the following lines:
export OFS_ROOTDIR=$OFS_BUILD_ROOT/ofs-d5005\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\n
cd ofs-d5005\n
Select the latest OFS Release
git checkout tags/ofs-2024.1-1\n
Console Output: ```sh You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:
git checkout -b HEAD is now at 7e4dc70 ofs-2024.1-1"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2113-directory-structure-of-ofs","title":"2.1.1.3. Directory Structure of OFS","text":"
Verify the following directories in the $OFS_BUILD_ROOT directory with the following command.
cd $OFS_ROOTDIR\nls\n
Console Output:
eval_scripts ipss ofs-common license LICENSE.md README.md sim src syn verification\n
The directories are arranged as shown below:
\u251c\u2500\u2500 eval_scripts\n\u2502 \u251c\u2500\u2500 ofs_d5005_eval.sh\n\u2502 \u251c\u2500\u2500 README_ofs_d5005_eval.txt\n|\n\u251c\u2500\u2500 ofs-common\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 src\n\u2502 \u251c\u2500\u2500 verification\n| \u251c\u2500\u2500 LICENSE.txt \u2502 \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 ipss **Directory ipss consists of Platform Designer subsystems used in FIM**\n\u2502 \u251c\u2500\u2500 hssi\n\u2502 \u251c\u2500\u2500 mem\n\u2502 \u251c\u2500\u2500 pcie\n| \u251c\u2500\u2500 pmic | \u251c\u2500\u2500 spi \u2502 \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 license\n\u2502 \u2514\u2500\u2500 quartus-0.0-0.01iofs-linux.run ** Quartus Patch with IP licenses. \u2502 ** Note, these licenses are not used for Intel\u00ae FPGA PAC D5005** \u251c\u2500\u2500 sim **Unit level simulation files**\n\u2502 \u251c\u2500\u2500 unit_test\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 bfm\n\u2502 \u251c\u2500\u2500 rp_bfm \u2502 \u2514\u2500\u2500 readme.txt \u2502 \u251c\u2500\u2500 LICENSE.txt\n\u251c\u2500\u2500 README.md\n|\n\u251c\u2500\u2500 src **Source RTL files**\n\u2502 \u251c\u2500\u2500 afu_top\n\u2502 \u251c\u2500\u2500 includes\n\u2502 \u251c\u2500\u2500 pd_qsys\n\u2502 \u251c\u2500\u2500 top\n| \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 external\n\u2502 \u2514\u2500\u2500 ofs-platform-afu-bbb\n|\n\u251c\u2500\u2500 syn **Quartus compilation settings**\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 setup\n\u2502 \u251c\u2500\u2500 syn_top\n\u2502 \u251c\u2500\u2500 readme.txt\n\u2502 \u2514\u2500\u2500 README\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2114-license-installation-for-ofs","title":"2.1.1.4 License Installation for OFS","text":"The required setup OFS License quartus-0.0-0.01iofs-linux.run, follow the following steps :
cd $OFS_ROOTDIR/license\nchmod +x quartus-0.0-0.01iofs-linux.run\nsudo ./quartus-0.0-0.01iofs-linux.run\n# Confirm the license instaltion using below command.\nquartus_syn --version\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2115-retrieve-pim-files","title":"2.1.1.5. Retrieve PIM Files","text":"The ofs-platform-afu-bbb repository contains the PIM files and example AFU that can be used for testing and demonstration purposes. This guide will use the host_chan_mmio example in the remaining sections to demonstrate OFS capabilities.
cd $OFS_BUILD_ROOT\n
git clone https://github.com/OFS/ofs-platform-afu-bbb.git\n
Edit your bashrc file ~/.bashrc to add the following lines: export OFS_PLATFORM_AFU_BBB=$OFS_BUILD_ROOT/ofs-platform-afu-bbb\n
Verify the following directories are present in $OFS_BUILD_ROOT directory.
cd $OFS_PLATFORM_AFU_BBB\n
ls\n
Console Output:
COPYING plat_if_develop plat_if_release plat_if_tests README.md\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#212-compiling-the-ofs-fim","title":"2.1.2. Compiling the OFS FIM","text":"OFS provides a build script with the following FPGA image creation options:
- Flat compile, which combines the FIM and AFU into one FPGA image loaded into the entire FPGA device as a static image.
- A PR compile that creates an FPGA image consisting of the FIM that is loaded into the static region of the FPGA and a default AFU that is loaded into dynamic region. Additional AFU may be loaded into the dynamic region using partial reconfiguration.
The build scripts included with OFS are verified to run in a bash shell. Other shells have not been tested. Each build script step will take several hours to complete. Building in Quartus GUI is not supported - you must build with the provided scripts.
The following sections describe how to set up the environment and build the provided FIM with a relocatable tree supporting PR . You will use this relocatable PR tree for all example AFU simulation and compilation steps in this guide.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2121-setting-up-required-environment-variables","title":"2.1.2.1. Setting Up Required Environment Variables","text":"Set required environment variables as shown below. These environment variables must be set before simulation or compilation tasks, so creating a simple script to set these variables saves time.
Edit your bashrc file ~/.bashrc to add the following lines:
export OPAE_SDK_REPO_BRANCH=release/2.12.0\n
Check point : Ensure you file ~/.bashrc have all the following lines:
export QUARTUS_MAINPATH=<Quartus install directory>\nexport QUARTUS_ROOTDIR=$QUARTUS_MAINPATH/quartus\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ipexport\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_MAINPATH/qsys/bin\nexport PATH=$PATH:$QUARTUS_ROOTDIR/bin\nexport OFS_BUILD_ROOT=<root location> ** Here should be located your ofs-d5005 and ofs-platform-afu-bbb\nexport OFS_ROOTDIR=$OFS_BUILD_ROOT/ofs-d5005\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\nexport OFS_PLATFORM_AFU_BBB=$OFS_BUILD_ROOT/ofs-platform-afu-bbb\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2122-compiling-your-base-fim","title":"2.1.2.2. Compiling Your Base FIM","text":"The usage of the compile build script is shown below:
ofs-common/scripts/common/syn/build_top.sh [-p] target_configuration work_dir \n\n * target_configuration - Specifies the project \n For example: d5005\n\n * work_dir - Work Directory for this build in the form a directory name. It is created in the <local repo directory>/ofs-d5005/<work_dir> \n - NOTE: The directory name must start with \"work\". If the working directory exists, the script stops and asks if you want to overwrite the directory.\n - e.g.\n - ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005\n\n work directory as a name will be created in <local repo directory>/ofs-d5005/work_d5005\n\n The obmission of <work_dir> results in a default work directory (<local repo directory>/ofs-d5005/work)\n\n - compile reports and artifacts (.rpt, .sof, etc) are stored in <work_dir>/syn/<OFS_PROJECT>/<OFS_FIM>/<OFS_BOARD>/syn_top/output_files\n\n - There is a log file created in ofs-d5005 directory. \n - [-p] Optional switch for creating a relocatable PR build tree supporting the creation of a PR -able AFU workload. \n The \"-p\" switch invokes generate_pr_release.sh at the end of the FIM build and writes the PR build tree to the top of the working directory. More information on this option is provided below. \n
In the following example, you will build the provided example design using a flat, non-PR build flow. If you use the -p, you could avoid the section."},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#21221-relocatable-pr-directory-tree","title":"2.1.2.2.1. Relocatable PR Directory Tree","text":"Build the provided base example design:
cd $OFS_ROOTDIR\n
ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005\n
Console Output:
... build takes ~5 hours to complete\nCompile work directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top\nCompile artifact directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top/output_files\n\n***********************************\n***\n*** OFS_PROJECT: d5005\n*** Q_PROJECT: d5005\n*** Q_REVISION: d5005\n*** SEED: 03\n*** Build Complete\n*** Timing Passed!\n***\n
Pro Tip: if the timing report fails, try to go into the iofs_pr_afu.qsf and modify the seed number from 3 to 4, it will create multiple seed/starting points of your design to find the best timing/fit. /home/<myuser>/<mainfolderforOFS>/ofs-d5005/work_d5005/syn/syn_top/iofs_pr_afu.qsf
set_global_assignment -name SEED 0 #0-4\n
The build script copies the ipss, sim, src, and syn directories to the specified work directory, and then these copied files are used in the Quartus compilation process. Therefore, do not edit the files in the work directory; these files are copies of source files.
Some of the critical output files are described below:
$OFS_ROOTDIR//syn/syn_top
\u251c\u2500\u2500 syn_top //Intel\u00ae FPGA PAC D5005 Quartus build area with Quartus files used this build\n\u2502 \u251c\u2500\u2500 d5005.ipregen.rpt // IP regeneration report states the output of IP upgrade\n\u2502 \u251c\u2500\u2500 d5005.qpf // Quartus Project File (qpf) mentions about Quartus version and project revision\n\u2502 \u251c\u2500\u2500 d5005.qsf // Quartus Settings File (qsf) lists current project settings and entity level assignments\n\u2502 \u251c\u2500\u2500 d5005.stp // Signal Tap file included in the d5005.qsf. This file can be modified as required\n\u2502 \u251c\u2500\u2500 fme_id.mif // the fme id hex value is stored in a mif file format\n\u2502 \u251c\u2500\u2500 iofs_pr_afu.json // PR JSON file\n\u2502 \u251c\u2500\u2500 iofs_pr_afu.qsf // PR AFU qsf file\n\u2502 \u251c\u2500\u2500 iofs_pr_afu_sources.tcl // AFU source file list\n
$OFS_ROOTDIR//syn/syn_top/output_files == Directory with build reports and FPGA programming files.
The programming files consist of the Quartus generated d5005.sof and d5005.pof. The Intel\u00ae FPGA PAC D5005 board hardware provides a 2 Gb flash device to store the FPGA programming files and a BMC CARD that reads this flash and programs the Intel\u00ae FPGA PAC D5005 Stratix\u00ae 10 FPGA. The ./ofs-common/scripts/common/syn/build_top.sh script runs script file ./ofs-common/scripts/common/syn/build_top.sh which takes the Quartus generated d5005.sof and creates binary files in the proper format to be loaded into the 2 Gb flash device. You can also run build_flash.sh by itself if needed.
The build script will run PACSign and create an unsigned FPGA programming file for both user1 and user2 locations of the Intel\u00ae FPGA PAC D5005 flash. Please note, if the Intel\u00ae FPGA PAC D5005 has the root entry hash key loaded, then PACsign must be run to add the proper key to the FPGA binary file. Please see Security User Guide: Open FPGA Stack for details on the security aspects of Open FPGA Stack and refer to Board Management User Guide for Flash partition.
The following table provides further detail on the generated bin files.
File Description d5005.sof This is the Quartus generated programming file created by Quartus synthesis and place and route. This file can be used to program the FPGA using a JTAG programmer. This file is the source file for the binary files used to program the FPGA flash. d5005.bin This is an intermediate raw binary image of the FPGA d5005_page1.bin This is the binary file created from the input file, d5005.sof. This file is used as the input file to the PACSign utility to generate d5005_page1_unsigned.bin binary image file. d5005_page1_unsigned.bin This is the unsigned PACSign output which can be programmed into the FPGA flash of an unsigned Intel\u00ae FPGA PAC D5005 using the OPAE SDK utility fpgasupdate mfg_d5005_reversed.bin A particular programming file for a third-party device used in board manufacturing. This file is typically not used. build/output_files/timing_report == Directory containing clocks report, failing paths and passing margin reports
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#213-relocatable-pr-directory-tree","title":"2.1.3. Relocatable PR Directory Tree","text":"If you are developing FIM to be used by another team developing the AFU workload, scripts are provided that create a relocatable PR directory tree. ODM and board developers will use this capability to enable a broad set of AFU to be loaded on a board using PR . The relocatable PR directory contains the Quartus *.qdb file that goes the FIM.
Creating the relocatable PR directory tree requires a clone of the Basic Building Blocks (BBB) repository. The OFS_PLATFORM_AFU_BBB environment variable must point to the repository. If not done previously, clone the Basic Building Blocks repository and create OFS_PLATFORM_AFU_BBB environment variable.
cd $OFS_BUILD_ROOT\n
git clone https://github.com/OFS/ofs-platform-afu-bbb.git\n
export OFS_PLATFORM_AFU_BBB=$OFS_BUILD_ROOT/ofs-platform-afu-bbb\n
cd $OFS_ROOTDIR\n
You can create this relocatable PR directory tree by either:
- Build FIM and AFU using ofs-common/scripts/common/syn/build_top.sh followed by running /syn/common/scripts/generate_pr_release.sh (section 2.1.3. Relocatable PR Directory Tree)
- Build FIM and AFU using ofs-common/scripts/common/syn/build_top.sh with optional -p switch included
The generate_pr_release.sh has the following command structure:
ofs-common/scripts/common/syn/generate_pr_release.sh -t <path to generated release tree> *Board Build Target* <work dir from build_top.sh>\nWhere:\n-t <path to generated release tree> = location for your relocatable PR directory tree\n*Board Build Target* is the name of the board target/FIM e.g. d5005\n<work dir from build_top.sh>
Here is an example of running the generate_pr_release.sh script in user mode: ofs-common/scripts/common/syn/generate_pr_release.sh -t work_d5005/build_tree d5005 work_d5005\n
Console Output:
**********************************\n********* ENV SETUP **************\nFIM Project:\n OFS_PROJECT = d5005\n OFS_FIM = .\n OFS_BOARD = .\n Q_PROJECT = d5005\n Q_REVISION = d5005\n Fitter SEED = 03\nFME id\n BITSTREAM_ID = 040100022c164db1\n BITSTREAM_MD = 0000000002212053\n...\n...\n
The resulting relocatable build tree has the following structure:
.\n\u251c\u2500\u2500 bin\n\u2502 \u251c\u2500\u2500 afu_synth\n\u2502 \u251c\u2500\u2500 build_env_config\n\u2502 \u251c\u2500\u2500 run.sh -> afu_synth\n\u2502 \u2514\u2500\u2500 update_pim\n\u251c\u2500\u2500 hw\n\u2502 \u251c\u2500\u2500 blue_bits\n\u2502 \u2502 \u251c\u2500\u2500 d5005_page1_unsigned.bin\n\u2502 \u2502 \u2514\u2500\u2500 d5005.sof -> ../lib/build/syn/syn_top/ output_files/d5005.sof\n\u2502 \u2514\u2500\u2500 lib\n\u2502 \u251c\u2500\u2500 build\n\u2502 \u251c\u2500\u2500 fme-ifc-id.txt\n\u2502 \u251c\u2500\u2500 fme-platform-class.txt\n\u2502 \u2514\u2500\u2500 platform\n\u251c\u2500\u2500 README\n
Edit your bashrc file ~/.bashrc to add the following line:
export OPAE_PLATFORM_ROOT=$OFS_ROOTDIR/work_d5005/build_tree\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#214-programing-the-fim","title":"2.1.4. Programing the FIM","text":" - Run the following command to find the PCIe address for your card.
sudo fpgainfo fme\n
Console Output:
Board Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nDevice Id : 0xBCCE\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#2141-load-fim-into-the-flash-of-the-intel-fpga-pac-d5005","title":"2.1.4.1. Load FIM into the Flash of the Intel\u00ae FPGA PAC D5005","text":"The base FIM used in AFU compilation must be loaded on the board. In this step, you will load the generated FIM binary into the Intel\u00ae FPGA PAC D5005 FPGA flash. By performing this step, subsequent AFU developed in this guide will use this base FIM and allow your newly created AFU to match the base FIM loaded on the board.
More information related to fpgaupdate is located Software Installation Guide: OFS for PCIe Attach FPGAs.
Run fpgasupdate to load the image into the user location of the Intel\u00ae FPGA PAC D5005 FPGA flash and the RSU command to reboot the PCIE Card:
sudo fpgasupdate $OFS_ROOTDIR/work_d5005/syn/syn_top/output_files/d5005_page1_unsigned.bin 3b:00.0\n
Run rsu command to re-configure FPGA on Intel\u00ae FPGA PAC D5005. sudo rsu bmcimg 3b:00.0\n
sudo fpgainfo fme\n
Console Output: ```sh
Board Management Controller, MAX10 NIOS FW version: 2.0.8 Board Management Controller, MAX10 Build version: 2.0.8 //****** FME ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:3b:00.0 Device Id : 0xBCCE Socket Id : 0x00 Ports Num : 01 Bitstream Id : 288511860124977321 Bitstream Version : 4.0.1 Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e Boot Page : user
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#30-opae-software-development-kit","title":"3.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines the integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and re-configure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, please visit the OPAE.io page.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE GitHub. This repository is open source and should not require any permissions to access.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#31-opae-sdk-build-environment-setup","title":"3.1 OPAE SDK Build Environment Setup","text":"This installation process assumes the user has access to an internet connection to pull specific GitHub repositories and satisfy package dependencies. If an offline install process is required, please reach out to your Altera\u00ae representative.
1. Before OPAE SDK installation, the user must remove any prior OPAE frameworks. To remove these packages:
sudo dnf remove opae*\n
2. The user must enable the following repository changes in order to install all dependencies on CentOS 8.3:
sudo dnf config-manager --set-enabled powertools\nsudo dnf install epel-release\n
3. The following package dependencies must be satisfied by the user. Double check that all packages have been found and installed:
sudo dnf install autoconf automake bison boost boost-devel cmake doxygen dwarves elfutils-libelf-devel \\\nflex gcc gcc-c++ git hwloc-devel json-c-devel libarchive libedit libedit-devel libpcap libpng12 libuuid libuuid-devel libxml2 libxml2-devel make ncurses \\\nncurses-devel ncurses-libs openssl-devel python2-pip python3-devel python3-jsonschema rsync tbb-devel libudev-devel\n
All steps in this installation will use a generic top-level directory at $OFS_BUILD_ROOT. If the user has created a different top-level directory, replace this path with the user's custom path.
4. Initialize an empty git repository and clone the tagged OPAE SDK source code:
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#32-install-opae-sdk","title":"3.2 Install OPAE SDK","text":"Perform the following steps to install OPAE SDK:
cd $OFS_BUILD_ROOT\ngit clone https://github.com/OFS/opae-sdk\ncd opae-sdk\ngit checkout tags/2.12.0-5 -b release/2.12.0\n
Verify proper branch is selected git describe\n 2.12.0-5\n\ngit branch\n master\n * release/2.12.0\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#33-building-and-installing-the-opae-sdk","title":"3.3 Building and Installing the OPAE SDK","text":"1. Build the OPAE SDK source code, and pack it into several local RPM packages. Building the code into packages allows for easier installation and removal. This build script can use multiple processors to parallelize the build process. Display how many processors are available with the nproc command, and then specify how many make threads to utilize with the -j option. Note that the number of threads can exceed the number of processors. In this case, the number of threads is set to the number of processors in the system.
cd $OFS_BUILD_ROOT/opae-sdk\nmkdir install-opae-sdk\ncd install-opae-sdk\ncmake .. -DCPACK_GENERATOR=RPM -DOPAE_BUILD_FPGABIST=ON -DOPAE_BUILD_PYTHON_DIST=ON -DCMAKE_BUILD_PREFIX=/install-opae-sdk make -j `nproc`\nmake -j `nproc` package_rpm\n
The install-opae-sdk directory location was selected for ease of use. If the user wishes to build the OPAE SDK in a different location, they will need to replace the '..' in the above command with the direct or relative path to the opae-sdk repository.
2. After a successful compile, there should be eight packages present:
cd $OFS_BUILD_ROOT/opae-sdk/install-opae-sdk\nls | grep rpm\nopae-2.12.0-5.x86_64.rpm opae-PACSign-2.12.0-5.x86_64.rpm opae-devel-2.12.0-5.x86_64.rpm opae-libs-2.12.0-5.x86_64.rpm opae-opae.admin-2.12.0-5.x86_64.rpm opae-packager-2.12.0-5.x86_64.rpm opae-tests-2.12.0-5.x86_64.rpm opae-tools-2.12.0-5.x86_64.rpm opae-tools-extra-2.12.0-5.x86_64.rpm\n
3. Install the OPAE SDK packages:
cd $OFS_BUILD_ROOT/opae-sdk/install-opae-sdk\nsudo dnf localinstall -y opae*.rpm\n
4. check that all packages have been installed:
rpm -qa | grep opae\nopae-devel-2.12.0-5.x86_64 opae-packager-2.12.0-5.x86_64 opae-2.12.0-5.x86_64 opae-tools-2.12.0-5.x86_64 opae-PACSign-2.12.0-5.x86_64 opae-tools-extra-2.12.0-5.x86_64 opae-opae.admin-2.12.0-5.x86_64 opae-tests-2.12.0-5.x86_64 opae-libs-2.12.0-5.x86_64\n
5. Setup required environment variables
export PATH=$PATH:$OFS_BUILD_ROOT/opae-sdk/install-opae-sdk/bin\nexport LIBRARY_PATH=$OFS_BUILD_ROOT/opae-sdk/install-opae-sdk/lib\nexport LD_LIBRARY_PATH=$OFS_BUILD_ROOT/opae-sdk/install-opae-sdk/lib64\n
cd ../lib/python*/site-packages\nexport PYTHONPATH=$PWD\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#4-compiling-an-afu","title":"4. Compiling An AFU","text":"This section will use the FIM build tree created in the previous steps to compile an example AFU. This section will continue the work with the host_chan_mmio AFU.. You can perform the build steps listed below to demonstrate the ease in building and running a real example on the Intel\u00ae FPGA PAC D5005.
To run the steps in this section, you must complete all steps in section 2. Set Up AFU Development Environment, and ensure the OPAE_PLATFORM_ROOT \"environment variable that points to the directory of the PR build tree generated previously.
Ensure your bashrc file ~/.bashrc have the following line:
export OPAE_PLATFORM_ROOT=$OFS_ROOTDIR/work_d5005/build_tree\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#41-set-afu-synthesis-environment","title":"4.1. Set AFU Synthesis Environment","text":"Here, you will create the synthesis environment to build the host_chan_mmio example. The PIM flow includes the synthesis environment creation script afu_synth_setup for this task. The usage of afu_synth_setup is shown below:
usage: afu_synth_setup [-h] -s SOURCES [-p PLATFORM] [-l LIB] [-f] dst\nGenerate a Quartus build environment for an AFU. A build environment is\ninstantiated from a release and configured for the specified AFU. AFU\nsource files are specified in a text file parsed by rtl_src_config,\nwhich is part of the OPAE base environment.\npositional arguments:\n dst Target directory path (directory must not exist).\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA platform name.\n -l LIB, --lib LIB FPGA platform release hw/lib directory. If not\n specified, the environment variables OPAE_FPGA_HW_LIB\n and then BBS_LIB_PATH are checked.\n -f, --force Overwrite target directory if it exists.\n
Execute afu_synth_setup \"as follows to create the synthesis environment for a host_chan_mmio \"AFU that fits the Intel\u00ae FPGA PAC D5005 FIM previously constructed.
cd $OFS_ROOTDIR/work_d5005/\nafu_synth_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt build_d5005_x16\n
Now, execute the afu_synth command that resides inside the $OFS_ROOTDIR/work_d5005/build_tree/bin directory, to actually build the host_chan_mmio AFU.
cd $OFS_ROOTDIR/work_d5005/build_d5005_x16\n$OPAE_PLATFORM_ROOT/bin/afu_synth\n...\n...\nWrote host_chan_mmio.gbs\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#411-loading-and-running-the-host_chan_mmio-example-afu","title":"4.1.1. Loading and Running the host_chan_mmio example AFU","text":"Once the compilation completes successfully, load the new bitstream file, host_chan_mmio.gbs, into the partial reconfiguration region of the target Intel\u00ae FPGA PAC D5005. Keep in mind, that the loaded image is dynamic - this image is not stored in flash, and if the card is power cycled, then the PR region is re-loaded with the default AFU.
To load the image, perform the following steps:
cd $OFS_ROOTDIR/work_d5005/build_d5005_x16\nsudo fpgasupdate host_chan_mmio.gbs 3b:00.0\n[sudo] password for <<Your username>>: [WARNING ] Update starting. Please do not interrupt.\n[INFO ] Partial Reconfiguration OK\n[INFO ] Total time: 0:00:01.88\n
Determine the BDF of the Intel\u00ae FPGA PAC D5005.
The PCIe BDF address is initially determined when the server powers on. The user can determine the addresses of all Intel\u00ae FPGA PAC D5005 using lspci:
lspci -d :bcce\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
Set up your board to work with the newly loaded host_chan_mmio.gbs
-
Create the Virtual Functions (VFs):
sudo pci_device 3b:00.0 vf 3\n
-
Verify that all three VFs have been created.
$ lspci -s 3b:00\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
- Bind the 3 VFs to the vfio-pci driver.
sudo opae.io init -d , e.g.
$ sudo opae.io init -d 0000:3b:00.1 user:user\nBinding (0x8086,0xbccf) at 0000:3b:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.1 is 142\nAssigning /dev/vfio/142 to <local user>\nChanging permissions for /dev/vfio/142 to rw-rw----\n\n$ sudo opae.io init -d 0000:3b:00.2 user:user\nBinding (0x8086,0xbccf) at 0000:3b:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.2 is 143\nAssigning /dev/vfio/143 to <local user>\nChanging permissions for /dev/vfio/143 to rw-rw-----\n\n$ sudo opae.io init -d 0000:3b:00.3 user:user\nBinding (0x8086,0xbccf) at 0000:3b:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.3 is 144\nAssigning /dev/vfio/144 to <local user>\nChanging permissions for /dev/vfio/144 to rw-rw----\n
- Verify the new AFU is loaded. The host_chan_mmio AFU GUID is 76d7ae9c-f66b-461f-816a-5428bcebdbc5.
$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x603B000000000000\nPCIe s:b:d.f : 0000:3B:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0x403B000000000000\nPCIe s:b:d.f : 0000:3B:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 76d7ae9c-f66b-461f-816a-5428bcebdbc5\n//****** PORT ******//\nObject Id : 0x203B000000000000\nPCIe s:b:d.f : 0000:3B:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n
Run the host_chan_mmio software application to demonstrate the newly loaded AFU image. You navigate to $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw and compile the software application and then run.
If OPAE SDK libraries were not installed in the default systems directory /usr/lib64/ \", define the OPAE_LOC environment variable to point to the directory where the OPAE SDK libraries were installed.
$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib64:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n
cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw\nmake ./host_chan_mmio\n
Console Output:
AFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 250 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#412-loading-and-running-the-hello_world-example-afu","title":"4.1.2. Loading and running the hello_world example AFU","text":"The platform-independent examples-afu repository provides some interesting examples AFU's. In this section, you will compile and execute the PIM-based hello_world AFU. The RTL of the hello_world AFU receives from the host application an address via memory-mapped I/O (MMIO) write and generates a DMA write to the memory line at that address. The content written to memory is the string \"Hello world!\". The host application spins, waiting for the memory line to be updated. Once available, the software prints out the string.
The hello_world example AFU consists of the following files.
hello_world\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 ccip\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u2514\u2500\u2500 hello_world.json\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 hello_world.c\n \u2514\u2500\u2500 Makefile\n
The hw directory contains the RTL to implement the hardware functionality using CCIP, Avalon, and AXI interfaces. However, this guide will use the AXI version of the AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the AFU hardware. The following instructions can be used to compile other AFU samples accompanying this repository.
- If not done already, download and clone the repository.
cd $OFS_BUILD_ROOT git clone https://github.com/OFS/examples-afu.git\n
- Make sure to set the next environment variables.
# OPAE and MPF libraries must either be on the default linker search paths or on both LIBRARY_PATH and LD_LIBRARY_PATH. \n$ export OPAE_LOC=/usr\n $ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n# OPAE_PLATFORM_ROOT points to a release tree that has been configured with the Platform Interface Manager (PIM). \n$ export OPAE_PLATFORM_ROOT=$OFS_ROOTDIR/work_d5005/build_tree\n
-
Compile the hello_word sample AFU.
$ cd $OFS_ROOTDIR/work_d5005\n $ afu_synth_setup -s $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt hello_world_synth\n $ cd hello_world_synth\n $ ${OPAE_PLATFORM_ROOT}/bin/afu_synth\n\n.\n.\n.\nInfo (19538): Reading SDC files took 00:00:06 cumulatively in this process.\nWrote hello_world.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'hello_world.gbs'\nDesign meets timing\n===========================================================================\n
-
To test the AFU in actual hardware, load the hello_world.gbs to the Intel\u00ae FPGA PAC D5005 card. For this step to be successful, the Intel\u00ae FPGA PAC D5005 FIM must have already been loaded to the Intel\u00ae FPGA PAC D5005 card following the steps described in Section 2 of this document.
$ cd $OFS_ROOTDIR/work_d5005/hello_world_synth\n $ sudo fpgasupdate hello_world.gbs 3b:00.0\n [sudo] password for <<Your username>>: [2022-12-06 13:25:10.22] [WARNING ] Update starting. Please do not interrupt.\n[2022-12-06 13:25:12.06] [INFO ] Partial Reconfiguration OK\n[2022-12-06 13:25:12.06] [INFO ] Total time: 0:00:01.83\n
Set up your Intel\u00ae FPGA PAC D5005 board to work with the newly loaded hello_world.gbs file.
# Create the Virtual Functions (VFs):\n$ sudo pci_device 3b:00.0 vf 3\n# Verify:\n$ lspci -s 3b:00\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.1 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.2 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.3 Processing accelerators: Intel Corporation Device bccf (rev 01)\n# Bond VFs to VFIO driver. Enter <<Your username>>\nsudo opae.io init -d 0000:3b:00.1 <Your username>\n Unbinding (0x8086,0xbcce) at 0000:3b:00.1 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:3b:00.1 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:3b:00.1 is 142\nAssigning /dev/vfio/142 to <Your username>\n Changing permissions for /dev/vfio/142 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.2 <Your username>\n Unbinding (0x8086,0xbccf) at 0000:3b:00.2 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:3b:00.2 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:3b:00.2 is 143\nAssigning /dev/vfio/143 to <Your username>\n Changing permissions for /dev/vfio/143 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.3 <Your username>\n Unbinding (0x8086,0xbccf) at 0000:3b:00.3 from dfl-pci\n Binding (0x8086,0xbccf) at 0000:3b:00.3 to vfio-pci\n iommu group for (0x8086,0xbccf) at 0000:3b:00.3 is 144\nAssigning /dev/vfio/144 to <Your username>\n Changing permissions for /dev/vfio/144 to rw-rw----\n\n# < Verify the new AFU is loaded. The hello_world AFU GUID is \"c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\".\n$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x603B000000000000\nPCIe s:b:d.f : 0000:3B:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n//****** PORT ******//\nObject Id : 0x403B000000000000\nPCIe s:b:d.f : 0000:3B:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n//****** PORT ******//\nObject Id : 0x203B000000000000\nPCIe s:b:d.f : 0000:3B:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n
- Compile and execute the host application of the
hello_world AFU. You should see the application outputs the \"Hello world!\" message in the terminal.
# Move to the sw directory of the hello_world AFU and run the following commands in user mode\ncd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw/\n\nmake\n\n# Launch the host application\n./hello_world\n Hello world TLP!\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#413-modify-the-afu-user-clocks-frequency","title":"4.1.3. Modify the AFU user clocks frequency","text":"An OPAE compliant AFU specifies the frequency of the uclk_usr and uclk_usr_div2 clocks through the JSON file for AFU configuration located under the <afu_example>/hw/rtl directory of an AFU design. For instance, the AFU configuration file of the host_chan_mmio example is $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/host_chan_mmio.json.
The AFU specifies the frequency for uClk_usr in its platform configuration file using the following key:value pairs:
\"clock-frequency-high\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\n \"clock-frequency-low\": [<float-value>|\u201dauto\u201d|\u201dauto-<float-value>\u201d]\n
These key:value tuples are used to configure the PLL of the target platform that provides the user clocks through the AFU clocks interface. In addition, the specified frequency affects the timing closure process on the user clocks during AFU compilation.
Setting the value field to a float number (e.g., 315.0 to specify 315 MHz) drives the AFU generation process to close timing within the bounds set by the low and high values and sets the AFU's JSON metadata to specify the user clock PLL frequency values.
The following example shows the JSON file of the host_chan_mmio to set the AFU uClk to 300 MHz and uClk_div2 to 150 MHz.
{\n \"version\": 1,\n \"afu-image\": {\n \"power\": 0,\n \"clock-frequency-high\": 300,\n \"clock-frequency-low\": 150,\n \"afu-top-interface\":\n {\n \"class\": \"ofs_plat_afu\"\n },\n \"accelerator-clusters\":\n [\n {\n \"name\": \"host_chan_mmio\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\"\n }\n ]\n }\n}\n
Save the changes to host_chan_mmio.json file, then execute the afu_synth_setup script to create a new copy of the AFU files with the modified user clock settigns.
$ cd $OFS_ROOTDIR/work_d5005\n $ afu_synth_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt build_d5005_afu_clks\n\nLoading platform database: /home/<Your username>/<Your localpath>/ofs-d5005/work_d5005/build_tree/hw/lib/platform/platform_db/ofs_d5005.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting platform/platform_afu_top_config.vh\nWriting platform/platform_if_addenda.qsf\nWriting ../hw/afu_json_info.vh\n
Compile the host_chan_mmio AFU with the new frequency values. cd $OFS_ROOTDIR/work_d5005/build_d5005_afu_clks\n $OFS_ROOTDIR/work_d5005/build_tree/bin/afu_synth\n
During the compilation phase, you will observe the Timing Analyzer uses the specified user clock frequency values as the target to close timing.
AFU developers must ensure the AFU hardware design meets timing. The compilation of an AFU that fails timing shows a message similar to the following.
.\n.\n.\n\nWrote host_chan_mmio.gbs\n\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\n*** Design does not meet timing\n *** See build/syn/syn_top/output_files/timing_report\n\n===========================================================================\n
The previous output indicates the location of the timing reports for the AFU designer to identify the failing paths and perform the necessary design changes. Next, is a listing of the timing report files from a host_chan_mmio AFU that fails to meet timing after modifying the user clock frequency values.
$ cd $OFS_ROOTDIR/work_d5005/build_d5005_afu_clks\n $ ls build/syn/syn_top/output_files/timing_report\n\nclocks.rpt clocks.sta.fail.summary clocks.sta.pass.summary iofs_pr_afu_2_slow_900mv_0c_recovery.rpt\niofs_pr_afu_2_slow_900mv_0c_setup.rpt\niofs_pr_afu_2_slow_900mv_100c_recovery.rpt\niofs_pr_afu_2_slow_900mv_100c_setup.rpt\niofs_pr_afu_2_slow_vid2_0c_recovery.rpt\niofs_pr_afu_2_slow_vid2_0c_setup.rpt\niofs_pr_afu_2_slow_vid2_100c_recovery.rpt\niofs_pr_afu_2_slow_vid2_100c_setup.rpt\niofs_pr_afu_MIN_fast_900mv_0c_recovery.rpt\niofs_pr_afu_MIN_fast_900mv_0c_setup.rpt\niofs_pr_afu_MIN_fast_900mv_100c_recovery.rpt\niofs_pr_afu_MIN_fast_900mv_100c_setup.rpt\n
Warning: AFU developers must inform software developers of the maximum operating frequency (Fmax) of the user clocks to avoid any unexpected behavior of the accelerator and potentially of the overall system.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#5-simulating-an-afu-using-ase","title":"5. Simulating an AFU using ASE","text":"The AFU Simulation Environment (ASE) is a hardware/software co-simulation environment for your AFU. See diagram below illustrating ASE operation:
ASE uses the simulator Direct Programming Interface (DPI) to provide HW/SW connectivity. The PCIe connection to the AFU under testing is emulated with a transactional model.
The following list describes ASE operation:
- Attempts to replicate the transactions that will be seen in real system.
- Provides a memory model to AFU, so illegal memory accesses can be identified early.
- Not a cache simulator.
- Does not guarantee synthesizability or timing closure.
- Does not model system latency.
- No administrator privileges are needed to run ASE. All code is user level.
The remainder of this section is a tutorial providing the steps on how to run ASE with either VCS or QuestaSim using an example AFU and the AFU build tree previously created in this guide.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#51-set-up-steps-to-run-ase","title":"5.1. Set Up Steps to Run ASE","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"The Intel\u00ae FPGA PAC D5005 PAC card requires opae-2.12.0-5. Follow the instructions documented in the Software Installation Guide: OFS for PCIe Attach FPGAs to build and install the required OPAE SDK for the Intel\u00ae FPGA PAC D5005 PAC card. Make sure to check out the cloned repository to tag 2.12.0-5 and branch release/2.12.0.
git checkout tags/2.12.0-5 -b release/2.12.0\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#512-install-ase-tools","title":"5.1.2 Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE-SDK. However, the recommendation is to install it in the same target directory as OPAE-SDK.
-
If not done already, set the environment variables as described in section, Set Up AFU Development Environment.
-
Clone the ase-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n $ cd opae-sim
2. Building ASE requires the include file mock/opae_std.h. If the OPAE-SDK was installed under the default system directories, the C_INCLUDE_PATH variable must be set as follows. export C_INCLUDE_PATH=\"/usr/src/debug/opae-2.12.0-5.el8.x86_64/tests/framework\"\n
- Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
mkdir build\n cd build\n cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr.
sudo make install
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#513-setup-required-ase-environment-variables","title":"5.1.3. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
cd /usr/bin\n export PATH=$PWD:$PATH\ncd ../lib/python*/site-packages\n export PYTHONPATH=$PWD\ncd /usr/lib\n export LIBRARY_PATH=$PWD\ncd /usr/lib64\n export LD_LIBRARY_PATH=$PWD\ncd $OFS_BUILD_ROOT/ofs-platform-afu-bbb\n export OFS_PLATFORM_AFU_BBB=$PWD\ncd $OFS_ROOTDIR/work_d5005/build_tree\n export OPAE_PLATFORM_ROOT=$PWD\n## For VCS, set the following:\nexport VCS_HOME=<Set the path to VCS installation directory>\n export PATH=$VCS_HOME/bin:$PATH\n## For QuestaSIM, set the following:\nexport MTI_HOME=<path to Modelsim installation directory>\n export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#52-simulating-the-host_chan_mmio-afu","title":"5.2. Simulating the host_chan_mmio AFU","text":"The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files:
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_axi.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\n
This example AFU contains examples using both Avalon and AXI interface buses. This guide will use the AXI version of the host_chan_mmio AFU.
ASE uses client-server application architecture to deliver hardware/software co-simulation. You require one shell for the hardware based simulation and another shell where the software application is running. The hardware is started first with a simulation compilation and simulator startup script, once the simulator has loaded the design, it will wait until the software process starts. Once the software process starts, the simulator proceeds. Transaction logging and waveform capture is performed.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#521-set-up-and-run-the-hw-simulation-process","title":"5.2.1 Set Up and Run the HW Simulation Process","text":"You will run the afu_sim_setup script to create the scripts for running the ASE environment. The afu_sim_setup script has the following usage:
usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n [-f] [--ase-mode ASE_MODE] [--ase-verbose]\n dst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\n Default simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
Run afu_sim_setup to create the ASE simulation environment for the host_chan_mmio example AFU. The '-t VCS' option indicates to prepare the ASE simulation environment for VCS.
cd $OFS_ROOTDIR/work_d5005/\n\nafu_sim_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt -t VCS host_chan_mmio_sim\n\nCopying ASE from /opae-sdk/install-opae-sdk/share/opae/ase...\nCopying ASE from /usr/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\n\nTool Brand: VCS\nLoading platform database: /home/<Your username>/<Your localpath>/ofs-d5005/work_d5005/build_tree/hw/lib/platform/platform_db/ofs_d5005.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup creates the ASE scripts in the directory host_chan_mmio_sim where the afu_sim_setup script was run. Start the simulator as shown below in user mode:
cd host_chan_mmio_sim\n make\n make sim\n
This process launches the AFU hardware simulator. Before moving to the next section, pay attention to the simulator output highlighted in the image below.
The simulation artifacts are stored in host_chan_mmio/work and consist of:
log_ase_events.tsv\nlog_ofs_plat_host_chan.tsv \nlog_ofs_plat_local_mem.tsv \nlog_pf_vf_mux_A.tsv \nlog_pf_vf_mux_B.tsv \n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#522-set-up-and-run-the-sw-process","title":"5.2.2 Set Up and Run the SW Process","text":"Open an additional shell to build and run the host application that communicates with the actual AFU hardware. Set up the same environment variable you have set up in the shell you have been working on until this point.
Additionally, as indicated by the hardware simulator output that is currently executing in the \"simulator shell\", copy and paste the line \"export ASE_WORKDIR=...\", into the new \"software shell\". See the last image of the previous section.
export ASE_WORKDIR= <<as directed in HW simulation shell>>\n
Then, go to the sw directory of the host_chan_mmio AFU example to compile the host application. cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw make\n\nafu_json_mgr json-info --afu-json=../hw/rtl/host_chan_mmio.json --c-hdr=obj/afu_json_info.h\nWriting obj/afu_json_info.h\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c main.c -o obj/main.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c test_host_chan_mmio.c -o obj/test_host_chan_mmio.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/connect.c -o obj/connect.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/csr_mgr.c -o obj/csr_mgr.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/hash32.c -o obj/hash32.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/test_data.c -o obj/test_data.o\ncc -o host_chan_mmio obj/main.o obj/test_host_chan_mmio.o obj/connect.o obj/csr_mgr.o obj/hash32.o obj/test_data.o -z noexecstack -z relro -z now -pie -luuid -lopae-c\n
Now, launch the host application to exercise the AFU hardware running on the simulator shell. The next image shows the AFU hardware simulation process on the left side shell. The right hand shell shows the host application's output of a successful simulation.
Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
make wave\n
This brings up the VCS simulator GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ofs_plat_afu | afu , as shown below.
Right click on the afu (afu) entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#53-simulating-the-hello_world-afu","title":"5.3 Simulating the hello_world AFU","text":"In this section, you will quickly simulate the PIM-based hello_world sample AFU accompanying the examples-afu repository.
-
Set the environment variables as described in section 5.1. Set Up Steps to Run ASE.
-
Prepare an RTL simulation environment for the AXI version of the hello_world AFU.
Simulation with ASE requires two software processes, one to simulate the AFU RTL and the other to run the host software that exercises the AFU. To construct an RTL simulation environment under the directory $OFS_ROOTDIR/work_d5005, execute the following.
$ cd $OFS_ROOTDIR/work_d5005\n $ afu_sim_setup -s $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt -t VCS hello_world_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/<Your username>/<Your localpath>/ofs-d5005/work_d5005/build_tree/hw/lib/platform/platform_db/ofs_d5005.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup script constructs an ASE environment in the hello_world_sim subdirectory. If the command fails, confirm that the path to the afu_sim_setup is on your PATH environment variable (in the OPAE SDK bin directory) and that your Python version is at least 3.7.
-
Build and execute the AFU RTL simulator in user mode.
cd $OFS_ROOTDIR/work_d5005/hello_world_sim\n make\n make sim
The previous commands will build and run the VCS RTL simulator, which prints a message saying it is ready for simulation. The simulation process also prints a message instructing you to set the ASE_WORKDIR environment variable in a second shell.
-
Open a second shell where you will build and execute the host software. In this new \"software shell\", set up the environment variables you have set up so far in the \"hardware simulation\" shell.
-
Also, set the ASE_WORKDIR environment variable following the instructions given in the \"hardware simulation\" shell.
export ASE_WORKDIR=$OFS_ROOTDIR/work_d5005/hello_world_sim/work\n
6. Then, move to the sw directory of the hello_world AFU sample to build the host software. cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n make
- Run the
hello_world host application to resume the work of the RTL simulation. The host software process and the RTL simulation execute in lockstep. If successful, you should see the Hello world! output.
$ with_ase ./hello_world\n\n[APP] Initializing simulation session ...\nHello world!\n [APP] Deinitializing simulation session\n [APP] Took 43,978,424 nsec\n [APP] Session ended\n
The image below shows the simulation of the AFU hardware and the execution of the host application side-by-side.
- Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
make wave\n
This brings up the DVE GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the AFU instance located under, ase_top | ase_top_plat | ofs_plat_afu, as shown below.
Right click on the ofs_plat_afu entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#6-adding-remote-signal-tap-logic-analyzer-to-debug-the-afu","title":"6. Adding Remote Signal Tap Logic Analyzer to debug the AFU","text":"The OPAE SDK provides a remote Signal Tap facility. It also supports the following in system debug tools included with the Quartus\u00ae Prime Pro Edition:
- In-system Sources and Probes
- In-system Memory Content Editor
- Signal Probe
- System Console
This section is a short guide on adding remote Signal Tap instances to an AFU for in-system debugging. In order of execution, you can follow the steps in the following sections to create an instrumented AFU. The host_chan_mmio AFU is used in this guide as the target AFU to be instrumented.
You need a basic understanding of Signal Tap. Please see the Signal Tap Logic Analyzer: Introduction & Getting Started Web-Based Training for more information.
You will run with a Signal Tap GUI running locally on the server with the Intel\u00ae FPGA PAC D5005 as shown below:
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#61-adding-rstp-to-the-host_chan_mmio-afu","title":"6.1. Adding RSTP to the host_chan_mmio AFU","text":"RSTP is added to an AFU by:
- Defining signals to be instrumented in Signal Tap. Create a new *.stp file.
- Modify ofs_top.qpf to include the new *.stp file
- Modify ofs_top.qsf
- Modify ofs_pr_afu.qsf
- Re-run afu_synth_setup to update project settings
- Re-run $OPAE_PLATFORM_ROOT/bin/afu_synth to build the PR -able image containing the RSTP instance
The following steps use the previously built host_chan_mmio AFU example. You can use these detailed steps to instrument your AFU.
-
Navigate to host_chan_mmio AFU Quartus project and open the project using Quartus GUI.
cd $OFS_ROOTDIR/work_d5005/build_d5005_x16/build/syn/syn_top\n $ quartus d5005.qpf &\n
-
Once the project is loaded in Quartus, review the project hierarchy as shown in the Project Navigator. This example will add Signal Tap probe points to the AFU region. Reviewing the code will give insight into the function of this block. You can up the code in the Project Navigator by expanding afu_top - port_gasket - pr_slot - afu_main - ofs_plat_afu, then select instance afu, right-click, select Locate Node - Locate in Design File as shown below.
-
Bring up Signal Tap to create the *.stp file. In the Quartus GUI, go to Tools - Signal Tap Logic Analyzer. Click Create to accept the default template in the New File from Template pop-up. The Signal Tap Logic Analyzer window comes up.
-
Set up the clock for the Signal Tap logic instance by clicking ... button as shown below:
5. The Node Finder comes up, and you will click ... as shown below to bring up the hierarchy navigator or copy-paste the following location at Look in:
iofs_top|afu_top|port_gasket|pr_slot|afu_main|ofs_plat_afu|afu\n
-
In the Select Hierarchy Level, navigate to top - afu_top - port_gasket - pr_slot - afu_main - ofs_plat_afu, then select instance afu and click Ok.
-
Enter *clk* in the Named: box and click Search. This brings up matching terms. Click mmio64_if.clk and >. Verify your Node Finder is as shown below and then click Ok:
-
Double click the Double-click to add nodes and once again, click ... and navigate to top - afu_top - port_gasket - pr_slot - afu_main - ofs_plat_afu, then select instance afu and click Ok. Enter mmio64 then click >> to add these signals to the STP instance as shown below:
Then click Insert and Close.
-
Save the newly created STP by clicking File - Save As in the save as navigate to $OFS_ROOTDIR/work_d5005/build_d5005_x16/build/syn/syn_top and save the STP file as host_chan_mmio.stp as shown below:
- Edit
ofs_top.qsf to add host_chan_mmio.stp file and enable STP. Open $OFS_ROOTDIR/work_d5005/build_d5005_x16/build/syn/syn_top/d5005.qpf in an editor and modify lines as shown below:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE host_chan_mmio.stp\nset_global_assignment -name SIGNALTAP_FILE host_chan_mmio.stp\n
Save the d5005.qpf and close Quartus.
- Edit
iofs_pr_afu.qsf to add host_chan_mmio.stp file and enable STP. Open $OPAE_PLATFORM_ROOT/hw/lib/build/syn/syn_top/iofs_pr_afu.qsf in an editor and ensure the lines are included as below (note: the verilog macro INCLUDE_REMOTE_STP will already be present), also copy and paste the file host_chan_mmio.stp in this location:
The updated lines are:
set_global_assignment -name VERILOG_MACRO \"INCLUDE_REMOTE_STP\"\nset_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE host_chan_mmio.stp\nset_global_assignment -name SIGNALTAP_FILE host_chan_mmio.stp\n
Save the iofs_pr_afu.qsf and ensure Quartus is closed. - The afu_synth script is run to create a new copy of AFU files. In your original build shell, enter the following commands:
$ cd $OFS_ROOTDIR/build_d5005_x16\n $ afu_synth_setup -s $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/hw/rtl/test_mmio_axi1.txt build_d5005_x16_stp\n\n Notice that your previous build_d5005_x16_stp directory is preserved, and a new build_d5005_x16_stp directory is created. You will use build_d5005_x16_stp to build the STP-enabled image.\n\n $ cd build_d5005_x16_stp\n $ $OPAE_PLATFORM_ROOT/bin/afu_synth\n\n...\n...\nWrote host_chan_mmio.gbs\n\n===========================================================================\n PR AFU compilation complete\n AFU gbs file is 'host_chan_mmio.gbs'\n Design meets timing\n===========================================================================\n
- Once compilation completes, the new host_chan_mmio.gbs file that contains the Signal Tap instance can be loaded.
$ sudo fpgasupdate host_chan_mmio.gbs 3b:00.0\n[sudo] password for <myuser>: \n[WARNING ] Update starting. Please do not interrupt.\n [INFO ] \nPartial Reconfiguration OK\n[INFO ] Total time: 0:00:01.87\n
- Use the OPAE SDK mmlink tool to create a TCP/IP connection to your Stratix\u00ae 10 FPGA card under test. The mmlink command has the following format:
Usage:\nmmlink\n<Segment> --segment=<SEGMENT NUMBER>\n<Bus> --bus=<BUS NUMBER> OR -B <BUS NUMBER>\n<Device> --device=<DEVICE NUMBER> OR -D <DEVICE NUMBER>\n<Function> --function=<FUNCTION NUMBER> OR -F <FUNCTION NUMBER>\n<Socket-id> --socket-id=<SOCKET NUMBER> OR -S <SOCKET NUMBER>\n<TCP PORT> --port=<PORT> OR -P <PORT>\n<IP ADDRESS> --ip=<IP ADDRESS> OR -I <IP ADDRESS>\n<Version> -v,--version Print version and exit\n
ProTip:
Open a new shell session for mmlink; this console needs to remain open to allow mmlink connection.
Enter the command below to create a connection using port 3333:
$ sudo mmlink -P 3333 -B 0x3b\n\n------- Command line Input START ----\n\nSocket-id : -1\n Port : 3333\nIP address : 0.0.0.0\n ------- Command line Input END ----\n\nPORT Resource found.\nServer socket is listening on port: 3333\n
Leave this shell open with the mmlink connection.
- In this step, you will open a new shell and enable JTAG over protocol. You must have Quartus Prime Pro \u00ae ${{ env.D5005_QUARTUS_PRIME_PRO_VER_S }} Programmer loaded on the Intel\u00ae FPGA PAC D5005 server for local debugging.
$ jtagconfig --add JTAG-over-protocol sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0\n\nVerify connectivity with jtagconfig --debug\n\n$ jtagconfig --debug\n1) JTAG-over-protocol [sti://localhost:0/intel/remote-debug/127.0.0.1:3333/0]\n(JTAG Server Version ${{ env.D5005_QUARTUS_PRIME_PRO_VER_S }}.0 Build 104 09/14/2022 SC Pro Edition)\n020D10DD VTAP10 (IR=10)\nDesign hash D41D8CD98F00B204E980\n + Node 00406E00 Virtual JTAG #0\nCaptured DR after reset = (020D10DD) [32]\nCaptured IR after reset = (155) [10]\nCaptured Bypass after reset = (0) [1]\nCaptured Bypass chain = (0) [1]\n
- Start Quartus Signal Tap GUI, connect to target, load stp file by navigating to $OPAE_PLATFORM_ROOT/hw/lib/build/syn/syn_top/ . The Quartus Signal Tap must be the same version of Quartus used to compile the host_chan_mmio.gbs. Quartus Prime Pro \u00ae ${{ env.D5005_QUARTUS_PRIME_PRO_VER_S }} Pro is used in the steps below:
cd $OPAE_PLATFORM_ROOT/hw/lib/build/syn/syn_top/\nquartus_stpw host_chan_mmio.stp\n
This command brings up Signal Tap GUI. Connect to the Signal Tap over protocol by selecting the Hardware button on the right side of the GUI and clicking the \"Please Select\" pull-down as shown below:
JTAG over protocol selected:
This connection process will take approximately 2-3 minutes for the Signal Tap instance to indicate \"Ready to acquire\".
8) Set the trigger condition for a rising edge on signal valid signal. 9) In the Signal Tap window, enable acquisition by pressing key F5. The Signal Tap GUI will indicate \"Acquisition in progress\". Run the hello_world application and observe that the Signal Tap instance has triggered. You should see signals being captured in the Signaltap GUI.
See captured image below:
To end your Signal Tap session, close the Signal Tap GUI, then in the mmlink shell, enter ctrl c to kill the mmlink process.
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#7-disabling-the-flr-function-level-reset-during-afu-debugging","title":"7. Disabling the FLR (Function Level Reset) during AFU Debugging","text":"The vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\n
Enable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\n
If you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\n
Sample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\n
"},{"location":"hw/d5005/dev_guides/afu_dev/ug_dev_afu_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/","title":"Shell Developer Guide: Open FPGA Stack for Stratix 10\u00ae FPGA","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a design guide for FPGA developers, system architects and hardware developers using OFS as a starting point for the creating the FPGA Interface Manager (FIM) for a custom FPGA acceleration board or Platform with Altera FPGAs.
This document uses the Intel\u00ae FPGA PAC D5005 as an example platform to illustrate key points and demonstrate how to extend the capabilities provided in OFS (Open FPGA Stack) to custom platforms. The demonstration steps serves as a tutorial for the development of your OFS knowledge.
This document covers OFS architecture lightly. For more details on the OFS architecture, please see Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
You are encouraged to read Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs to fully understand how AFU Developers will use your newly developed FIM.
Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#12-introduction","title":"1.2. Introduction","text":"Open FPGA Stack (OFS) addresses the scalability for FPGA acceleration boards and workloads by providing a powerful and systematic methodology for the rapid development of FPGA-based Acceleration systems. This methodology addresses the key challenges of hardware, software and workload developers by providing a complete FPGA project consisting of RTL and simulation code, build scripts and software. The FPGA project released in OFS can be rapidly customized to meet new market requirements by adding new features, custom IPs and interface subsystems IPs.
A high-level overview of the OFS Stratix\u00ae 10 FPGA hardware architecture on the Stratix\u00ae 10 FPGA reference platform, Intel\u00ae FPGA PAC D5005 is shown in the below figure. The provided FPGA architecture is divided into two main components
- The outer area in white, the FPGA Interface manager (or FIM) - The inner area in green, the Acceleration Function Unit or AFU Region.
The outer area, the FIM, provides the core infrastructure and interfaces within the FPGA. The AFU region is where a user\u2019s custom logic would reside for their specific workload.
* FPGA external interfaces and IP cores (e.g. Ethernet, DDR-4, PCIe, etc) * PLLs/resets * FPGA - Board management infrastructure * Interface to Acceleration Function Unit (AFU)
The AFU region has both static and dynamic partial reconfiguration regions enabling a lot of customization.
* Uses the FIM interfaces to perform useful work inside the FPGA * Contains logic supporting partial reconfiguration * Remote Signal Tap core for remote debugging of workload
Outside of the FPGA is the Board Management Controller which provides board management, root of trust, board monitoring, and remote system updates.
The overall architecture is built to be very composable and modular in blocks that can be modified while leaving the rest of the infrastructure intact so you may only need to modify a few of these blocks.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#12-release-capabilities","title":"1.2. Release Capabilities","text":"This release of OFS FIM supports the following key features:
- 1 - Host channel interface via PCIe Gen 3 x 16 SRIOV (1PF, 3 VF, AXI-S TLP packets)
- DDR4 SDRAM External memory interface (AXI-M)
- 1 - 10G Ethernet interfaces (1x10G)
- MSI-X Interrupts (PF, VF)
- 1 - AFU
- Exercisers demonstrating PCIe, external memory and Ethernet interfaces
- Port, FME CSR
- Remote Signal Tap
OFS is extensible to meet the needs of a broad set of customer applications, however not all use cases are easily served. The general uses cases listed below are examples where the OFS base design can be easily re-used to build a custom FIM: 1. Use OFS reference design as-is - Porting the code to another platform that is identical to the OFS reference platform only changing target FPGA device and pinout - Change I/O assignments without changing design 2. Update the configuration of peripheral IP in OFS reference design, not affecting FIM architecture - External memory settings - HSSI analog settings 3. Remove/update peripheral feature in OFS reference design, not affecting FIM architecture - External memory speed/width change - Change 10G Ethernet to 25 or 100G Ethernet IP - Change number of VFs supported 4. Add new features as an extension to OFS reference design, not affecting FIM architecture - Add/remove external memory interface to the design - Add/remove user clocks for AFU - Add/remove IP to the design with connection to AFU
More advanced use cases requiring changes or additions to the host PCIe channel are not easily supported with this release of the OFS FIM.
Reuse of the provided host management FPGA logic and software is the fastest and most simple approach to FIM customization.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#13-prerequisites","title":"1.3. Prerequisites","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#131-base-knowledge-and-skills-prerequisites","title":"1.3.1. Base Knowledge and Skills Prerequisites","text":"OFS is an advanced application of FPGA technology. This guide assumes you have the following FPGA logic design-related knowledge and skills:
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition design flow.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL and coding practices for FPGA implementation.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Signal Tap Logic Analyzer tool software.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#132-development-environment","title":"1.3.2. Development Environment","text":"To run the tutorial steps in this guide requires this development environment:
Item Version Quartus Prime Pro Quartus Prime Pro 23.4 (with license patch) Target D5005 Sever Operating System RHEL 8.6 OPAE SDK 2.12.0-5 Linux DFL ofs-2024.1-6.1-2 Python 3.6.8 cmake 3.15 GCC 8.5.0 perl 5.8.8 The following server and Intel PAC card are required to run the examples in this guide:
- Qualified Intel Xeon \u00ae server see Qualified Servers.
- Intel\u00ae FPGA PAC D5005 with root entry hash erased (Please contact Altera for root entry hash erase instructions). The standard Intel\u00ae FPGA PAC D5005 card is programmed to only allow the FIM binary files signed by Altera to be loaded. The root entry hash erase process will allow newly created, unsigned FIM binary files to be loaded.
- Intel\u00ae FPGA PAC D5005 installed in the qualified server following instructions in Board Installation Guide: OFS for Acceleration Development Platforms.
The steps included in this guide have been verified in the Dell R740 and HPE ProLiant DL380 Gen10 servers.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#2-high-level-description","title":"2. High Level Description","text":"The FIM targets operation in the Intel\u00ae FPGA PAC D5005 card. The block diagram of the D5005 is shown below:
The key D5005 FPGA interfaces are:
- Host interface - PCIe Gen3 x 16
- Network interface
- 2 - QSFP28 cages
- Current FIM supports 1 x 10 GbE, other interfaces can be created
- External Memory
- 2 or 4 channels of DDR4-2400 to RDIMM modules
- RDIMM modules = 8GB organized as 1 Gb X 72
- Board Management
- SPI interface
- FPGA configuration
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#21-fpga-interface-manager-overview","title":"2.1. FPGA Interface Manager Overview","text":"The FPGA Interface Manager architecture is shown in the below diagram:
The FIM consists of the following components - PCIe Subsystem - Memory Subsystem - HSSI Subsystem - Platform Management Component Intercommunications (PMCI) - Board Peripheral Fabric (BPF) - AFU Peripheral Fabric (APF) - Port Gasket - AXI-S PF/VF Demux/Mux - Host Exerciser Modules - HE-MEM, HE-LB, HE-HSSI - FPGA Management Engine (FME)
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#22-fim-fpga-resource-usage","title":"2.2. FIM FPGA Resource Usage","text":"The FIM uses a small portion of the available FPGA resources. The table below shows resource usage for a base FIM built with 2 channels of external memory, a small AFU instantiated that has host CSR read/write, external memory test and Ethernet test functionality.
Entity ALMs Used % ALMS Used M20Ks % M20Ks used DSP Blocks Pins IOPLLs OFS_top 125009.4 13.0% 661 5.4% 0 630 15 afu_top 70522.7 7.0% 228 2.4% 0 0 1 auto_fab_0 1305.7 0.0% 9 0.1% 0 0 0 bpf_rsv_5_slv 0.6 0.0% 0 0.0% 0 0 0 bpf_rsv_6_slv 0.6 0.0% 0 0.0% 0 0 0 bpf_rsv_7_slv 0.4 0.0% 0 0.0% 0 0 0 bpf 241.9 0.0% 0 0.0% 0 0 0 emif_top_inst 10508.6 1.0% 0 0.0% 0 0 12 eth_ac_wrapper 6024.8 0.5% 9 0.1% 0 0 0 fme_top 615.5 0.2% 7 0.1% 0 0 0 pcie_wrapper 35424.7 3.5% 348 2.9% 0 0 1 pmci_top 318.5 0.1% 0 0.0% 0 0 0 rst_ctrl 40.2 0.0% 0 0.0% 0 0 0 sys_pll 0.5 0.0% 0 0.0% 0 0 1 Total ALMS 933,120 Total M20Ks 11,721 Summary FPGA Resource Utilization Logic utilization (in ALMs) 124,092 / 933,120 ( 13 % ) Total dedicated logic registers 282822 Total pins 630 / 912 ( 69 % ) Total block memory bits 3,425,120 / 240,046,080 ( 1 % ) Total RAM Blocks 661 / 11,721 ( 6 % ) Total DSP Blocks 0 / 5,760 ( 0 % ) Total eSRAMs 0 / 75 ( 0 % ) Total HSSI P-Tiles 17 / 48 ( 35 % ) Total HSSI E-Tile Channels 17 / 48 ( 35 % ) Total HSSI HPS 0 / 1 ( 0 % ) Total HSSI EHIPs 0 / 2 ( 0 % ) Total PLLs 36 / 104 ( 35 % )"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#23-ofs-directory-structure","title":"2.3. OFS Directory Structure","text":"The OFS Git OFS repository ofs-d5005 directory structure is shown below:
\u251c\u2500\u2500 eval_script\n| \u251c\u2500\u2500 ofs_d5005_eval.sh\n| \u2514\u2500\u2500 README_ofs_d5005_eval.txt\n\u251c\u2500\u2500 ipss\n\u2502 \u251c\u2500\u2500 hssi\n| \u251c\u2500\u2500 mem\n| \u251c\u2500\u2500 pcie\n| \u251c\u2500\u2500 pmci\n| \u251c\u2500\u2500 spi\n| \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 license\n\u2502 \u2514\u2500\u2500 quartus-0.0-0.01Intel OFS-linux.run\n\u251c\u2500\u2500 ofs-common\n| \u251c\u2500\u2500 scripts\n| \u251c\u2500\u2500 src\n| \u251c\u2500\u2500 verification\n| \u251c\u2500\u2500 LICENSE.txt\n| \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 sim\n| \u251c\u2500\u2500 bfm\n| \u251c\u2500\u2500 rp_bfm\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 scripts\n| \u251c\u2500\u2500 unit_test \u2502\u00a0\u00a0 \u2514\u2500\u2500 readme.txt\n\u251c\u2500\u2500 src\n\u2502 \u251c\u2500\u2500 afu_top\n\u2502 \u251c\u2500\u2500 includes\n\u2502 \u251c\u2500\u2500 pd_qsys\n\u2502 \u251c\u2500\u2500 README.md\n\u2502 \u2514\u2500\u2500 top\n\u251c\u2500\u2500 syn\n\u2502 \u251c\u2500\u2500 scripts\n\u2502 \u251c\u2500\u2500 setup\n\u2502 \u251c\u2500\u2500 syn_top\n\u2502 \u251c\u2500\u2500 readme.txt\n\u2502 \u2514\u2500\u2500 README\n\u251c\u2500\u2500 LICENSE.txt\n\u2514\u2500\u2500 README.md\n
The contents of each directory are described below:
Eval Script - Contains scripts for evaluation of OFS for D5005 including compiling FIM/AFU from source, unit level test. Also includes resources to report and setup D5005 development environment
ipss - Contains the code and supporting files that define or set up the IP subsystems (HSSI, PCIe, memory, PMCI, SPI, etc...) contained in the D5005 FPGA Interface Manager (FIM).
license - License file for the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core.
ofs-common - This directory contains resources that may be used across the board-specific repositories. This directory is referenced via a link within each of the FPGA-specific repositories.
sim - Contains the testbenches and supporting code for all the unit test simulations. - Bus Functional Model code is contained here. - Scripts are included for automating a myriad of tasks. - All of the individual unit tests and their supporting code is also located here.
src - SystemVerilog source and script files - Contains all of the structural and behavioral code for the FIM. - Scripts for generating the AXI buses for module interconnect. - Top-level RTL for synthesis is located in this directory. - Accelerated Functional Unit (AFU) infrastructure code is contained in this directory.
syn - This directory contains all of the scripts, settings, and setup files for running synthesis on the FIM.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#3-description-of-sub-systems","title":"3. Description of Sub-Systems","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#31-host-control-and-data-flow","title":"3.1. Host Control and Data Flow","text":"The host control and data flow are shown in the diagram below:
The control and data flow is composed of the following:
- Host Interface Adapter (PCIe)
- Low Performance Peripherals
- Slow speed peripherals (I2C, Smbus, etc)
- Management peripherals (FME)
- High Performance Peripherals
- Memory peripherals
- Acceleration Function peripherals
- HPS Peripheral
- Fabrics
- Peripheral Fabric (multi drop)
- AFU Streaming fabric (point to point)
Peripherals are connected to one another using AXI:
- Via the peripheral fabric (AXI4-Lite, multi drop)
- Via the AFU streaming fabric (AXI-S, point to point)
Peripherals are presented to software as:
- OFS managed peripherals that implement DFH CSR structure.
- Native driver managed peripherals (i.e. Exposed via an independent PF, VF)
The peripherals connected to the peripheral fabric are primarily OPAE managed resources, whereas the peripherals connected to the AFU are \u201cprimarily\u201d managed by native OS drivers. The word \u201cprimarily\u201d is used since the AFU is not mandated to expose all its peripherals to OPAE. It can be connected to the peripheral fabric, but can choose to expose only a subset of its capability to OPAE.
OFS uses a defined set of CSRs to expose the functionality of the FPGA to the host software. These registers are described in Open FPGA Stack Reference Manual - MMIO Regions section.
If you make changes to the FIM that affect the software operation, then OFS provides a mechanism to communicate that information to the proper software driver. The Device Feature Header (DFH) structure provides a mechanism to maintain compatibility with OPAE software. Please see FPGA Device Feature List (DFL) Framework Overview for an excellent description of DFL operation from the driver perspective.
When you are planning your address space for your FIM updates, please be aware that the OFS FIM targeting Intel\u00ae FPGA PAC D5005, 256KB of MMIO region is allocated for external FME features and 128kB of MMIO region is allocated for external port features. Each external feature must implement a feature DFH, and the DFH needs to be placed at 4KB boundary. The last feature in the external feature list must have the EOL bit in its DFH set to 1 to mark the end of external feature list. Since the FPGA address space is limited, consider using an indirect addressing scheme to conserve address space.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#4-fim-development-flow","title":"4. FIM Development Flow","text":"OFS provides a framework of FPGA synthesizable code, simulation environment, and synthesis/simulation scripts. FIM designers can take the provided code and scripts and modify existing code or add new code to meet their specific product requirements.
FIM development for a new acceleration card consists of the following steps:
- Installation of OFS and familiarization with scripts and source code
- Development of high-level block diagram with your specific functionality
- Determination of requirements and key performance metrics
- Selection of IP cores
- Selection of FPGA device
- Software memory map
- Selection and implementation of FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Transceivers
- External memories
- FPGA programming methodology
- Device physical implementation
- FPGA device pin assignment
- Inclusion of logic lock regions
- Creation of timing constraints
- Create Quartus FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- FIM design implementation
- RTL coding
- IP instantiation
- Development of test AFU to validate FIM
- Unit and device level simulation
- Timing constraints and build scripts
- Timing closure and build validation
- Creation of FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Hardware/software integration, validation and debugging
- High volume production preparation
The FIM developer works closely with the hardware design of the target board, software development and system validation.
Understanding how the AFU developer utilizes the FIM is important for FIM development success. Please read Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs for a detailed description of AFU development.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#41-installation-of-ofs","title":"4.1. Installation of OFS","text":"In this section you set up a development machine for compiling the OFS FIM. These steps are separate from the setup for a deployment machine where the FPGA acceleration card is installed. Typically, FPGA development and deployment work is performed on separate machines, however, both development and deployment can be performed on the same server if desired. Please see Software Installation Guide: OFS for PCIe Attach FPGAs for instructions on installing software for deployment of your FPGA FIM, AFU and software application on a server.
Building the OFS FIM requires the development machine to have at least 64 GB of RAM.
The following is a summary of the steps to set up for FIM development:
- Install Quartus Prime Pro 23.4 Linux and setup environment
- Clone the github
ofs-d5005 repository - Test installation by building the provided FIM
Quartus Prime Pro version 23.4 is the currently verified version of Quartus used for building the FIM and AFU images for this release. Porting to newer versions of Quartus may be performed by developers. Download Quartus Prime Pro Linux version 23.4 from Quartus\u00ae Prime Pro Edition Linux.
Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.6 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are No patches for this release.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\n
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=$PATH:<Quartus install directory>/quartus/bin\n
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=$PATH:/home/intelFPGA_pro/23.4/quartus/bin\n
Verify, Quartus is discoverable by opening a new shell:
which quartus\n## Output\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\n
Note, for some Linux distributions such as RHEL 8.6, Quartus requires installation of the following libraries: sudo dnf install libnsl\nsudo dnf install ncurses-compat-libs\nsudo ln -s /usr/bin/python3 /usr/bin/python\n
You will need to obtain a license for Quartus Prime Pro version 23.4 to compile the design. Additionally, OFS for Stratix\u00ae 10 FPGA requires a license for the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core. This license is required to generate a programming file using the provided OFS source code. The Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core license patch installer is provided in the ofs-d5005 git repository in the /license directory. After cloning the OFS release in step 4 below, you can install this IP license.
- Install git.
sudo dnf install git\n
- Retrieve OFS repositories:
\u200b The OFS FIM source code is included in the GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files.
- Navigate to location for storage of OFS source, create the top-level source directory and clone OFS repositories.
mkdir OFS_fim_build_root\ncd OFS_fim_build_root\nexport OFS_BUILD_ROOT=$PWD\ngit clone --recurse-submodules https://github.com/OFS/ofs-d5005.git\ncd ofs-d5005\ngit checkout tags/ofs-2024.1-1\n
Verify proper tag is selected: git describe --tags\nofs-2024.1-1\n
2. Install the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP license by running provided license installer. cd license\nchmod +x quartus-0.0-0.01Intel OFS-linux.run\nsudo ./quartus-0.0-0.01Intel OFS-linux.run\n
- Verify patch installed
quartus_sh --version\n##Output\nQuartus Prime Shell\nVersion 23.4 Pro Edition\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#42-compiling-ofs-fim","title":"4.2. Compiling OFS FIM","text":"OFS provides a build script with the following FPGA image creation options:
- Flat compile which combines the FIM and AFU into one FPGA image that is loaded into the entire FPGA device
- A PR compile which creates a FPGA image consisting of the FIM that is loaded into the static region of the FPGA and a default AFU that is loaded into dynamic region. Additional AFUs maybe loaded into the dynamic region using partial reconfiguration.
The build scripts included with OFS are verified to run in a bash shell. Other shells have not been tested. Each build script step will take several hours to completes, Please note, building directly in Quartus GUI is not supported - you must build with the provided scripts.
The following sections describe how to set up the environment and build the provided FIM and AFU. Follow these steps as a tutorial to learn the build flow. You will use this environment and build scripts for the creation of your specialized FIM.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#421-setting-up-required-environment-variables","title":"4.2.1. Setting Up Required Environment Variables","text":"Set required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks so creating a simple script to set these variables saves time.
cd $OFS_BUILD_ROOT/ofs-d5005\nexport OFS_ROOTDIR=$PWD\n## Note, OFS_ROOTDIR is the directory where you cloned the repo, e.g. /home/MyProject/ofs-d5005 *\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\n## Note, QUARTUS_ROOTDIR is your Quartus installation directory, e.g. $QUARTUS_ROOTDIR/bin contains Quartus executuable*\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#422-compiling","title":"4.2.2. Compiling","text":"The usage of the compile build script is shown below:
ofs-common/scripts/common/syn/build_top.sh [-p] target_configuration work_dir Usage: ofs-common/scripts/common/syn/build_top.sh [-k] [-p] <build target> [<work dir name>]\nBuild a FIM instance specified by <build target>. The target names an FPGA architecture, board and configuration.\n\nThe FIM is built in <work dir name>. If not specified, the target is ${OFS_ROOTDIR}/work.\n\nThe -k option preserves and rebuilds within an existing work tree instead of overwriting it.\n\nWhen -p is set, if the FIM is able then a partial reconfiguration template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable\n and uses only relative paths. See ofs-common/scripts/common/syn/generate_pr_release.sh for details.\n\nThe -e option runs only Quartus analysis and elaboration.\n\n* target_configuration - Specifies the project For example: d5005\n\n* work_dir - Work Directory for this build in the form a directory name. It is created in the <local repo directory>/ofs-d5005/<work_dir> - NOTE: The directory name must start with \"work\". If the work directory exists, then the script stops and asks if you want to overwrite the directory.\n - e.g.\n - ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005\n\nwork directory as a name will be created in <local repo directory>/ofs-d5005/work_d5005\n\nThe obmission of <work_dir> results in a default work directory (<local repo directory>/ofs-d5005/work)\n- compile reports and artifacts (.rpt, .sof, etc) are stored in <work_dir>/syn/syn_top/output_files\n\n- There is a log file created in ofs-d5005 directory. - [-p] Optional switch for creation of a relocatable PR build tree supporting the creation of a PR-able AFU workload. The \"-p\" switch invokes generate_pr_release.sh at the end of the FIM build and writes the PR build tree to the top of the work directory. More information on this option is provided below.
In the next example, you will build the provided example design using a flat, non-PR build flow. Build the provided base example design:
```bash cd $OFS_BUILD_ROOT/ofs-d5005
ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005 ```
```bash ... build takes ~5 hours to complete
Compile work directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top Compile artifact directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top/output_files
*** OFS_PROJECT: d5005 *** Q_PROJECT: d5005 *** Q_REVISION: d5005 *** SEED: 03 *** Build Complete *** Timing Passed!
The build script copies the ipss, sim, src and syn directories to the specified work directory and then these copied files are used in the Quartus compilation process. Do not edit the files in the work directory, these files are copies of source files.\n\nSome of the key files are described below:\n\n<work_dir>/syn/syn_top == \n```bash\n\u251c\u2500\u2500 syn_top // D5005 Quartus build area with Quartus files used this build\n\u2502\u00a0\u00a0\u251c\u2500\u2500 d5005.ipregen.rpt // IP regeneration report states the output of IP upgrade\n\u2502\u00a0\u00a0\u251c\u2500\u2500 d5005.qpf // Quartus Project File (qpf) mentions about Quartus version and project revision\n\u2502 \u251c\u2500\u2500 d5005.qsf // Quartus Settings File (qsf) lists current project settings and entity level assignments\n\u2502\u00a0\u00a0\u251c\u2500\u2500 d5005.stp // Signal Tap file included in the d5005.qsf. This file can be modified as required if you need to add an Signal Tap instance\n\u2502\u00a0\u00a0\u251c\u2500\u2500 fme_id.mif // the fme id hex value is stored in a mif file format\n\u2502 \u251c\u2500\u2500 OFS_pr_afu.json // PR JSON file\n\u2502\u00a0\u00a0\u251c\u2500\u2500 OFS_pr_afu.qsf // PR AFU qsf file\n\u2502\u00a0\u00a0\u251c\u2500\u2500 OFS_pr_afu_sources.tcl // AFU source file list\n\u2502\u00a0\u00a0\u251c\u2500\u2500 ip_upgrade_port_diff_reports // IP upgrade report files for reference\n
/syn/syn_top/output_files == Directory with build reports and FPGA programming files. The programming files consist of the Quartus generated d5005.sof and d5005.pof. The D5005 board hardware provides a 2 Gb flash device to store the FPGA programming files and a MAX10 BMC that reads this flash and programs the D5005 Stratix\u00ae 10 FPGA FPGA. The syn/build_top.sh script runs script file syn/syn_top/build_flash/build_flash.s which takes the Quartus generated d5005.sof and creates binary files in the proper format to be loaded into the 2 Gb flash device. You can also run build_flash.sh by yourself if needed. The build_flash script runs PACSign (if installed) to create an unsigned FPGA programming file that can be stored in the D5005 FPGA flash. Please note, if the D5005 has the root entry hash key loaded, then PACsign must be run with d5005_page1.bin as the input with the proper key to create an authenticated FPGA binary file. Please see Security User Guide: Open FPGA Stack for details on the security aspects of Open FPGA Stack.
The following table provides further detail on the generated bin files.
File Description d5005.sof This is the Quartus generated programming file created by Quartus synthesis and place and route. This file can be used to programming the FPGA using a JTAG programmer. This file is used as the source file for the binary files used to program the FPGA flash. d5005.bin This is an intermediate raw binary image of the FPGA d5005_page1.bin This is the binary file created from input file, d5005.sof. This file is used as the input file to the PACSign utility to generate d5005_page1_unsigned.bin binary image file. d5005_page1_unsigned.bin This is the unsigned PACSign output which can be programmed into the FPGA flash of an unsigned D5005 usign the OPAE SDK utility fpgasupdate mfg_d5005_reversed.bin A special programming file for a third party programming device used in board manufacturing. This file is typically not used. build/output_files/timing_report == Directory containing clocks report, failing paths and passing margin reports
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#423-relocatable-pr-directory-tree","title":"4.2.3. Relocatable PR Directory Tree","text":"If you are developing a FIM to be used by another team developing the AFU workload, scripts are provided that create a relocatable PR directory tree. ODM and board developers will make use of this capability to enable a broad set of AFUs to be loaded on a board using PR. The relocatable PR directory contains the Quartus *.qdb file that goes the FIM.
The creation of the relocatable PR directory tree requires a clone of the Basic Building Blocks (BBB) repository. The OFS_PLATFORM_AFU_BBB environment variable must point to the repository, for example.
cd $OFS_BUILD_ROOT\ngit clone https://github.com/OPAE/ofs-platform-afu-bbb\ncd ofs-platform-afu-bbb\nexport OFS_PLATFORM_AFU_BBB=$PWD\ncd $OFS_ROOTDIR\n
You can create this relocatable PR directory tree by either:
- Build FIM and AFU using /syn/build_top.sh followed by running./ofs-common/scripts/common/syn/generate_pr_release.sh
- Build FIM and AFU using /syn/build_top.sh with optional -p switch included
The generate_pr_release.sh has the following command structure:
./ofs-common/scripts/common/syn/generate_pr_release.sh -t <path to generated release tree> *Board Build Target* <work dir from build_top.sh>\n\nWhere:\n\n-t <path to generated release tree> = location for your relocatable PR directory tree\n*Board Build Target* is the name of the board target/FIM e.g. d5005\n<work dir from build_top.sh>
Here is an example of running the generate_pr_release.sh script: ofs-common/scripts/common/syn/generate_pr_release.sh -t work_d5005/build_tree d5005 work_d5005\n
**********************************\n********* ENV SETUP **************\n\nFIM Project:\n OFS_PROJECT = d5005\n OFS_FIM = .\n OFS_BOARD = .\n Q_PROJECT = d5005\n Q_REVISION = d5005\n Fitter SEED = 03\nFME id\n BITSTREAM_ID = 04010002c7cab852\n BITSTREAM_MD = 0000000002204283\n...\n...\n
The resulting relocatable build tree has the following structure: .\n\u251c\u2500\u2500 bin\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 afu_synth\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build_env_config\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 run.sh -> afu_synth\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 update_pim\n\u251c\u2500\u2500 hw\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 blue_bits\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 d5005_page1_unsigned.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 d5005.sof -> ../lib/build/syn/syn_top/output_files/d5005.sof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 lib\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-ifc-id.txt\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-platform-class.txt\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 platform\n
This build tree can be moved to a different location and used for AFU development of a PR capable AFU to be used with this board.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#424-unit-level-simulation","title":"4.2.4. Unit Level Simulation","text":"Unit level simulation of key components is provided. These simulations provide verification of the following areas:
- HSSI
- PCIe
- External Memory
- FIM management
These simulations use the Synopsys VCS simulator. Each simulation contains a readme file explaining how to run the simulation. Refer to UVM Simulation User Guide: OFS for Stratix\u00ae 10 PCIe Attach for details of simulation examples. Your simulation shell requires Python, Quartus, and VCS to run. To run a simulation of the dfh_walker that simulates host access to the internal DFH registers, perform the following steps:
Before running unit simulation, you must set environment variables as described below:
cd $OFS_BUILD_ROOT/ofs-d5005\nexport OFS_ROOTDIR=$PWD\n## *Note, OFS_ROOTDIR is the directory where you cloned the repo, e.g. /home/MyProject/ofs-d5005 *\nexport WORKDIR=$OFS_ROOTDIR\nexport VERDIR=$OFS_ROOTDIR/verification\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\n## *Note, QUARTUS_ROOTDIR is your Quartus installation directory, e.g. $QUARTUS_ROOTDIR/bin contains Quartus executuable*\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\n
To compile all IPs:
To Generate Simulation Files & compile all IPs, run the following command:
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\nsh gen_sim_files.sh d5005\n
The RTL file list for unit_test is located here: $OFS_ROOTDIR/sim/scripts/rtl_comb.f The IPs are generated here:
$OFS_ROOTDIR/sim/scripts/qip_gen\n
The IP simulation filelist is generated here: $OFS_ROOTDIR/sim/scripts/ip_flist.f\n
Once the IPs are generated, they can be used for any unit test. To run the simulation, run the following command:
cd $OFS_ROOTDIR/sim/unit_test/<Unit Test Name>/scripts\nsh run_sim.sh VCS=1\n
Simulation files are located in the sim/unit_test//sim directory. To view simulation waveform:
cd $OFS_ROOTDIR/sim/unit_test/<test_name>/script/sim/unit_test/<test_name>/scripts/sim_vcs\ndve -full64 -vpd vcdplus.vpd &\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#4241-dfh-walking-unit-simulation-output","title":"4.2.4.1. DFH Walking Unit Simulation Output","text":"********************************************\n Running TEST(0) : test_fme_dfh_walking\n********************************************\nREAD64: address=0x00000000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010000000\n\nFME_DFH\n Address (0x0)\nDFH value (0x4000000010000000)\nREAD64: address=0x00001000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000020000001\n\nTHERM_MNGM_DFH\n Address (0x1000)\nDFH value (0x3000000020000001)\nREAD64: address=0x00003000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000007\n\nGLBL_PERF_DFH\n Address (0x3000)\nDFH value (0x3000000010000007)\nREAD64: address=0x00004000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000c0001004\n\nGLBL_ERROR_DFH\n Address (0x4000)\nDFH value (0x30000000c0001004)\nREAD64: address=0x00010000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000000e\n\nSPI_DFH\n Address (0x10000)\nDFH value (0x300000010000000e)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000100000020\n\nPCIE_DFH\n Address (0x20000)\nDFH value (0x3000000100000020)\nREAD64: address=0x00030000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000100f\n\nHSSI_DFH\n Address (0x30000)\nDFH value (0x300000010000100f)\nREAD64: address=0x00040000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000500000009\n\nEMIF_DFH\n Address (0x40000)\nDFH value (0x3000000500000009)\nREAD64: address=0x00090000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010001005\n\nFME_PR_DFH\n Address (0x90000)\nDFH value (0x3000000010001005)\nREAD64: address=0x00091000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010001001\n\nPORT_DFH\n Address (0x91000)\nDFH value (0x4000000010001001)\nREAD64: address=0x00092000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000014\n\nUSER_CLOCK_DFH\n Address (0x92000)\nDFH value (0x3000000010000014)\nREAD64: address=0x00093000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000d0002013\n\nPORT_STP_DFH\n Address (0x93000)\nDFH value (0x30000000d0002013)\nREAD64: address=0x000a0000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000010000002010\n\nAFU_INTF_DFH\n Address (0xa0000)\nDFH value (0x3000010000002010)\nMMIO error count matches: x\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_fme_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n
The simulation transcript is displayed while the simulation runs. The transcript is saved to the file transcript.out for review after the simulation completes. The simulation waveform database is saved as vcdplus.vpd for post simulation review. You are encouraged to run the additional simulation examples to learn about each key area of the OFS shell.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#43-compiling-the-ofs-fim-using-eval-script","title":"4.3. Compiling the OFS FIM using Eval Script","text":"The Evaluation Script provides resources to setup and report D5005 development environment. You can use the evaluation script to compile and simulate the FIM. Refer to README_ofs_d5005_eval.txt for details of using the evaluation script.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#44-debugging","title":"4.4. Debugging","text":"For debugging issues within the FIM, Signal Tap can be used to gain internal visibility into your design. This section describes the process of adding a Signal Tap instance to your FIM design
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#441-signal-tap-prerequisites","title":"4.4.1. Signal Tap Prerequisites","text":"To use Signal Tap with OFS, you will need the following:
- Understanding of Signal Tap fundamentals - please review Quartus Prime Pro Edition User Guide: Debug Tools. section 2. Design Debugging with the Signal Tap Logic Analyzer.
- The Intel\u00ae FPGA PAC D5005 has a built in FPGA Download Cable II allowing JTAG access to the S10 FPGA. You can access the D5005 built in FPGA Download Cable II by connecting your server to the Micro USB connector as shown below:
- If you are using a custom board without a built-in FPGA Download Cable then an external FPGA Download Cable II (see Download Cables for more information) can be used for Signal Tap access. The custom board must have JTAG access to the target FPGA for the FPGA Download Cable II.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#442-adding-signal-tap","title":"4.4.2. Adding Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware, the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized. These steps assume the use of the Intel\u00ae FPGA PAC D5005.
- Perform a full compile using the script build_top.sh.
- Once the compile completes open the Quartus GUI using the FIM project. The Quartus project is named d5005 and is located in the work directory syn/syn_top/d5005.qpf. Once the project is loaded, go to Tools > Signal Tap Logic Analyzer to bring up the Signal Tap GUI.
- Accept the \"Default\" selection and click \"Create\".
- This brings up Signal Tap Logic Analyzer window as shown below:
- Set up clock for STP instance. In this example, the EMIF CSR module is being instrumented. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the specific block of interest and open the design instance for review. For example, see snip below using Project Navigator to open emif_csr block:
- After reviewing code, assign clock for sampling Signal Tap instrumented signals of interest. Note, the clock selected and the signals you want to view should be the same for best trace fidelity. Different clocks can be used however there maybe issues with trace inaccuracy due sampling time differences. In the middle right of the Signal Tap window under Signal Configuration, Clock: select \"\u2026\" as shown below:
- After reviewing code, assign clock for sampling Signal Tap instrumented signals of interest. In the middle right of the Signal Tap window under Signal Configuration, Clock: select \"\u2026\" as shown below: This brings up the Node Finder tool. Input \"emif_csr\" into Named and select \"Search\". This brings up all nodes from the pre-synthesis view. Expand, \"mem\" and \"emif_csr\" and scroll through this list to become familiar with nodes, and then select csr_if.clk and click \">\" to select this clock as shown below and click \"OK\":
- Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
- In the Signal Tap GUI add nodes to be instrumented by double clicking on \"Double-click to add nodes\".
- This brings up the Node Finder. Add signals to be traced by the Signal Tap instance. Click \"Insert\" to add the signals.
- To provide a unique name for your Signal Tap instance, select \"auto signaltap_0\", right click and select rename instance and provide a descriptive name for your instance.
- Save the newly created Signal Tap file and click \"Yes\" to add the new Signal Tap file to the project.
- Compile the project with the Signal Tap file added to the project.
- Once the compile successfully completes with proper timing, you can load the generated d5005.sof using the FPGA Downloader cable.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#443-signal-tap-trace-acquisition","title":"4.4.3. Signal Tap trace acquisition","text":"To acquire signals using SignalTap, first load the Signal Tap instrumented SOF file into your target board, open the STP file in the Signal Tap GUI and start the signal acquisition.
Avoid system hang during programming the sof file, mask AER regsiter using below steps
Find Root complex - End Point mapping using the below command
lspci -vt\n
+-[0000:3a]-+-00.0-[3b-3c]----00.0 Intel Corporation Device bcce\n | +-05.0 Intel Corporation Sky Lake-E VT-d\n | +-05.2 Intel Corporation Sky Lake-E RAS Configuration Registers\n | +-05.4 Intel Corporation Sky Lake-E IOxAPIC Configuration Registers\n | +-08.0 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-09.0 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.0 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.1 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.2 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.3 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.4 Intel Corporation Sky Lake-E Integrated Memory Controller\n | +-0a.5 Intel Corporation Sky Lake-E LM Channel 1\n
Use the bus information from the lspci logs to mask the AER (Advanced Error Reporting) register
sudo su\n\nsetpci -s 0000:3b:00.0 ECAP_AER+0x08.L=0xFFFFFFFF setpci -s 0000:3b:00.0 ECAP_AER+0x14.L=0xFFFFFFFF\nsetpci -s 0000:3a:00.0 ECAP_AER+0x08.L=0xFFFFFFFF\nsetpci -s 0000:3a:00.0 ECAP_AER+0x14.L=0xFFFFFFFF\necho \"1\" > /sys/bus/pci/devices/0000:3b:00.0/remove\n\nexit\n
- The SOF file is located in the work directory work_d5005/syn/syn_top/output_files/d5005.sof. If the target FPGA is on a different server, then transfer d5005.sof and STP files to the server with the target FPGA. Load the SOF using the Intel\u00ae FPGA PAC D5005 built-in FPGA Download Cable II.
sudo su\necho \"1\" > /sys/bus/pci/rescan\n
- Make sure D5005 is present by checking expected bitstream ID using command:
sudo fpgainfo fme\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.13 Board Management Controller, MAX10 Build version: 2.0.8 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x40100022C164DB1\nBitstream Version : 4.0.1\nPr Interface Id : 210a4631-18bb-57d1-879f-2c3d59b26e37\nBoot Page : user\n
- Once the SOF file is loaded, start the Quartus Signal Tap GUI.
quartus_stpw\n
The Signal Tap GUI comes up. - In the Signal Tap GUI,
Hardware: selection box select the cable \"Stratix10 Darby Creek [ JTAG cable number ]\" as shown below:
- In
File open your STP file. Your STP file settings will load. If not already set, you can create the trigger conditions, then start analysis with F5.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#5-fim-modification-example","title":"5. FIM Modification Example","text":"An example of FIM modification is provided in this section. This example can be used in your specific application as a starting point. This example shows the basic flow and listing of files that are to be changed.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#51-hello-fim-example","title":"5.1. Hello FIM example","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example adds a simple DFH register set to the FIM. You can use this example as the basis for adding a new feature to your FIM.
The steps to add this simple DFH register are described below.
- Review current design documentation: OFS Tech Ref MMIO Regions
- Understand FME and Port regions, DFH walking, DFH register structure
- Run unit level simulations and review output: i. sim/unit_test/dfh_walker
- Note DFH link list order, see DFH Walker Unit Level Simulation Output
- Make code changes to top level FIM file to instantiate new DFH register
- The DFH registers follow a link list. This example inserts the hello_fim DFH register after the EMIF DFH register, so the emif_csr.sv parameters are updated to insert the hello_fim DFH register as next register in the link list.
- Create the new hello_fim SystemVerilog files.
- Update and run the dfh_walker unit simulation files
- Update synthesis files to include the new hello_fim source files
- Build and test the new FIM
The following sections describe changes to add the hello_fim DFH example to the provided FPGA design.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#511-srctopiofs_topsv","title":"5.1.1. src/top/iofs_top.sv","text":" - Edit top level design module: src/top/iofs_top.sv
- Instantiate new hello_fim module in OFS_top.sv at line 294
//*******************************\n// FME\n//*******************************\nfme_top fme_top(\n.clk (clk_1x ),\n.rst_n (rst_n_d_1x ),\n.pwr_good_n (ninit_done ),\n.i_pcie_error ('0 ),\n.axi_lite_m_if (bpf_fme_mst_if ),\n.axi_lite_s_if (bpf_fme_slv_if )\n);\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top hello_fim_top (\n.clk (clk_1x),\n.rst_n (rst_n_d_1x),\n.csr_lite_if (bpf_rsv_5_slv_if)\n);\n`endif\n//*******************************\n// AFU\n//*******************************\n
You will connect the Hello_FIM DFH register to the existing BPF reserved link 5. The provided OFS reference design includes 3 reserved BPF interfaces available for custom usage such as new OPAE controlled modules. The address map of BPF is shown below:
Address Size (Byte) Feature Master 0x00000 \u2013 0x0FFFF 64K FME (FME, Error, etc) Yes 0x10000 \u2013 0x1FFFF 64K PMCI Proxy (SPI Controller) Yes 0x20000 \u2013 0x2FFFF 64K PCIe CSR 0x30000 \u2013 0x3FFFF 64K HSSI CSR 0x40000 \u2013 0x4FFFF 64K EMIF CSR 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x6FFFF 64K Reserved 0x70000 \u2013 0x7FFFF 64K Reserved The BPF reserved link 5 is connected to a dummy connection to prevent this link from being optimized out the design during synthesis. You will add a compiler `define that will cause this dummy connection to be removed when the variable INCLUDE_HELLO_FIM is defined by editing line 575 if iofs_top.sv as shown below:
// Reserved address response\n`ifndef INCLUDE_HELLO_FIM\nbpf_dummy_slv\nbpf_rsv_5_slv (\n.clk (clk_1x),\n.dummy_slv_if (bpf_rsv_5_slv_if)\n);\n`endif\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#512-ipssmememif_csrsv","title":"5.1.2. ipss/mem/emif_csr.sv","text":"The Hello_FIM DFH is inserted in the DFH link list after the EMIF CSR DFH and before the FME_PR DFH. The file ipss/d5005/emif/emif_csr.sv contains a parameter defining the next address for the next DFH in in the link list chain. You will change the next address offset to be 0x10000 so the reveserved BPF AXI lite link connected to the Hello_FIM DFH register is next in the DFH link list.
module emif_csr #(\nparameter NUM_LOCAL_MEM_BANKS = 1,\nparameter END_OF_LIST = 1'b0,\n`ifndef INCLUDE_HELLO_FIM\nparameter NEXT_DFH_OFFSET = 24'h05_0000\n`else\nparameter NEXT_DFH_OFFSET = 24'h01_0000//New for Hello_FIM, next offset now at 0x50000\n`endif\n)\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#513-srchello_fimhello_fim_topsv","title":"5.1.3. src/hello_fim/hello_fim_top.sv","text":"Create hello_fim_top.sv, and store it in src/hello_fim directory. The main purpose of this RTL is to convert AXI4-Lite interface to a simple interface to interface with the registers in hello_fim_com.sv.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0xfff which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2021 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : Nov 2021\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'hfff,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h04_0000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic rst_n,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (rst_n),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.rst_n (rst_n ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#514-srchello_fimhello_fim_comsv","title":"5.1.4. src/hello_fim/hello_fim_com.sv","text":"Create hello_fim_com.sv, and store it in src/hello_fim directory. This is the simple RTL to implement the Hello FIM registers. You may use this set of registers as the basis for your custom implementation.
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2021 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : Nov 2021\n// Module Name : hello_fim_com.sv\n// Project : IOFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'hfff,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput rst_n,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nreg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(posedge clk) if (!rst_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(posedge clk)\nif (!rst_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( posedge clk)\nif (!rst_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#515-unit-level-simulations","title":"5.1.5. Unit Level Simulations","text":"To run a unit level simulation test for the updated RTL files, make modifications to your cloned /my_ofs_project/ofs-d5005/sim/d5005/unit_test/dfh_walker files. The following simulation files are updated to test the new hello_fim.
-
Edit sim/unit_test/dfh_walker/testbench/test_csr_defs.sv
- Update enum line 38
typedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nSPI_DFH_IDX,\nPCIE_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX,//New for HELLO_FIM\nFME_PR_DFH_IDX,\nPORT_DFH_IDX,\nUSER_CLOCK_DFH_IDX,\nPORT_STP_DFH_IDX,\nAFU_INTF_DFH_IDX,\nMAX_FME_DFH_IDX\n} t_fme_dfh_idx;\n
1. Edit function dfh_name line 78
function automatic dfh_name[MAX_FME_DFH_IDX-1:0] get_fme_dfh_names();\ndfh_name[MAX_FME_DFH_IDX-1:0] fme_dfh_names;\nfme_dfh_names[FME_DFH_IDX] = \"FME_DFH\";\nfme_dfh_names[THERM_MNGM_DFH_IDX] = \"THERM_MNGM_DFH\";\nfme_dfh_names[GLBL_PERF_DFH_IDX] = \"GLBL_PERF_DFH\";\nfme_dfh_names[GLBL_ERROR_DFH_IDX] = \"GLBL_ERROR_DFH\";\nfme_dfh_names[SPI_DFH_IDX] = \"SPI_DFH\";\nfme_dfh_names[PCIE_DFH_IDX] = \"PCIE_DFH\";\nfme_dfh_names[HSSI_DFH_IDX] = \"HSSI_DFH\";\nfme_dfh_names[EMIF_DFH_IDX] = \"EMIF_DFH\";\nfme_dfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\";//New for HELLO_FIM\nfme_dfh_names[FME_PR_DFH_IDX] = \"FME_PR_DFH\";\nfme_dfh_names[PORT_DFH_IDX] = \"PORT_DFH\";\nfme_dfh_names[USER_CLOCK_DFH_IDX] = \"USER_CLOCK_DFH\";\nfme_dfh_names[PORT_STP_DFH_IDX] = \"PORT_STP_DFH\";\nfme_dfh_names[AFU_INTF_DFH_IDX] = \"AFU_INTF_DFH\";\nreturn fme_dfh_names;\nendfunction\n
1. Update get_fme_dfh_values
function automatic [MAX_FME_DFH_IDX-1:0][63:0] get_fme_dfh_values();\nlogic[MAX_FME_DFH_IDX-1:0][63:0] fme_dfh_values;\nfme_dfh_values[FME_DFH_IDX] = 64'h4000_0000_1000_0000;\nfme_dfh_values[THERM_MNGM_DFH_IDX] = 64'h3_00000_002000_0001;\nfme_dfh_values[GLBL_PERF_DFH_IDX] = 64'h3_00000_001000_0007;\nfme_dfh_values[GLBL_ERROR_DFH_IDX] = 64'h3_00000_00C000_1004; fme_dfh_values[SPI_DFH_IDX] = 64'h3_00000_010000_000e; fme_dfh_values[PCIE_DFH_IDX] = 64'h3_00000_010000_0020; fme_dfh_values[HSSI_DFH_IDX] = 64'h3_00000_010000_100f; fme_dfh_values[EMIF_DFH_IDX] = 64'h3_00000_010000_0009; //Update to link to Hello_FIM \nfme_dfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_040000_0FFF; //New for Hello_FIM\nfme_dfh_values[FME_PR_DFH_IDX] = 64'h3_00000_001000_1005; fme_dfh_values[PORT_DFH_IDX] = 64'h4000_0000_1000_1001;\nfme_dfh_values[USER_CLOCK_DFH_IDX] = 64'h3_00000_001000_0014;\nfme_dfh_values[PORT_STP_DFH_IDX] = 64'h3_00000_00D000_2013;\nfme_dfh_values[AFU_INTF_DFH_IDX] = 64'h3_00001_000000_2010; return fme_dfh_values;\nendfunction\n
- Update verification/scripts/Makefile_VCS.mk to set macro for INCLUDE_HELLO_FIM starting at line 56 to add +define+INCLUDE_HELLO_FIM
VLOG_OPT += +define+SIM_MODE +define+VCS_S10 +define+RP_MAX_TAGS=64 +define+INCLUDE_DDR4 +define+INCLUDE_SPI_BRIDGE +define+INCLUDE_USER_CLOCK +define+INCLUDE_HSSI +define+SIM_USE_PCIE_DUMMY_CSR +define+INCLUDE_HELLO_FIM\n
- Update sim/scripts/rtl_comb.f to add the path to your new hello_fim_top and hello_top_com SystemVerilog files. The update is shown below as the new line - 329 below:
$WORKDIR/src/hello_fim/hello_fim_com.sv\n$WORKDIR/src/hello_fim/hello_fim_top.sv\n
After making these changes, run the unit level simulation using sim/unit_test/dfh_walker test. Before running, ensure your shell has the environment variables set properly as defined in Setting Up Required Environment Variables.
cd verification/scripts\ngmake -f Makefile_VCS.mk cmplib\ngmake -f Makefile_VCS.mk build run [DUMP=1]\n
Expected output:
********************************************\n Running TEST(0) : test_fme_dfh_walking\n********************************************\nREAD64: address=0x00000000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010000000\n\nFME_DFH\n Address (0x0)\nDFH value (0x4000000010000000)\nREAD64: address=0x00001000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000020000001\n\nTHERM_MNGM_DFH\n Address (0x1000)\nDFH value (0x3000000020000001)\nREAD64: address=0x00003000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000007\n\nGLBL_PERF_DFH\n Address (0x3000)\nDFH value (0x3000000010000007)\nREAD64: address=0x00004000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000c0001004\n\nGLBL_ERROR_DFH\n Address (0x4000)\nDFH value (0x30000000c0001004)\nREAD64: address=0x00010000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000000e\n\nSPI_DFH\n Address (0x10000)\nDFH value (0x300000010000000e)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000100000020\n\nPCIE_DFH\n Address (0x20000)\nDFH value (0x3000000100000020)\nREAD64: address=0x00030000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x300000010000100f\n\nHSSI_DFH\n Address (0x30000)\nDFH value (0x300000010000100f)\nREAD64: address=0x00040000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000100000009\n\nEMIF_DFH\n Address (0x40000)\nDFH value (0x3000000100000009)\nREAD64: address=0x00050000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000400000fff\n\nHELLO_FIM_DFH\n Address (0x50000)\nDFH value (0x3000000400000fff)\nREAD64: address=0x00090000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010001005\n\nFME_PR_DFH\n Address (0x90000)\nDFH value (0x3000000010001005)\nREAD64: address=0x00091000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x4000000010001001\n\nPORT_DFH\n Address (0x91000)\nDFH value (0x4000000010001001)\nREAD64: address=0x00092000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000000010000014\n\nUSER_CLOCK_DFH\n Address (0x92000)\nDFH value (0x3000000010000014)\nREAD64: address=0x00093000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x30000000d0002013\n\nPORT_STP_DFH\n Address (0x93000)\nDFH value (0x30000000d0002013)\nREAD64: address=0x000a0000 bar=0 vf_active=0 pfn=0 vfn=0\nREADDATA: 0x3000010000002010\n\nAFU_INTF_DFH\n Address (0xa0000)\nDFH value (0x3000010000002010)\nMMIO error count matches: x\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_fme_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#516-synsyn_topd5005qsf","title":"5.1.6. syn/syn_top/d5005.qsf","text":" -
Edit syn/syn_top/d5005.qsf
-
Add new macro \"INCLUDE_HELLO_FIM\" line 107
set_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\"\n
-
Add new line 211 to source TCL script with new hello_fim files
set_global_assignment -name SOURCE_TCL_SCRIPT_FILE ../../../syn/setup/hello_fim_design_files.tcl\n
Create \"hello_fim_design_files.tcl\" file and store in the syn/setup directory. This tcl file is called from d5005.qsf.
# Copyright 2021 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE src/hello_fim/hello_fim_top.sv\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#517-synsetuphello_fim_design_filestcl","title":"5.1.7. syn/setup/hello_fim_design_files.tcl","text":""},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#518-build-hello_fim-example","title":"5.1.8. Build hello_fim example","text":"With the preceding changes complete, build the new hello_fim example using the following steps:
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh d5005 work_d5005_hello_fim\n
Verify the design successfully compiled and timing closure is achieved by checking work_d5005_hello_fim/syn/syn_top/output_files/timing_report/clocks.sta.fail.summary - this file should be empty. If there are timing failures, then this file will list the failing clock domain(s).
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#519-test-the-hello_fim-on-a-d5005","title":"5.1.9. Test the hello_fim on a D5005","text":"Load the built FPGA binary file using an unsigned image. The FPGA image will be in work_d5005_hello_fim/syn/syn_top/output_files/d5005_page1_unsigned.bin
Provide the file d5005_page1_unsigned.bin on the server with the Intel\u00ae FPGA PAC D5005.
sudo fpgasupdate d5005_page1_unsigned.bin <D5005 PCIe B:D.F>\nsudo rsu bmcimg <D5005 PCIe B:D.F>\n
Verify FPGA image is loaded. sudo fpgainfo fme\n## Output\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.13 Board Management Controller, MAX10 Build version: 2.0.8 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x40100022C164DB1\nBitstream Version : 4.0.1\nPr Interface Id : 7d91e0d0-4dcd-58c3-a93d-b9295e6e29b0\nBoot Page : user\n
Use the OPAE SDK tool opae.io to check default driver binding using your card under test PCIe B:D.F. The steps below will use 0000:12:00.0 as the card under test PCIe B:D.F.
sudo opae.io init -d 0000:12:00.0 $USER\n##Output\n[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: dfl-pci)\n
The dfl-pci driver is used by OPAE SDK fpgainfo commands. The next steps will bind the card under test to the vfio driver to enable access to the registers. sudo opae.io init -d 0000:12:00.0 $USER\n##Output\nopae.io 0.2.3\nUnbinding (0x8086,0xbcce) at 0000:12:00.0 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:12:00.0 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:12:00.0 is 35\nAssigning /dev/vfio/35 to $USER\n
Confirm the vfio driver is bound to the card under test. opae.io ls\n## Output\nopae.io 0.2.3\n[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: vfio-pci)\n
Run the following command to walk DFH link list. The new hello_fim register is located at offset 0x50000. opae.io walk -d 0000:12:00.0\n## Output\nopae.io 0.2.3\noffset: 0x0000, value: 0x4000000010000000\n dfh: id = 0x0, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x1000, value: 0x3000000020000001\n dfh: id = 0x1, rev = 0x0, next = 0x2000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x3000, value: 0x3000000010000007\n dfh: id = 0x7, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x4000, value: 0x30000000c0001004\n dfh: id = 0x4, rev = 0x1, next = 0xc000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x10000, value: 0x300000010000000e\n dfh: id = 0xe, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000100000020\n dfh: id = 0x20, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x30000, value: 0x300000010000100f\n dfh: id = 0xf, rev = 0x1, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x40000, value: 0x3000000100000009\n dfh: id = 0x9, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x50000, value: 0x3000000400000fff\n dfh: id = 0xfff, rev = 0x0, next = 0x40000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x90000, value: 0x3000000010001005\n dfh: id = 0x5, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x91000, value: 0x4000000010001001\n dfh: id = 0x1, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x92000, value: 0x3000000010000014\n dfh: id = 0x14, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x93000, value: 0x30000000d0002013\n dfh: id = 0x13, rev = 0x2, next = 0xd000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0xa0000, value: 0x3000010000002010\n dfh: id = 0x10, rev = 0x2, next = 0x0, eol = 0x1, reserved = 0x0, feature_type = 0x3\n
Read the default values from the hello_fim registers: $ opae.io -d 0000:12:00.0 -r 0 peek 0x50000\nopae.io 0.2.3\n0x3000000400000fff\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0x0\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50038\nopae.io 0.2.3\n0x6626070150000034\n
Write the scratchpad register at 0x50030 $ opae.io -d 0000:12:00.0 -r 0 poke 0x50038 0x123456789abcdef\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50038\nopae.io 0.2.3\n0x6626070150000034\n$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0x123456789abcdef\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0x123456789abcdef\n$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0xfedcba9876543210\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0xfedcba9876543210\n$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0x55550000aaaaffff\nopae.io 0.2.3\n$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030\nopae.io 0.2.3\n0x55550000aaaaffff\n
Release the card under test from the vfio driver to re-bind to the dfl-pci driver:
sudo opae.io release -d 0000:12:00.0\n## Output\nopae.io 0.2.3\nReleasing (0x8086,0xbcce) at 0000:12:00.0 from vfio-pci\nRebinding (0x8086,0xbcce) at 0000:12:00.0 to dfl-pci\n$ sudo opae.io ls\nopae.io 0.2.3\n[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: dfl-pci)\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#52-memory-subsystem-modification","title":"5.2. Memory Subsystem Modification","text":"OFS enables modifications on the different subsystems that encompass the FIM. To customize the Memory Subsystem follow these instructions.
-
Set up the environment variables as described in section 4.2.1. Setting Up Required Environment Variables
-
Modify the NUM_MEM_CH parameter in src/afu_top/mux/top_cfg_pkg.sv Change NUM_MEM_CH from 4 to 2 as shown in below code
//=========================================================================================================================\n// OFS Configuration Parameters \n//=========================================================================================================================\nparameter NUM_MEM_CH = 2 ,// Number of Memory/DDR Channel \nNUM_HOST = 1 ,// Number of Host/Upstream Ports \nNUM_PORT = 4 ,// Number of Functions/Downstream Ports \nDATA_WIDTH = 512 ,// Data Width of Interface \nTOTAL_BAR_SIZE = 20 ,// Total Space for APF/BPF BARs (2^N) \n//------------+-------------+-------------+-----------------+ //--------------------------------------\n// VF Active | PF # | VF # | Mux Port Map | // PF/VF Mapping Parameters \n//------------+-------------+-------------+-----------------+ //--------------------------------------\nCFG_VA = 0 , CFG_PF = 0 , CFG_VF = 0 , CFG_PID = 3 , // Configuration Register Block \nHLB_VA = 1 , HLB_PF = 0 , HLB_VF = 0 , HLB_PID = 0 , // HE Loopback Engine \nPRG_VA = 1 , PRG_PF = 0 , PRG_VF = 1 , PRG_PID = 1 , // Partial Reconfiguration Gasket \nHSI_VA = 1 , HSI_PF = 0 , HSI_VF = 2 , HSI_PID = 2 ; // HSSI interface \n
Compile a new FIM that incorporates the newly configured Memory Subsystem.
cd $OFS_BUILD_ROOT/ofs-d5005\nofs-common/scripts/common/syn/build_top.sh d5005 work_d5005_mem_2channel\n
***********************************\n***\n*** OFS_PROJECT: d5005\n*** Q_PROJECT: d5005\n*** Q_REVISION: d5005\n*** SEED: 03\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
Program d5005_page1_unsigned.bin file using below command
sudo fpgasupdate d5005_page1_unsigned.bin 3b:00.0\n
Run rsu command
sudo rsu bmcimg 3b:00.0\n
Check if binary was loaded correctly
fpgainfo fme\n## Output\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\n
Run Host Excersiser to check Memory Subsystem performance
sudo host_exerciser mem\n## Output\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5365\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.054 GB/s\n Test mem(1): PASS\n
Verify Memory controller placement in syn/syn_top/output_files/d5005.fit.place.rpt file. Open fitter place stage report in any text editor of your choice, find keyword emif in the file. You should see emif[0] & emif[1] for Memory channel 0 & 1 respectively.
|emif[0].ddr4_pr_freeze_sync| ; 0.4 (0.0) ; 0.5 (0.0) ; 0.1 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.4 (0.4) ; 0.5 (0.5) ; 0.1 (0.1) ;\n|emif[0].ddr4_softreset_sync| ; 0.5 (0.0) ; 0.7 (0.0) ; 0.2 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.5 (0.5) ; 0.7 (0.7) ; 0.2 (0.2) ;\n|emif[0].pr_frz_afu_avmm_if| ; 647.5 (647.5) ; 917.3 (917.3) ; 272.8 (272.8) ;\n|emif[1].ddr4_pr_freeze_sync| ; 0.4 (0.0) ; 0.8 (0.0) ; 0.4 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.4 (0.4) ; 0.8 (0.8) ; 0.4 (0.4) ;\n|emif[1].ddr4_softreset_sync| ; 0.4 (0.0) ; 1.0 (0.0) ; 0.6 (0.0) ;\n|resync_chains[0].synchronizer_nocut| ; 0.4 (0.4) ; 1.0 (1.0) ; 0.6 (0.6) ;\n|emif[1].pr_frz_afu_avmm_if| ; 641.1 (641.1) ; 914.0 (914.0) ; 272.9 (272.9) ;\n|p[0].pr_frz_fn2mx_a_port| ; 435.4 (0.0) ; 476.2 (0.0) ; 40.8 (0.0) ;\n|r.axis_pl_stage[0].axis_reg_inst| ; 435.4 (435.4) ; 476.2 (476.2) ; 40.8 (40.8) ;\n|p[0].pr_frz_fn2mx_b_port| ; 434.6 (0.0) ; 494.3 (0.0) ; 59.6 (0.0) ;\n
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#6-conclusion","title":"6. Conclusion","text":"Using the OFS reference design and OPAE SDK enables the rapid creation of market leading FPGA based Acceleration systems. OFS facilitates customization of the FIM area for your custom board or platforms.
"},{"location":"hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/doc_modules/Glossary/","title":"Glossary","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/","title":"FPGA Interface Manager Technical Reference Manual: Open FPGA Stack for Stratix 10\u00ae FPGA","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1-overview","title":"1 Overview","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#11-about-this-document","title":"1.1 About this Document","text":"This document describes the hardware architecture of the\u200b Open FPGA Stack (OFS) targeting the Stratix 10 FPGA. After reviewing this document you should understand the features and functions of the components that comprise the FPGA Interface Manager (FIM), also known as the \"shell.\"
Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12-introduction-to-the-open-fpga-stack","title":"1.2 Introduction to the Open FPGA Stack","text":"The Open FPGA Stack (OFS) is a modular collection of hardware platform components, open source upstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS Provides a framework of FPGA synthesizable code, simulation environment and synthesis/simulation scripts. The key components of OFS include: - Target development platforms such as Intel-branded Programmable Acceleration Cards (PACs), Acceleration Development Platforms (ADPs) and third-party platforms.
- Board Management Controller RTL and firmware that supports telemetry monitoring, remote configuration updates and most importantly a root of trust for the platform.
- Source accessible, modular FPGA Interface manager (FIM) RTL with unit tests that can be leveraged for your own custom FIM design
- Basic building blocks for interconnect and PF/VF translation and arbitration; Platform Interface Manager (PIM) which provides Avalon\u00ae bus compliant interfaces.
- AFU examples both in the git repository and workload examples provided by 3rd party vendors
- The OneAPI shim provides a layer that is used by the OneAPI runtime to communicate with the kernel.
- OPAE software development kit (APIs, upstreamed Linux drivers and software tools)
- Support for other frameworks to be built on top of the OPAE such as DPDK
The OFS hardware repository supports hardware development and simulation. Repositories for OFS high level design support and board management controller RTL and firmware source code are also provided. These repositories can be found in the Altera Opensource Technology GitHub location, which requires entitlement access. To request access, please contact your local Altera sales representative.
Table 1-2 OFS GitHub Repositories
OFS GitHub repositories can be found in the OFS.
Repository Contains ofs-fim-common Contains common modules shared by all OFS designs. This repository is a submodule of each platform repository. ofs-d5005 Contains FIM or shell RTL design, automated compilation scripts, unit tests. The OPAE software GitHub site is fully opensource and contains resources for both software and workload developers.
Table 1-3 OPAE Public Git Repositories
OPAE GitHub repositories can be found in the OFS.
OPAE Git Repository Folder Contains linux-dfl Contains OFS Linux drivers that are being upstreamed to the Linux kernel. linux-dfl-backport Backport versions of the linux-dfl to older kernel versions. opae-sdk Contains the files for building and installing OPAE SDK from source. opae-sim Contains an AFU/Workload simulator for software/hardware co-simulation. examples-afu Contains simple AFU tutorials. Providing the hardware and software source code and supporting test frameworks in a GitHub repository allows you to easily customize your own designs with the latest versions.
Most hardware and software ingredients are available in our OFS GitHub location. For access to the board management controller firmware and RTL or our security guide for OFS, please contact a local Altera sales representative.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#13-ofs-features","title":"1.3 OFS Features","text":"The OFS architecture within the FPGA comprises two partitions:
- FPGA Interface Manager (FIM)
- Accelerator Functional Unit (AFU)
The FIM or shell provides platform management functionality, clocks, resets and interface access to the host and peripheral features of the acceleration platform. The FIM architecture along with the supporting OPAE software supports features such as partial reconfiguration and virtualization. The FIM provides a standard Arm* AMBA* 4 AXI4 datapath interface. The FIM resides in the static region of the FPGA.
The AFU partition is provided for custom acceleration workloads and may contain both static and partial reconfiguration regions.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#131-fpga-interface-manager-fim","title":"1.3.1 FPGA Interface Manager (FIM)","text":"The updated OFS architecture for Stratix\u00ae 10 FPGA devices improves upon the modularity, configurability and scalability of the first release of the OFS architecture while maintaining compatibility with the original design. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem
- HSSI Subsystem
- Memory Subsystem
- Reset Controller
- FPGA Management Engine
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- SPI Interface to BMC controller
The AFU Region provides design space for custom workloads and contains both static and partial reconfiguration regions. Partial reconfiguration allows you to update your specific logic blocks or entire workload while the rest of your static design is still in operation.
Note that as discussed previously, the BMC RTL and firmware, the OFS OPAE software stack and support for building your own customer board support package are also provided in separate OFS repositories.
Figure 1-2 OFS for Stratix 10 Block Diagram
The table below details the features of the OFS release targeting the Stratix\u00ae 10 FPGA .
Table 1-4 Features
Key Feature OFS Update Comments PCIe H-tile PCIe Gen3x16 Interface Integrates PCIe TLP adapter for new data mover packet format. MSI-X vector and PBA tables are located in the PCIe subsystem. Interrupts from FME as well as four user interrupts coming from PF0.VF1 are supported. Memory Two Avalon Memory Mapped channels provided as default with capability to compile design with four channels support. - HSSI 1 Arm* AMBA* 4 AXI4-Stream channel of 10G Ethernet, using the low latency Ethernet 10G MAC FPGA IP interfacing to an E-tile PHY. - Manageability SPI interface to Board Management Controller targeting Intel FPGA PAC D5005 - CoreFIM Flexible configuration support using Arm* AMBA* 4 AXI4-Stream Physical Function/Virtual Function (PF/VF) Demux/Mux and AFU Peripheral Fabric (APF) and Board Peripheral (BPF) Fabric Interconnects. APF and BPF fabrics are Platform Designer generated IPs. The Arm* AMBA* 4 AXI4-Stream PF/VF Demux/Mux is a new component. Physical Function/Virtual 1 PF/3VF configuration is provided as an example but the architecture now supports full virtualization with the ability to expand to whatever the PCIe tile supports. - Partial Reconfiguration 1 Partial Reconfiguration region supported in hardware and software - Sample test PR AFUs Host exerciser modules provided to exercise interfaces. These modules are provided in both the flat and PR AFU examples. - OneAPI Yes Available Q1 2023 Software Support OFS software stack with support for full virtualization. -"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FIM contains only one FME, regardless of the number of host interfaces to the FIM. The FME provides management features for the platform and controls reset and loading of the AFU into the partial reconfiguration region of the FPGA.
Each FME feature exposes its capability to host software drivers through a device feature header (DFH) register found at the beginning of its control status register (CSR) space. The FME CSR maps to physical function 0 (PF0) Base address register 0 (BAR0) so that software can access it through a single PCIe link. For more information about DFHs, refer to the Device Feature Header (DFH) structure.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#streaming-datapath","title":"Streaming Datapath","text":"The FIM implements an AXI4-Stream bus protocol for data transfer in the FIM. AXI4-Stream channels send data packets to and from the host channel IP without data abstraction. Memory-mapped I/O (MMIO) CSR accesses are routed to the ST2MM module which converts the AXI4-Stream to an AXI4 memory mapped protocol.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#virtualization","title":"Virtualization","text":"This design supports virtualization by making use of the virtualization functionality in the PCIe Hard IP and mapping packets to the appropriate physical or virtual function through a PF/VF multiplexer. This reference FIM supports 1 PF and 3 VFs as an example; however, you may extend your configuration to whatever the PCIe Hard IP can support or what your application requires.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#132-afu","title":"1.3.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space. The port is part of the FPGA Interface Unit (FIU) that resides in the FIM.
You can compile your design in one of the following ways: * Your entire AFU resides in a partial reconfiguration region of the FPGA * The AFU is part of the static region and is compiled a flat design
In this design, PF0.VF1 and PF0.VF2 map to host exerciser modules (HEM) that map to HE-LB and HE-HSSI respectively.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#133-platform-interface-manager","title":"1.3.3 Platform Interface Manager","text":"The PIM provides a way to abstract the AXI4-Stream interface to the AFU by providing a library of shims that convert the host channel native packet into other protocols such as CCI-P, AXI4 memory-mapped, Avalon\u00ae streaming (Avalon-ST) or Avalon\u00ae memory-mapped (Avalon-MM). The FPGA or AFU developer implement these interface abstractions in the AFU region of the design.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#134-opae-sdk-fpga-platform-feature-discovery","title":"1.3.4 OPAE SDK FPGA Platform Feature Discovery","text":"The OPAE C library in the OPAE software development kit is built on top of the OPAE FPGA driver stack that abstracts the hardware and operating system specific details of the platform to the host. The FIM implements a DFH linked list to allow an FPGA platform driver running on the host to discover FME, port and AFU features. This model is similar to how PCIe enumeration occurs. You must implement a 64-bit DFH Device Feature Header register at the beginning (first 8B aligned address) of the feature CSR space for a new feature to be discovered or enumerated by a driver.
A driver starts the traversing by reading the DFH of the first feature from the first address on PF0 BAR0. Based on the information in the DFH, a driver can determine the CSR address range of the feature and other associated details of the feature. The end of the DFH contains a \"next DFH offset\" field that points the driver to the DFH of the next feature. The software must continue traversing the linked list until it sees the EOL (End-Of-List) bit set to 1 in the \"next DFH offset\" field it is inspecting. A 1 indicates this is the last feature in the feature set. Figure below gives a simple illustration of the feature discovery by traversing the DFH registers.
Figure 1-3 Device Feature Header Linked List Traversal
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#135-ofs-reference-design","title":"1.3.5 OFS Reference Design","text":"OFS provides FIM designs you can use as a starting point for your own custom design. These designs target a specific programmable acceleration card or development kit and exercise key FPGA device interfaces. The Stratix\u00ae 10 code line for OFS targets the Intel FPGA PAC D5005. FIM designs are released to OFS D5005 FIM Github Branch for evaluation and use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#136-fim-simulation","title":"1.3.6 FIM Simulation","text":"OFS provides a UVM environment for the FIM and a framework for new feature verification. UVM provides a modular, reusable, and scalable testbench structure by providing an API framework that can be deployed across multiple projects. The FIM testbench is UVM compliant and integrates third-party verification IPs from Synopsys that require license to use. Verification components include:
- FIM monitor to detect correct design behavior
- FIM assertions for signal level integrity testing
- Arm AMBA AXI4 scoreboards to check data integrity
- FIM coverage to collect functional data
The verification infrastructure can be in the verification folder here OFS D5005 FIM Github Branch for evaluation and use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#2-ofs-high-level-architecture","title":"2 OFS High Level Architecture","text":"OFS provides distinct datapaths that simplifies the design and integration process for add or for removing interface modules:
- High Bandwidth datapath for AFU-attached high performance peripherals (HSSI, Memory, HPS, workload).
- Low Bandwidth datapath for OFS management and slow peripheral components (JTAG, I2C, SMBus).
- AFU Peripheral Fabric (APF) to Board Peripheral Fabric (BPF) path to communicate with interface control and status registers (CSRs) and board components.
- Peer-to-peer datapath between AFU components.
- Peer-to-peer datapath between BPF components.
Depending on your design goals, you can present peripherals to software as:
- OFS managed peripherals with a device feature header that is part of a device feature list.
- Native driver managed peripherals that are exposed through an independent physical function or virtual function.
Figure 2-1 OFS Datapath Structure
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#3-pcie-interface","title":"3 PCIe Interface","text":"The FIM's H-tile PCIe* hard IP is a Gen3x16 design. The IP supports SR-IOV and is configured to provide one PF and three VFs. Native PCIe TLP packets are sent through the PCIe using Arm AMBA 4 AXI-4 Stream Protocol. Before they reach the AFU, however, the packets go through an adapter that converts any headers to a data mover format that is forward compatible with Agilex FPGA devices and beyond.
Figure 3-1 OFS FIM RX-TX Datapath
Some key features of the PCIe interface are:
Feature OFS for Stratix 10 Configuration Mode PCIe Gen3x16 Port Mode Native Endpoint SR-IOV 1 PF, 3 VFs MSI-X Support Yes Functional Mode Data Mover Profile Virtual+ TLP Bypass No Header Packing Scheme Simple Data Width 512-bit (64-byte) PLD Clock Frequency 250 MHz Tags Supported 128 Reordering No reordering of requests, no completion reordering Maximum Payload Size 256 Bytes Memory Requests Supported 1CL, 2CL, 4CL MMIO transaction Size 4B, 8B"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#31-receiver-datapath","title":"3.1 Receiver Datapath","text":"The path of data received by the FIM is as follows:
-
The PCIe Hard IP receives TLP packets from the host. Host response types can be:
* MMIO read or response * Memory write request * Interrupt response * Memory read request or response
-
The PCIe IP routes the TLP packets to the PCIe bridge interface in the FIM, where they get buffered into the RX FIFO.
-
The TLP checker in the bridge examines the packets and filters erroneous packets including packets with unsupported requests or fields. Errors are sent to the error logger and are logged as advance error reporting (AER). Error status is sent to the PCIe hard IP and to the FIM error status registers. Appropriate action is taken for the errors as described in the Reliability, Accessibility, Serviceability (RAS) and Error Handling section. The TLP checker also maintains RX buffer credits for TLP completions with data (CplD) that are received from the host in response to a memory read request (MRd request sent by the AFU). The TLP checker notifies the PCIe TX bridge when there are not enough RX buffer credits available in the PCIe RX bridge. If there are not enough credits, the PCIe TX bridge pauses MRd requests from the AFU to Host until there is availability.
-
The TLP checker forwards packets that pass TLP checking to an AXI4 adapter which moves the packets into an Arm AMBA 4 AXI4-Stream bus.
-
AXI4-Stream packets are sent downstream to datamover AXI4-Stream adapter to be modified into the new datamover packet format.
-
Datamover packets are sent to the PF/VF Mux in the AFU.
-
The AXI4-Stream to Memory Mapped (ST2MM) module in the AFU region routes MMIO requests targeting FIM CSRs (FME, peripherals and AFU).
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#32-transmit-datapath","title":"3.2 Transmit Datapath","text":"The Transmit (TX) datapath refers to all TLP packet types that originate in the FPGA. A single Arm AMBA 4 AXI4-Stream channel at the port interface boundary of the FIM carries PCIe TLP packets and interrupt requests upstream from the AFU.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#33-data-handshaking","title":"3.3 Data Handshaking","text":"The Arm AMBA 4 AXI4 interfaces to the AFU use the VALID and READY signal for handshaking and backpressure management. The FIM holds the DATA and VALID asserted until the receiver asserts the READY signal. The AFU accepts the data when both the VALID and READY signals are asserted.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#34-arm-amba-4-axi4-stream-interface","title":"3.4 Arm AMBA 4 AXI4-Stream Interface","text":"The table below shows the high-level signal mapping of the channels for the OFS for Stratix 10 FPGA. If you have previously used the OFS EA architecture, that mapping is provided as a comparison as well in this table.
Table 3-1 AXI4-Stream RX Channel
AXI4-Stream Signal Source OFS Stratix 10 Mapping OFS Early Access Mapping ACLK Clock Source PCLK = 250 MHz PCLK = 250 MHz AResetn Reset Source System Reset System Reset TVALID Master Data Valid Data Valid TREADY Slave Ready Ready TDATA Master Width=512 bits When packet includes a header (32 bytes) and data the packing scheme is: 16B data mover header: 4B prefix PF NumberVM Number VF Active Slot Number Memory Mapped Number8B Address/Metadata Data: 32B If the packet is only data then all 64 bytes comprise data [TLP_CH] [8n] n=49 (392 bits)TLP_CH=2 (2TLP data streams)Mapping of each TLP data stream [391:136] payload (32-byte data payload) [135:8]: hdr (16 byte header) [7:3]: rsvd0 (reserved bits)[2]: end of packet (eop) [1]: start of packet (sop) [0]: valid (TLP packet on data stream is valid) TLAST Master Set to 1'b1 when end of packet is sent; otherwise TLAST is 1'b0 Set to 1'b1 when end of packet is sent; otherwise TLAST is 1'b0 TKEEP Byte Qualifier Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. TUSER Master WIDTH = 10 Bit 0 is always equal to 1'b1 to indicate data mover header format [4:1] - Indicates header position on the TDATA bus and is always equal to 4'b0001 indicating that the header starts from Byte0.All other bits of TUSER are unused. [TLP_CH][u-1:0] u=21 Sideband of each TLP data stream[20]: ummio_rd (Unsupported MMIO request)[19:0]: destination routing ID, where:[19:17]= BAR offset[2:0][16:4]=VF number[12:0][3:1]=PF number[2:0][0]=vf_active, indicating if the virtual function feature is enabled Figure 3-3 AXI4-Stream RX Request Cycle Header Format
All Host requests sent to the AFU are memory-mapped I/O requests. Of the fields below, the following are not supported in the design: * Prefix * Slot number (set to 0) * Local Address
Figure 3-4 AXI4-Stream RX Completion Header Format
All completions in the RX direction are data completions. Of the fields below, the following are not supported in the design: * Prefix * MM mode * Slot number (set to 0) * Meta Data
Note that: * VF Active, VF Num and PF Num are obtained from TUSER. * Data packet responses (for memory read requests from the AFU) from the PCIe may come out of order when the size is greater than 64 bytes.
Table 3-2 AXI4-Stream TX Channel
AXI4-Stream Signal Source OFS Stratix 10 Mapping OFS Early Access Mapping ACLK Clock Source PCLK = 250 MHz PCLK = 250 MHz AResetn Reset Source System Reset System Reset TVALID Master Data Valid Data Valid TREADY Slave Ready ReadyOnly 1 ready signal for the two channels TDATA Master Width=512 bits When packet includes a header (32 bytes) and data the packing scheme is: 16B data mover header: 4B prefix PF NumberVM Number VF Active Slot Number Memory Mapped Number8B Address/Metadata Data: 32B If the packet is only data then all 64 bytes comprise data [TLP_CH] [8n] n=49 (392 bits)TLP_CH=2 (2TLP data streams)Mapping of each TLP data stream [391:136] payload (32 byte data payload) [135:8]: hdr (16 byte header) [7:3]: rsvd0 (reserved bits)[2]: end of packet (eop) [1]: start of packet (sop) [0]: valid (TLP packet on data stream is valid) TLAST Master Per protocol Set to 1'b1 TKEEP Byte Qualifier Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. Signal indicates whether content of the associated byte is valid. Invalid bytes are allowed only during TLAST cycle. Valid bytes always start from Byte 0. TUSER Master WIDTH = 10 Bit 0 \u2013 Indicates Header Format: 0 \u2013 Power user mode header format 1 \u2013 Data mover header format Bit [4:1] - Indicates header position on the TDATA bus and is always equal to 4'b0001 indicating that the header starts from Byte0.All other bits of TUSER are unused. [TLP_CH][u-1:0] u=2Sideband of each TLP data stream[0]=vf_active, indicating if the virtual function feature is enabled[0]: afu_irq (AFU interrupt) Table 3-3 Interrupt Response Channel
AXI4-Stream Signal Source Mapping ACLK Clock Source PCLK AResetn Reset Source System Reset TVALID Master Valid TREADY Slave Ready TDATA [8n-1:0] n=3 (24 bits) Master [23:16]: 8-bit interrupt ID [15:0]: Requester ID Figure 3-5: AXI4-Stream TX Request Cycle Header Format
All requests in the TX direction are Memory Read/Write. The requester ID does not come from the AFU; the AXI-Stream adapter supplies it. The tag must come from the AFU. Of the fields below, the following are not used in the H-Tile PCIe subsystem design: * Prefix * MM Mode * Slot number (set to 0) * Local Address
Note that VF Active, VF Num and PF Num are obtained from the header packet.
Figure 3-4 AXI4-Stream TX Completion Header Format
All completions in the TX direction are for MMIO. Of the fields below, the following are not supported in the design: * Prefix * MM mode * Slot number (set to 0) * Meta Data
Note that: * VF Active, VF Num and PF Num are obtained from TUSER.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#4-platform-interface-manager","title":"4 Platform Interface Manager","text":"The FIM interfaces to an AFU through AXI4-Stream channels. This format allows the AFU to access the host channel's raw interface without any translation. As a FIM developer, you have the option to provide the raw data format associated with the host interface channel to the workload or AFU developer or you can provide an intermediate protocol using Platform Interface Manager Components or your own custom interface. If you expose the raw AXI4-Stream interface of the FIM, workload developers also have the option to convert to a desired protocol using the PIM resources as well.
Refer to https://github.com/OPAE/ofs-platform-afu-bbb for more information on options for implementing the PIM.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#5-afu-interface-handler","title":"5 AFU Interface Handler","text":"The AFU Interface Handler resides inline between the PCIe AXI4-Stream Adapter and the AXI4-Stream PF/VF Demux/Mux logic. Its main function is to provide: * Unique PCIe tags \u2013 Each PCIe transaction shares the 128 tags across all VFs in the AFU region * AFU error logging for all VFs in the AFU region
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#51-afu-error-handling","title":"5.1 AFU Error Handling","text":"In this OFS design, the AFU Interface Handler handles error logging for all VFs in the AFU. Errors handled are as follows
Checker Field Description AFU protocol checker (PCIe TLP) TxReqCounterOverflow Pending memory write or memory read requests exceed the predefined limit TxFifoOverflow Tx FIFO in the port that buffers TLP packets from AFU is overflow Malformed TLP AFU PCIe TLP contains unsupported format type MaxPayloadError AFU memory write payload size exceeds max_payload_length limit MaxReadReqSizeError AFU memory read payload size exceeds max_read_request_size limit MaxTagError AFU memory read request tag value exceeds the maximum supported tag count TagOccupiedErr AFU sends out memory read request using a tag that is already used for a pending memory read request UnalignedAddrErr The address field in AFU memory write/read request TLP is not DW-aligned. UnexpMMIOResp AFU is sending a MMIO read response with no matching MMIO read request. MMIOTimedOutAFU is not responding to a MMIO read request within the pre-defined response timeout period. MMIODataPayloadOverrunThe number of data payload sent by AFU for a MMIO response (cplD) is more than the data length specified in the response. MMIOInsufficientDataThe number of data payload sent by AFU for a MMIO response (cplD) is less than the data length specified in the response. TxMWrDataPayloadOverrun The number of data payload sent by AFU for a memory write request is more than the data length specified in the request. TxMWrInsufficientData The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU Protocol Checker (AXI4-Stream)TxValidViolationThree checkers are implemented in the FIM to catch errors and protocol violations. To view the CSR space for the AFU interface handle, go to the src/afu_top/AFU_INTF_CSR.xls file OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#6-interconnect-fabric","title":"6 Interconnect Fabric","text":"There are three types of interconnect fabric in the OFS FIM design: * AXI4-Stream mux/demux fabric * AFU Periheral Fabric (APF) * Board Peripheral Fabric (BPF)
Figure 6-1 Interonnect Fabric Diagram
TLP packets sent from upstream PCIe Subsystem on AXI4-Stream channel are demultiplexed in the AXI4-Stream PF/VF mux/demux fabric and routed to the respective PF/VF function based on the PF/VF information in the TLP header, such as vf_active or the PF/VF number. On the opposite direction, TLP packets from downstream PF/VF function are muxed in the fabric and sent to PCIe subsystem over AXI4-Stream channel.
All host MMIO requests targeting PF0 BAR0 are routed to the ST2MM module. The ST2MM converts MMIO TLP packets into AXI-Lite memory requests and places the requests onto AFU Peripheral Fabric (APF). AFU peripherals, such as OFS managed AFU features and ST2MM) and Board Peripheral Fabric (BPF) are interconnected by APF. The BPF is the interconnect fabric one hiearchy below APF which connects all the board peripherals. Both APF and BPF allow multiple AXI4-Lite master and slave interconnect topology.
The following table summarizes the mechanism for configuring PF/VF functions:
Table 6-1 Interconnect Configuration Methods
InterconnectConfiguration Mechanism APF or BPFEither:Use Platform Designer (PD) to generate the fabrics directly, or Specify desired cfg in iofs_dfl.txt and run dfh2tcl.pl script to generate the necessary HW TCL scripts as needed by PD. This PERL script also takes care of invoking PD in script mode to generate the end result of RTL design Verilog files. This is the preferred method for generation. AXI-S PF/VF Demux/MuxUpdate parameters in these RTL files: * src/includes/top_cfg_pkg.sv * src/common/pf_vf_mux.sv Then make the corresponding update to AFU top level instantiation and connections: * src/FIMs/.../afu_top.sv Note:
In all cases, you must configure and regenerate the PCIe IP to match the new PF/VF configuration if it deviates from the reference FIM provided.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#61-fabric-generation-flow","title":"6.1 Fabric Generation Flow","text":"You have two options to generate APF/BPF IPs.
- Use Quartus Platform Designer. This method requires familiarity with Quartus Platform Designer. With this option, you manually enter each address space and define the associated master and slave interface in a table provided by the Platform Designer. The parameters and attributes such as data width, address width, number of outstanding cycles, \u2026 etc. are also set to the desired values. After this is completed, you then connect each master and slave interface in the table. Platform Designer then generates the RTL files according to the table. For more details, please refer to the Intel Quartus Prime Pro Edition User Guide.
- Use the APF/BPF script that reads in
iofs_dfl.txt and automatically generates APF/BPF IPs. Both APF and BPF are generated from Platform Designer using hardware TCL scripts. To provide a more user friendly experience to OFS Rel 1 and AC ADP customers, a perl script dfh2tcl.pl has been developed to provide a higher level of abstraction. The below figure illustrates the high level flow:
Figure 6-2 APF/BPF Generation
Note that the only input required is the iofs_dfl.txt text file which allows you to specify how many ports, which fabric, type of AXI4-Lite port (master, slave, or both), port addresses and sizes. Using iofs_dfl.txt and dfh2tcl.pl to generate the APF and BPF is the preferred method.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#62-iofs_dfltxt-format","title":"6.2 iofs_dfl.txt Format","text":"The following table describes the format of the iofs_dfl.txt input file:
Table 6-2 iofs_dfl.txt Format Types
ColumnDescription REGISTERName of the port on the APF/BPF fabrics. Note that each entry must be unique. FABRICAllows the user to specify which fabric this port is connected to, and also the type of AXI4-Lite port (master, slave, or both). The format is FABRIC-PORT_TYPE, where: * FABRIC = APF or PBF * PORT_TYPE = MST, SLV, or BID (Master, Slave, or Bi-Directional) BASE_ADDRSpecifies the address offset of the AXI4-Lite port. BAR_SIZE=2^N Specifies the size of the port. For simplicity reasons, all AXI4-Lite ports are configured as 64KB. i.e. 2^16 Additional AXI4-Lite ports can be easily created by adding more rows in the iofs_dfl.txt file.
Note that there are several reserved ports in both APF and BPF so it might not be necessary for you to regenerate the fabrics if the design does not exceed the number of ports as implemented in the APF/BPF in the reference FIMs.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#63-afu-peripheral-fabric-apf","title":"6.3 AFU Peripheral Fabric (APF)","text":"The AFU Peripheral Fabric (APF) is a 64-bit AXI4-lite compliant interconnect fabric that connects AFU peripheral modules to board peripheral modules through the Board Peripheral Fabric (BPF). The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by the APF is listed below. All components are mapped to PF0 BAR0 and implement AXI-lite slave interface. The Master column indicates if a component also implements AXI4-lite master interface which can send request to APF.
Table 6-3 APF Address Mapping
AddressSize (Byte)FeatureMaster 0x00000 \u2013 0x7FFFF512KBoard Peripherals (See BPF address mapping) No AFU Peripherals 0x80000 \u2013 0x8FFFF64KST2MMYes (Send MMIO request to all the peripherals) 0x90000 \u2013 0x9FFFF64KPort GasketYes 4KPR Control & Status 4KUser clock 16KRemote STP 0xA0000 \u2013 0xAFFFF64KAFU Interface HandlerNo 0xB0000 \u2013 0xBFFFF64KRSV_b_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xC0000 \u2013 0xCFFFF64KRSV_c_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xD0000 \u2013 0xDFFFF64KRSV_d_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xE0000 \u2013 0xEFFFF64KRSV_e_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. 0xF0000 \u2013 0xFFFFF64KRSV_f_DFHAvailable for customer useMay be programmed as master if used. By default this is a reserved base address. The five reserved regions have associated ports available in the APF for customer use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#64-board-peripheral-fabric-bpf","title":"6.4 Board Peripheral Fabric (BPF)","text":"The Board Peripheral Fabric is the 64-bit AXI4-Lite compliant interconnect fabric that connects board peripheral modules to APF. The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by BPF is listed below. All components are mapped to PF0 BAR0 and implement AXI4-lite slave interface. The Master column indicates if a component also implements AXI4-lite master interface which can send request to BPF.
Table 6-4 BPF Address Mapping
AddressSize (Byte)FeatureMaster 0x00000 \u2013 0x0FFFF64KFME (FME, Error, etc)Yes 0x10000 \u2013 0x1FFFF64KSPI ControllerYes 0x20000 \u2013 0x2FFFF64KPCIe CSR- 0x30000 \u2013 0x3FFFF64KHSSI CSR- 0x40000 \u2013 0x4FFFF64KEMIF CSR- 0x50000 \u2013 0x5FFFF64KReservedAvailable for customer use.Dependent on user programming 0x60000 \u2013 0x6FFFF64KReservedAvailable for customer use.Dependent on user programming 0x70000 \u2013 0x7FFFF64KReservedAvailable for customer use.Dependent on user programming The three reserved regions have associated ports available in the BPF for customer use.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#65-axi4-stream-pfvf-muxdemux","title":"6.5 AXI4-Stream PF/VF Mux/Demux","text":"The AXI4-Stream PF/VF Mux/Demux routes the PCIe TLP packets from the PCIe subsytem AXI4-Stream RX channel to downstream PF/VF based on the pf_num and vf_num information in the PCIe TLP header.
The AXI4-Stream PF/VF mux arbitrates PCIe TLP packets from downstream PF/VF to the PCIe SS AXI-S TX channel. The PF/VF Mux/Demux is an M x N switch that allows any M port to target any N port, and any N port to target any M port, where M is the number of host/upstream ports, and N is the numbers functions/downstream ports. M and N values are parameterized in the RTL for adding, removing, or remapping of FPGA functional units/modules to PF/VF.
The fpga top package file, found in the src/includes/ofs_fim_cfg_pkg.sv file OFS D5005 FIM Github Branch contains these parameters as well as the mapping of N port\u2019s PF/VF.
Structurally, M x N switch is composed of M number of N:1 mux, and N number of M:1 mux. Each mux output has an arbiter that perform round robin priority arbitration of its inputs. At the mux output is a FIFO with depth greater than the handshake round trip delay. The FIFO allows the switch to arbitrarily insert pipeline/register stages for timing.
Note that M x N switch is design for AXI streaming, but it can be easily converted to AVST. The protocol signals pass through switch intact \u2013 only ready, valid, and last (common between AVST and AXI) effect switch operation. The data width of the switch is also parameterized in the src/includes/ofs_fim_cfg_pkg.sv file OFS D5005 FIM Github Branch.
The default mapping is shown below:
Table 6-5 PF/VF Mapping
DevicePhysical Function #Virtual Function #Switch Port ID APF/BPF0x3 HE Loopback000 Port Gasket011 HSSI022 For information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#66-unified-tag-remapping","title":"6.6 Unified Tag Remapping","text":"When a FPGA function sends out a read cycle, it allocates a unique tag which is subsequently used to identify the read completion. The tag is considered busy; it cannot be assigned to another read cycle until read completion. While a tag may be unique within a unit, two different units could unknowingly send out two read cycles of the same tag. The PCIe subsystem requires unique tags for all read cycles irrespective of their origins. Therefore, a mechanism is needed to uniquify tag globally across different units.
OFS contains a tag remapper (tag_remap) that intercepts the read cycle, finds a globally unique tag, and replaces the original tag value. It also restores the original tag value when returning completion to the read requester. tag_remap is placed between the AXI4-Stream interface of the PCIE subsystem and the PF/VF Mux/Demux.
The logic is described as follows:
- A sub-module (ofs_fim_tag_pool) maintains a pool of available tags.
- TX read requests are held until a tag is available from the pool by setting tvalid=0 to the host, and tready=0 to the PF/VF Mux/Demux.
- When a TX read is dispatched, the tag is marked busy in the pool.
- The original tag is stored in tag_reg, so it can be recovered when returning a completion to the unit/function.
- Since completion to a read request can split into multiple smaller transfer sizes, responses are monitored and the final completion is detected using PCIe TLP rules.
- Tags are released in the pool only when all requested data are transferred.
- When the completion returns, the original tag is restored from tag_reg.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#67-tlp-to-axi4-lite-memory-mapped-bridge-st2mm","title":"6.7 TLP to AXI4-Lite Memory Mapped Bridge (ST2MM)","text":"ST2MM implements the following key features: * Host MMIO bridge * Maps MMIO TLP packets received from the PCIe Subsystem over streaming interface to AXI4-Lite memory-mapped request. The memory-mapped request is sent to AFU or Board peripherals over APF and BPF. * Maps AXI4-lite MM response received from AFU or Board peripherals to TLP packets and send the packets over ST streaming channel to host HIA subsystem. * Sends MMIO response of all 0\u2019s for MMIO read to unused BAR region. * Interrupt * Sends interrupt packets to the PCIe subsystem when interrupt requests are received from the peripherals. Interrupts can be requested by a peripheral through a memory write to interrupt CSR registers in the ST2MM.
Figure 6-2 APF/BPF Generation
ST2MM implements both AXI4-lite master and slave interfaces that are connected to the designated slave and master port on APF. Host memory requests are sent on the ST2MM master interface to AFP where the requests are routed to the targeted peripherals.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#7-mmio-regions","title":"7 MMIO Regions","text":"The FIM and AFU expose their functionalities to the host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). An MMIO region is an address space within a base address register (BAR) region to which features are memory mapped. For example, when a feature is mapped to an MMIO region, the CSR registers of that feature are located within the address range of that region. There can be multiple MMIO regions within a BAR region.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#71-base-address-register-bar-layout","title":"7.1 Base Address Register (BAR) Layout","text":"The function, BAR and external feature region starting address are put into a platform specific parameter SystemVerilog package file src/includes/ofs_fim_cfg_pkg.sv file OFS D5005 FIM Github Branch.
You can modify the parameterization according to your platform requirements, however you must ensure the corresponding software driver is also updated to align with the new assignment.
Table 7-1 BAR Layouts
PF VF Feature BAR BAR Size PF0 - OFS Managed Peripherals BAR 0 512K AFU PeripheralsBoard Peripherals 256K256K VF0 HE-LB BAR0 4K VF1 HE-MEM in PR slot BAR0 4K VF2 HE-HSSI BAR0 4K"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#72-feature-region","title":"7.2 Feature Region","text":"A group of related CSRs can be categorized as a feature region. For example, a DMA engine has queue management function and quality of service (QoS) function; these are two different features of the DMA engine. A feature region is contained within a single PCIe BAR and cannot span across two BAR region boundaries. You can view the PF0 BAR0 MMIO mapping by referencing thesrc/common/fme/fme_csr_pkg.sv file OFS D5005 FIM Github Branch file.
A Device Feature Header (DFH) register marks the start of the feature region and sub-feature region, and you must place it at the first address of the region. Each DFH starts at 4KB boundary. A DFH register contains information that OPAE software requires to enumerate the feature. It also has an offset field that points to the next DFH in a feature list. OPAE software traverses the linked list of DFHs in each BAR region to discover all the features implemented on the platform. The EOL field in a DFH marks the end of a DFH list and is only set in the DFH of the last feature in the feature list. The feature type field in the DFH is used to differentiate between the different types of feature region. Basic building blocks (BBB) and private features are always a child of an AFU or FPGA Interface Unit (FIU) and must be contained within an AFU or FIU, respectively.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#721-device-feature-header-dfh-structure","title":"7.2.1 Device Feature Header (DFH) Structure","text":"All DFHs must follow the following structure to be compatible with OPAE software.
Table 7-2: DFH Structure
Bitfield Name Range Access Description FeatureType 63:60 RO 4\u2019b0000 \u2013 Reserved 4\u2019b0001 \u2013 AFU4\u2019b0010 \u2013 BBB4\u2019b0011 \u2013 Private Feature4'b0100 \u2013 FIU/FIM Reserved 59:41 Rsvd Reserved EOL 40 RO End of DFH List1'b0=No other feature header beyond this one1'b1=This is the last feature header NextDFHByteOffset 39:16 RO Next DFH byte offsetNext DFH Address= Current DFH address + Next DFH byte offset. You can also use this value as an indication of the maximum size of the MMIO region occupied by this feature. FeatureRev 15:12 RO For AFU Feature type= AFU major version number that is user defined.All other feature types= Feature revision number FeatureID 11:0 RO For AFU feature type= CoreFIM version numberFor BBB feature type= Intel defined ID for BBBFor private feature type= User-defined ID to identify within an AFU For FIU type=ID for FIU unit (ex. 0 for FME, 1 for Port) You must increment a feature revision number if a feature changes. This change requires a corresponding change in the software to detect the new version and report mismatches between the hardware and software revision number.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#73-control-and-status-registers","title":"7.3 Control and Status Registers","text":"All the Control and Status Registers (CSRs) in the FIM are 64-bit registers with the following MMIO write and MMIO read support.
Table 7-3: CSR MMIO Read and Write Support
Request Memory Attribute Payload size Memory Ordering MMIO Write UC 4B or 8B Strongly ordered MMIO Read UC 4B or 8B Strongly ordered The FIM does not reorder the MMIO requests or responses. For MMIO writes, there is no reordering of requests in FIM, and UC ordering rules are followed. Similarly, for MMIO reads, there is no re-ordering of requests or responses in the FIM. An AFU may opt to re-order the MMIO read responses but the FIM does not enforce read response ordering.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#731-software-access-to-registers","title":"7.3.1 Software Access to Registers","text":" - Software accesses 64-bit registers as aligned quadwords. For example, to modify a field (bit or byte) in a 64-bit register, the entire quadword is read, the appropriate field(s) are modified, and the entire quadword is written back.
- When updating registers through multiple accesses (whether in software or due to hardware disassembly), certain registers may have specific requirements on how the accesses must be ordered for proper behavior. These are documented as part of the respective register descriptions.
- For compatibility with future extensions or enhancements, software must assign the last read value to all \u201cReserved and Preserved\u201d (RsvdP) fields when written. In other words, any updates to a register must be read so that the appropriate merge between the RsvdP and updated fields occurs. Also, software must assign a value of zero for \u201cReserved and Zero\u201d (RsvdZ) fields when written.
- PCIe locked operations to FPGA hardware registers are not supported. Software must not issue locked operations to access FPGA hardware registers.
In the following two cases, the FIM terminates MMIO Read requests by sending a completion with the data (CplD) specified below: * MMIO Timeout: This occurs when the AFU does not respond within a set timeout. The timeout value is currently configured to 512 pclks (clk_2x). In this case, the FIM returns all 1s.
- Illegal MMIO Accesses: This occurs when the read is accessing undefined registers in the FIM or if an AFU access violation. An example of an access violation is when a PF attempts to access the AFU when it is set to VF mode, or when a VF attempts to access the AFU when it is set to PF mode. In this case, the FME will return all 0s.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#732-register-attribute-definition","title":"7.3.2 Register Attribute Definition","text":"Table 7-4: OFS Register Attribute Definitions
Attribute Expansion Description RW Read/Write This bit can be read or written by software. RO Read Only The bit is set by hardware only. Software can only read this bit. Writes do not have any effect. RW1C Read/ Write 1 to Clear Software can read or clear this bit. The software must write 1 to clear this bit. Writing zero to RW1C bit has no effect. Note that a multi-bit RW1C field may exist. In this case, all bits in the field are cleared if a 1 is written to any of the bits. RW1S Read/ Write 1 to Set Software can read this bit. Writing a 1 to the bit sets it to 1. Writing a 0 has no effect. It is not possible for software to set this bit to 0. The 1 to 0 transition can only be performed by HW. RW1CS Read/Write 1 to Clear Sticky Software can read and clear this bit. Writing a 1 to a bit clears it, while writing a 0 to a bit has no effect. This bit is only reinitialized to its default value by a power-on reset. RWD Read/Write Sticky across Hard Reset The bit can be read or written by SW. This bit is sticky or unchanged by any reset type, including Hard Reset. The bit gets cleared only with power on. *S Sticky across Soft Reset The bit will be sticky or unchanged by soft reset. These bits are only re-initialized to their default value by a power-on reset. *D Sticky across Hard Reset The bit is sticky or unchanged by or unchanged by any reset type, including hard reset. The bit gets cleared only with power on. Rsvd Reserved Reserved for future definitions. Currently don\u2019t care bits. RsvdP Reserved and Protected Reserved for future RW implementations. The software must preserve the value of this bit by read modify write. RsvdZ Reserved and Zero Reserved for future RW1C implementations. The software must write zero to this bit."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#733-csr-offset-in-bars","title":"7.3.3 CSR Offset in BARs","text":"The table below captures the FIM and AFU features in the supported BAR regions. The offset highlighted in red indicates the first DFH in the DFH list of a BAR region where device driver starts the DFH traversal.
Table 3-6: PF0 BAR0 Features
Offset Feature CSR set 0x00000 FME 0x03000 Global Performance 0x04000 Global Error 0x10000 SPI Controller 0x20000 PCIe CSR Interface 0x30000 HSSI CSR Interface 0x40000 EMIF CSR Interface 0x80000 Reserved for ST2MM Bridge 0x90000 PR Control & Status (Port Gasket) 0x91000 Port CSRs (Port Gasket) 0x92000 User Clock (Port Gasket) 0x93000 Remote SignalTap (Port Gasket) 0xA000 AFU Errors (AFU Interface Handler) Table 3-7: PF0 BAR4 Features
Offset Feature CSR set 0x02000 MSI-X 0x03000 MSI-X PBA Tables Table 3-8: PF0-VF0 BAR0 Features
Offset Feature CSR set 0x00000 HE-LBK Table 3-9: PF0-VF1 BAR0 Features
Offset Feature CSR set 0x00000 HE-MEM Table 3-10: PF0-VF2 BAR0 Features
Offset Feature CSR set 0x00000 HE-HSSI"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#8-fim-clocks","title":"8 FIM Clocks","text":"The following table provides the clocks available in the OFS reference design that targets the Intel FPGA PAC D5005. Clocks that the high speed serial interface (HSSI) or external memory interface provide to the FIM may be different depending on if you modify your external features with different components.
Table 8-1: External Clock Source
Clock Frequency Description SYS_RefClk 100 MHz Reference clock to system IOPLL (sys_pll) which provides FIM system clocks. qsfp*_644_53125_clk 644.5312 5MHz HSSI reference clocks ddr4_mem*.ref_clk 150 MHz Reference clocks to DDR4 interfaces PCIE_REFCLK 100MHz PCIe reference clock Table 8-2: Internal Clocks
Clock Frequency Description clk_1x 250 MHz Generated by the system IOPLL (sys_pll). This clock drives CoreFIM datapath and the AFU interface. clk_div2 125 MHz Generated by the system IOPLL, synchronous to clk_1x. This clock drives IM datapath and AFU interface. clk_100 100 MHz Generated by the system IOPLL, synchronous to clk_1x. This clock also supplies the HSSI reconfiguration clock. avl_clk 250 MHz PCIe H-tile Hard IP clock output. This clock is not synchronous to Clk_* DDR4x_USERCLK 299.76 MHz Each of the four DDR interfaces generates one of these clocks, which provides the the clock to the DDR4* Avalon Memory Mapped interfaces. Your memory clock output may be different depending on the memory interface you are implementing in your design. uclk_usr User defined Provides an AFU user clock running at a user specified frequency. Generated by user IOPLL. Not synchronous to clk_*. uclk_usr_div2 uclk_usr/2 Second user clock to AFU running at half the frequency of uclk_usr. Synchronous to uclk_usr. Generated by user IOPLL. hssi[*].f2a_tx_parallel_clk_x1 156.2 MHz 1x TX clock generated from fPLL in the HSSI module in FIM, used to clock the HSSI TX datapath in AFU. hssi[*].f2a_tx_parallel_clk_x2 312.5 MHz 2x TX clock generated from fPLL in the HSSI module in FIM, used to clock the HSSI TX datapath in AFU. hssi[*].f2a_rx_clkout 322.265625MHz RX parallel clock from the HSSI PHY channels, used to clock the HSSI RX data datapath in AFU."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#9-reset","title":"9 Reset","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#91-reset-signals","title":"9.1 Reset Signals","text":"The system reset of OFS reference platform is driven by nPERST pin, pcie_reset_status signal from the PCIe hard IP, the INIT_DONE and nCONFIG pins of the FPGA, and the locked signal of the SYS IOPLL that provides system clocks to FIM.
Upon power-on, the reset module in the FIM holds the FIM in reset until all the reset conditions are de-activated:
nPERST signal is asserted.- The
INIT_DONE pin is driven high to indicate core configuration is complete. - The SYS IOPLL is locked.
- The reset status from PCIe hard IP is de-asserted indicating the IP is ready for transfer.
The reset module places the FIM back into reset if any of these conditions becomes active again. The only way to invoke a system reset to the FIM after power-up is to deassert the nPERST pin either by performing a warm reboot or through PCIe driver intervention. There are soft reset signals set aside to allow software to reset the Port, AFU and partial reconfiguration IP.
Table 9-1: FIM System Resets
Reset Description nPERST pin Active low PCIe reset pin that serves as the system reset pin on the platform. nCONFIG pin Active low input to the FPGA that causes the FPGA to lose its configuration data, enter a reset state, and tri-state all I/O pins. Host software must reload the FPGA FIM after nCONFIG is activated. ninit_done Active low signal derived from the INIT_DONE pin which indicates the FPGA core configuration is complete and has entered usermode. pcie_reset_status Active high reset status from PCIe hard IP. When driven high, this signal indicates that the PCIe IP core is not ready for usermode. pll_locked Active high SYS IOPLL locked signal Table 9-2: Soft Resets
Soft Reset Bitfield Register Description PortSoftReset PORT_CONTROL[0] Resets Port and AFU. FlrPortReset PORT_CONTROL[3] PCIe function level reset that resets Port and AFU when SR-IOV is enabled. PRReset FME_PR_CTRL[0] Resets the partial reconfiguration (PR) controller."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#92-platform-power-up-sequence","title":"9.2 Platform Power up Sequence","text":"Upon power up, the HSSI interfaces of the FIM go through an internal reset and calibration sequence. After the nPERST is de-activated, the PCIe interface and EMIF interfaces are first released from reset, followed by SYS IOPLL when the ninit_done is de-asserted. The rest of the FIM logic is still being hold in reset. The nPOR signal to the PCIe hard IP de-activates following nPERST assertion, which releases PCIe hard IP from reset. The PCIe hard IP asserts pld_clk_inuse to indicate to the application layer that the HIP transaction layer is using the pld_clk as its clock and is ready for operation with the Application layer (pld_clk is stable). Finally, reset_status from PCIe IP is de-asserted. When SYS IOPLL is locked and the reset_status from the PCIe interface is de-asserted, the FIM is released from reset while the Port and AFU is still held in reset by the PortSoftReset register bit (PORT_CONTROL[0]) that is held high until software writes a 0 to this bit to de-activate port reset. At this point, the platform is fully released from reset and PCIe link is going through link training and PCIe enumeration. Once the platform is successfully enumerated, driver can then be loaded to start the device feature discovery process and provision the platform for AFU application usage.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#10-interrupts","title":"10 Interrupts","text":"The OFS platform supports interrupt through MSI-X feature. The OFS reference platform supports at least 4 FME interrupts (PF only) and 4 AFU interrupts (PF and VF).
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#101-msi-x","title":"10.1 MSI-X","text":"In the default implementation the MSI-X feature that handles FME and AFU interrupts is inside the PCIe Subsystem. The MSI-X vector table and Pending Bit Array (PBA) table for PF0 and PF0/VF1 are provided as an example. FME interrupts are primarily used to notify the host of error events occurring in the FIM.
All interrupt requests arrive inband through the AXI4-Stream interface to the TX AXI4-stream adapter inside the PCIe Subsystem.
An AFU sends an interrupt to the MSI-X module on the AXI interrupt request channel. After the interrupt request is serviced, the MSI-X module sends an interrupt response back to the AFU on the AXI interrupt response channel. The AFU has the flexibility to define the use of each AFU interrupt.
The following interrupts are supported: PF supports 7 interrupt vectors: - 0-3: User AFU triggered - 4: Port error triggered - 6: FME error triggered
VF supports 5 interrupt vectors: - 0-3: User AFU triggered - 4: Port error triggered
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#11-external-memory-interface-emif","title":"11 External Memory Interface (EMIF)","text":"There are four DDR4 external memory interfaces on the OFS EA FIM that targets the Intel FPGA PAC D5005 FIM for each of the four DDR4 banks (DDR4a, DDR4b, DDR4c, and DDR4d). Two of the DDR4 external memory interfaces and the associated clocks are directly exposed to AFU except for two Avalon Memory Mapped pipeline bridges to facilitate timing closure across PR boundary. The Avalon Memory Mapped interfaces of each external memory interface are connected to an Avalon-MM pipeline bridge (avmm_bridge) in the FIM, which is then connected to another Avalon-MM pipeline bridge in the PR or AFU region. An AFU should use the USER_CLK associated with a memory interface when interfacing with the memory interface for better timing performance.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#111-emif-csr","title":"11.1 EMIF CSR","text":"The CSR for the EMIF feature is memory mapped to the FME BAR region. Following table captures the EMIF CSR registers.
Table 9-1: EMIF CSR Registers
EMIF_DFH 0x40000 0x3000000050000009 EMIF Management DFH FIELD NAMERANGEACCESSDEFAULT DESCRIPTION FeatureType [63:60] RO 0x3 Feature Type = Private Feature Reserved40 [59:40] RsvdZ 0x0 Reserved NextDfhByteOffset [39:16] RO 0x050000 Next DFH Byte offset FeatureRev [15:12] RO 0x0 Feature Revision FeatureID [11:0] RO 0x9 Feature Id EMIF_STAT 0x40008 0x0000000000000000 EMIF Status FIELD NAMERANGEACCESSDEFAULT DESCRIPTION Reserved [63:16] RsvdZ 0x0 Reserved Reserved [15:12] RsvdZ 0x0 Reserved EmifCalFail [11:8] RO 0x0 EMIF PHY Calibration Failure (1 bit per interface) Reserved [7:4] RsvdZ 0x0 Reserved EmifCalSuccess [3:0] RO 0x0 EMIF PHY Calibration Successful (1 bit per interface) \u200b \u200b
EMIF_CTRL 0x40010 0x0000000000000000 EMIF Control FIELD NAMERANGEACCESSDEFAULT DESCRIPTION Reserved [63:0] RsvdZ 0x0 Reserved"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#92-afu-emif-interface","title":"9.2 AFU EMIF Interface","text":"The FIM exposes 576-bits of Avalon Memory-Mapped data to the AFU, with 512-bit data and additional 64 bits that can either be used for additional metadata, parity or ECC. The AFU has the flexibility to decide the use of the extra 64 bits of data. The ECC soft IP is not enabled in the EMIF IP to allow for the afore-mentioned flexibility. AFU developers can implement the ECC logic in the AFU by making use of the extra 64-bit of data. Avalon Memory-Mapped is the native interface protocol used by EMIF IP. AFU developers who desire other interface protocol in their designs over Avalon Memory-Mapped, such as AXI4 Memory-Mapped, can leverage the bridge in the PIM library.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12-hssi-subsystem","title":"12 HSSI Subsystem","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#121-hssi-subsystem-overview","title":"12.1 HSSI Subsystem Overview","text":"The high speed serial interface (HSSI) subsystem architecture provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack. This reference FIM contains the Low Latency Ethernet 10G MAC IP and provides a Linux driver that can be leveraged for customization.
The HSSI design is leveraged from our OFS EA release so prior customers can easily maintain compatibility with past designs.
A host exerciser, named he-hssi, is provided in the pr_slot of the AFU partition. The Ethernet interface to the AFU has an AXI4-Stream data and sideband interface. The HSSI control and status registers in the FIM are accessible by the AXI4-Stream memory mapped interface.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#122-ofs-hssi-subsystem-interfaces","title":"12.2 OFS HSSI Subsystem Interfaces","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1221-hssi-subsystem-fim-interfaces","title":"12.2.1 HSSI Subsystem FIM Interfaces","text":"There are three interfaces to the HSSI Subsystem that is part of the FIM:
- AXI4 Memory Mapped to access HSSI-CSR (to FIM)
- AXI4-Stream Ethernet Data Interface (from FIM)
- AXI4-Stream Ethernet Sideband Interface (from FIM)
The PCIe subystem uses AXI Memory Mapped accesses to read and write HSSI Control and Status Registers in the FIM. The Ethernet MAC interface typically has a data streaming interface which is mapped to standard AXI4-Stream.
Figure 12-1: HSSI Subsystem
Additionally, the Ethernet MAC interface has an interface for status and flow control. Status and flow control signals vary across IPs and operating modes so for this design we group the signals into a AXI4-Stream sideband interface which provides a standard interface to the AFU along with platform customizations if needed. The Avalon to Arm AMBA 4 AXI4 bridge (av_axi_st_bridge) converts native Avalon interfaces to an AXI4 Stream interface.
The following flow control are implemented in the Ethernet MAC: * IEEE 802.3 flow control: this flow control implements the IEEE 802.3 Annex 31B standard to manage congestion. When the Low Latency Ethernet 10G MAC IP experiences congestion, the core sends a pause frame to request its link partner to suspend transmission for a given period of time. This flow control is a mechanism to manage congestion at the local or remote partner. When the receiving device experiences congestion, it sends an XOFF pause frame to the emitting device to instruct the emitting device to stop sending data for a duration specified by the congested receiver. Data transmission resumes when the emitting device receives an XON pause frame (pause quanta = zero) or when the timer expires.
\u2022 Priority-based flow control (PFC): this flow control implements the IEEE 802.1Qbb standard. PFC manages congestion based on priority levels. It supports up to 8 priority queues. When the receiving device experiences congestion on a priority queue, it sends a PFC frame requesting the emitting device to stop transmission on the priority queue for a duration specified by the congested receiver. When the receiving device is ready to receive transmission on the priority queue again, it sends a PFC frame instructing the emitting device to resume transmission on the priority queue
To use flow control, set the following registers:
On the TX datapath: 1. Set tx_pfc_priority_enable[7:0] (Address :0x0046 -> 11A0) to 0 to disable the PFC. The rest of the bits are unused. 2. Set tx_pauseframe_enable[0] (Address :0x0044 -> 1142) to 1 to enable the flow control.
On the RX datapath:
\u2022 Set rx_pfc_control[7:0] (Address :0x00C0 -> 818) to 1 to disable the PFC. The rest of the bits are mostly unused. \u2022 Set the IGNORE_PAUSE (Address :0x00AC -> 800) bit in the rx_frame_control register to 0 to enable the flow control.
To use Priority-Based Flow Control Follow these steps to use the priority-based flow control (PFC): 1. Enable the Priority-based flow control (PFC) parameter and specify the number of priority levels using the Number of PFC priorities parameter. You can specify between 2 to 8 PFC priority levels. 2. Set the following registers.
On the TX datapath: * Set tx_pauseframe_enable (Address :0x0044 -> 1142) to 0 to disable the flow control. * Set tx_pfc_priority_enable[n] (Address :0x0046 -> 11A0) to 1 to enable the PFC for priority queue n.
On the RX datapath: * Set the IGNORE_PAUSE bit in the rx_frame_control (Address :0x00AC -> 800) register to 1 to disable the flow control. * Set the rx_pfc_control[7:0] (Address :0x00C0 -> 818) register bits to 0 to enable the PFC. Most of the rest of the bits are unused. * Set PFC Quanta bit for the appropriate queue. Eg: pfc_pause_quanta_0 (0x048 -> 1180) for queue0 and so on. * Connect the avalon_st_tx_pfc_gen_data signal to the corresponding RX client logic and the avalon_st_rx_pfc_pause_data signal to the corresponding TX client logic. * You have the option to configure the MAC RX to forward the PFC frame to the client by setting the rx_pfc_control[16] register to 1. By default, the MAC RX drops the PFC frame after processing it
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1222-hssi-subsystem-afu-interfaces","title":"12.2.2 HSSI Subsystem AFU Interfaces","text":"The HSSI subsystem provides the following interfaces to the AFU region:
- AXI4-Memory Mapped access to the HSSI CSR (to FIM)
- AXI4-Stream Ethernet Data Interface (from FIM)
- AXI4-Stream Ethernet Sideband Interface (from FIM)
- Ethernet Clock Interface (eth_clock) (from FIM)
The he-hssi uses the APF interface for HSSI CSR (MMIO) accesses. The AXI4-Stream Ethernet data and side band interface along with Ethernet clocks communicate directly to the he-hssi module in the AFU region through platform independent data structures provided by the PIM. Even if you implement a different MAC you typically can leverage these data structures defined in the hssi/inc/ofs_fim_eth_avst_if.sv file here without modification.
While the platform-independent interfaces in ofs_fim_eth_if.sv are convenient containers for passing data streams through the design hierarchy, both the MAC and AFU traffic generator require platform-specific data types. The payloads of the streams in ofs_fim_eth_if.sv are defined in platform-specific structures, with fields that are MAC-specific. In this 10GbE reference design, the payload datatypes are defined in the ipss/hssi/s10/includes/ofs_fim_eth_plat_if_pkg.sv file in the OFS D5005 FIM Github Branch. Implementers connecting a new MAC should generally edit only ofs_fim_eth_plat_if_pkg.sv when defining payloads.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1223-hssi-sideband-interface","title":"12.2.3 HSSI Sideband Interface","text":"The AXI4-Stream sideband interface has been been defined to allow for time sensitive status and flow control information. It does not have the optional tready signal and assumes the slave always accepts information. The Ethernet sideband interface varies widely across IPs and operating modes and device generations. In OFS Stratix 10 FIM, the Ethernet sideband signals are mapped to the AXI4-Stream interface and interface variations can be accommodated by customizing the tdata signals of the sideband interface in platform specific interface packages using the PIM.
As an example, please refer to ofs_fim_eth_sideband_tx_axis_if interface in the ipss/hssi/inc/ofs_fim_eth_if.sv found OFS D5005 FIM Github Branch.
The t_axis_eth_sideband_tx and t_axis_eth_sideband_rx structures are found in the ipss/hssi/s10/includes/ofs_fim_eth_plat_if_pkg.sv file OFS D5005 FIM Github Branch.
Platform specific details for the 10GbE example are from the ipss/hssi/s10/includes/ofs_fim_eth_plat_if_pkg.sv file OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1224-reconfiguration-interfaces","title":"12.2.4 Reconfiguration Interfaces","text":"The reconfiguration interface in the OFS EA design consists of abstracted and consolidated memory-mapped transceiver reconfiguration interfaces that are exposed to the HSSI CSRs. The reconfiguration interface directly exposes the address space of the MAC and PHY IPs in order to allow a software driver to perform dynamic reconfiguration of those IPs (i.e. read/write access to the Native PHY CSRs). Therefore, to use this interface you must be familiar with the CSR memory maps of the corresponding IP cores.
The table below summarizes all the ports associated with Reconfiguration Interfaces.
Name Width Domain Description i_xcvr_reconfig_cmd 2 i_reconfig_clk Command port used to specify a read or write access operation to a MAC/PHY CSRs on a selected CSR_interface i_xcvr_reconfig_addr 20 i_reconfig_clkCSR access address of MAC/PHY IP. Note that only the lower 16 bits are used, i_xcvr_reconfig_addr[15:0]. The remaining upper bits, i_xcvr_reconfig_addr[20:16], should be set to zero. i_xcvr_reconfig_writedata 32 i_reconfig_clk CSR data to be written on a write command. o_xcvr_reconfig_readdata 32 i_reconfig_clk CSR data that was read on a read command. o_xcvr_reconfig_ack 1 i_reconfig_clk Reconfiguration acknowledgement bit. Asserted when Reconfiguration Controller is finished performing an Avalon-MM read or write request (i.e. waitrequest is low). Deasserted when i_xcvr_reconfig_cmd is all-zero (i.e. command is invalidated). Note that the controller assumes a valid command when i_xcvr_reconfig_cmd is non-zero."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12241-reconfiguration-sequence","title":"12.2.4.1 Reconfiguration Sequence","text":"The diagram below explains the sequence of operation and handshaking between software and a memory-mapped dynamic reconfiguration interface.
Figure 12-2: Sequence of Operation and Handshaking between Software and a Memory-Mapped Dynamic Reconfiguration Interface.
(0) Idle state \u2013 command and address buses are cleared (all-zero). 1. Software sets a desired non-zero command and address to initiate reconfiguration.
a. Memory-Mapped Reconfiguration Controller (MM CTRL) converts the command and address to a single Avalon Memory Mapped read or write request and handles Avalon Memory Mapped protocol details.\n\nb. MM CTRL completes the Avalon Memory Mapped transaction when the `waitrequest` signal of a given Avalon Memory Mapped interface is deasserted.\n\nc. MM CTRL sets the reconfiguration acknowledgment bit and `readdata` (in case of a read command) back to the FME and waits for command and address ports to be cleared by software. \n1.\n
-
Meanwhile, software continuously polls the reconfiguration acknowledgment bit and waits for it to get asserted. Assertion of the acknowledgment bit confirms that the MM CTRL has completed the current Avalon Memory Mapped read/write request.
-
Software reads readdata from the HSSI_RCFG_DATA CSR that was returned by the MM CTRL (in case of a read command).
-
Software clears the command and address buses to communicate back to the MM CTRL that the operation is finished from the CPU\u2019s perspective.
a. MM CTRL gets cleared (all-zero) command and address signals from the HSSI_CSR. b. MM CTRL clears (deasserts) the reconfiguration acknowledgment bit back to the HSSI_CSR and is finished / back to idle state. 5. Meanwhile, software continuously polls the reconfiguration acknowledgment bit and waits for it to get deasserted. Deassertion of the acknowledgment bit confirms that the MM CTRL has completed its handshake and is now back to idle state.
NOTE
Reads and writes cannot be performed at the same time. Remember that when multiple CSRs are at the same address, a Read-Modify-Write operation may be required to change the desired CSR without changing the CSRs in the same address.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1225-hssi-control-and-status-register-csr-map","title":"12.2.5 HSSI Control and Status Register (CSR) Map","text":"The HSSI CSR Map structure is designed to scale according to IP capabilities.
- HSSI_DFH allows for identifying HSSI as an external FME feature.
- HSSI_CAPABILITY register exposes design capabilities and provides direction to SW/Driver. The fields num_channels, Num_channels_CSR interface, Num_CSR_interface indicate to the software how many CSR interfaces are exposed by the design and how to use mailbox registers (HSSI_RCFG_CMD and HSSI_RCFG_DATA). The number of mailbox registers(HSSI_RCFG_CMD and HSSI_RCFG_DATA) must scale to the number of CSR interfaces exposed by the design. This implementation facilitates flexibility in the design and reuse of the software stack. If you are modifying the HSSI interface, you must update these CSR fields according to HW configuration .
Example: If Num_CSR_interface=2 & Num_channels_CSR_interface=2 then channel(0,1) are behind CSR interface 0 handled by HSSI_RCFG_CMD0/DATA0 , channel (2,3) are behind CSR interface 1 handled by HSSI_RCFG_CMD1/DATA1 HSSI_CTRL, HSSI_STATUS0, HSSI_STATUS1 provide control and status information and can scale upto 8 channels. HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers are for the reconfiguration interface, and additional mailbox registers could be added depending on the number of CSR interfaces the design exposes.
The HSSI CSR Map can be found in the ipss/hssi/s10/hssi_ss_csr.xls file OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1226-hssi-host-exercisier-he-hssi","title":"12.2.6 HSSI Host Exercisier (HE-HSSI)","text":"HE-HSSI is an Ethernet AFU that handles client side ethernet traffic. The reference HE-HSSI has following features:
- HE-HSSI wraps the 10G Ethernet AFU that was provided in the OFS EA FIM with a wrapper that provides an E-tile compatible interface with OFS for Stratix 10 and Agilex FPGAs.
- Includes 10GbE traffic generator and checker (monitor)
- Provides pause signals to the HSSI subsystem for XON and XOFF generation
- It can generate traffic or incoming traffic that can be looped back into transmit path by enabling loopback mode, which will bypass traffic generator
- At the HE-HSSI interface boundary the Ethernet data interface is AXI4-Stream with 64-bit data at eth_clk clock
- An AXI4-Stream to Avalon-ST bridge converts Avalon-ST Ethernet traffic from the HE-HSSI traffic generator to AXI4-Stream traffic and AXI4-Stream RX data from the FIM to Avalon-ST for the HE-HSSI traffic checker. The data width for all the interfaces in this bridge is 64 bits at eth_clk clock.
- The Traffic generator and checker modules have a 64-bit data interface at eth_clk clock.
- The traffic generator supports the following modes: * Fixed length or Random Length * Incremental pattern or Random pattern
- The traffic checker does a 32-bit CRC check
- The CSR of this AFU is accessible through AXI4-Stream PCIe TLP interfac
- The PCIe TLP to CSR Interface Conversion module converts PCIe TLPs into simple CSR interface
- The CSR space of the traffic generator and checker modules are accessed in an indirect way using mailbox registers
- Though the default configuration for this reference HE-HSSI is 1x10GbE, it can be scaled up to eight 10G ethernet traffic generators and checkers in one HE-HSSI
- If used for more than one channel, each channel has a separate traffic generator and traffic checker with separate CSR space.
- Reads and Writes to individual traffic controller CSR spaces can be done by selecting that particular channel using channel select register.
The HE-HSSI Ethernet block diagram is below.
Figure 12-6: HE-HSSI Block Diagram Block Diagram
Figure 12-7: 10G Ethernet AFU Clock Domains
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#12261-he-hssi-csr-map","title":"12.2.6.1 HE-HSSI CSR Map","text":"The reference HSSI AFU contains the following registers and a similar arrangement of register space can be implemented for other usecase specific HSSI AFUs. * AFU DFH Register: Device feature header for the AFU (AFU_DFH) * AFU ID Registers: 128-bit UUID for the AFU which occupies two 64-bit registers (AFU_ID_L, AFU_ID_H) * Mailbox Registers: Command and Data register for traffic controller register access. It follows the standard access method defined for OFS. Access method and implementation is same as Reconfiguration Interface defined for the HSSI FIM. (TRAFFIC_CTRL_CMD, TRAFFIC_CTRL_DATA) * Channel Select Register: Channel select register for traffic controller mailbox access. It is used in cases where more than one channel is in the AFU, else it defaults to zero, meaning channel-0 is selected. (TRAFFIC_CTRL_PORT_SEL) * Scratchpad Register: Scratchpad register for CSR access checking. (AFU_SCRATCHPAD)
The CSR excel for the 10G HSSI reference AFU can be found ipss/hssi/s10/hssi_ss_csr.xls OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#123-hssi-software","title":"12.3 HSSI Software","text":"There are two pieces of software related to running the HSSI Subsystem and the HE-HSSI host exerciser: The Linux* dfl network driver and a user space application.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1231-hssi-linux-driver","title":"12.3.1 HSSI Linux Driver","text":"The HSSI subystem is exposed as a feature in the PCIe PF BAR0 region. It has a Device Feature Header (DFH) indicating the HSSI interface. The feature ID in the DFH causes the following driver to be instantiated for the HSSI interface: drivers/net/ethernet/intel/s10hssi.c Kernel Driver Branch
The primary functionality of the driver is to interact with the ethernet MAC and PHY through an indirect register access mailbox implemented by the HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers described above. To aid in RTL bringup, the driver provides a debugfs interface directly to the indirect register access mailbox. For each HSSI interface in the system there would be a directory with the following form containing two files, regaddr and regval: /sys/kernel/debug/dfl-fme.X.Y
To read a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then read the value as string out of regval file. To write a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then write the value as a C hex string to regval file.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1232-hssi-user-space-tool","title":"12.3.2 HSSI User Space Tool","text":"The HSSI user space application exports a control interface to the HSSI AFU's packet generator logic. Context-sensitive help is given by the --help option, doc/src/fpga_tools/hssi/hssi.md, OPAE SDK Branch.
$ hssi --help\n\n"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#124-user-guidelines","title":"12.4 User Guidelines","text":"You can either leverage Ethernet example designs from platform designer or use your own custom IP\u2019s. However below recommendations would help leverage the infrastructure of the OFS stack:\n* Follow the Ethernet-GBS interface standard, customize platform specific sideband and clock intefaces.\n* Follow the reconfiguration interface example and reuse the hssi_csr block by modifying the memory map (change address decoder map accordingly as well.)
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#13-partial-reconfiguration","title":"13 Partial Reconfiguration","text":"Partial Reconfiguration (PR) is an Altera FPGA technology that allows user to reconfigure parts of the FPGA device dynamically, while the remainder of the device continues to operate. In a non-partial reconfiguration flow, any change to the design requires full reprogramming of the entire configuration RAM (CRAM) arrays in the device. With partial reconfiguration, you can dynamically reprogram one or more CRAM frames. A partial reconfiguration design has a static region, and one or more PR regions, which can be modified to implement new logic. The portion of the CRAM on the chip to be reconfigured is contained within a PR region.\nFor the PR flow, the design should be partitioned into static region and reconfigurable region. The static region is the area of your FPGA that is not reconfigured without reprogramming the entire FPGA. An area of the chip that you plan to partially reconfigure is a PR region.
\nThe Port Gasket contains all the PR specific modules and logic, such as PR slot reset/freeze control, user clock, remote STP etc. For this reference example only one PR slot is supported.\nThe following figure depicts the high level view of the Port Gasket:
\nFigure 13-1 Partial Reconfiguration Gasket
\n\n\nThe isolation logic is provided on the output signals of the PR region to ensure they don\u2019t glitch and affect the interfacing logic in the Static Region (SR). The isolation logic is controlled by the PR Freeze logic during PR operation.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14-reliability-accessibility-serviceability-ras-and-error-handling","title":"14 Reliability, Accessibility, Serviceability (RAS) and Error Handling","text":"\n- Downstream AFU checker: Identifies AFU violations. For example, this checker flags violations of the interface specification.
\n- Upstream software or PCIe link checker: Identifies invalid traffic from PCIe that violates either FIM specifications or PCIe specifications. For example, this checker flags an application sending traffic if it violates the FIM specification or creates a PCIe link issue by causing completion timeout or malformed TLP.
\n- FIM - Checks for bugs in the FIM fabric.
\n
\nErrors reported by the checker are logged in either the FME error registers or Port error registers, or both, as shown in the table below. For more details on each of the registers, please refer to src/common/protocol_checker/protocol_checker_csr.xml file OFS FIM_COMMON Github Branch or the SystemVerilog file src/common/fme/xls/d5005/FME_CSR.xls found OFS FIM_COMMON Github Branch .
\nTable 14-1: Error Registers
\nMMIO Region\nArea\nRegister\nDescription\nFME\nCoreFIM\nFME_ERROR\nFME Error Status Register 0. Registers parity errors, underflow or overflow errors and access mismatches.\nFME\nCoreFIM\nFME_ERROR0_MASK\nFME Error Mask Register 0. Write a 0 to mask errors in the FME Error Status Register 0.\nFME\nExternal\nPCIE0_ERROR\nPCIe0 Error Status Register.\nFME\nExternal\nPCIE0_ERROR_MASK\nPCIe0 Error Mask Register 0. Write a 0 to mask errors in the PCIe0 Error Status Register 0.\nFME\nCoreFIM\nFME_FIRST_ERROR\nFirst FME Error Register.\nFME\nCoreFIM\nFME_NEXT_ERROR\nFME Next Error Register.\nFME\nCoreFIM\nRAS_NOFAT_ERR_STAT\nReliability/Accessibility/Serviceability (RAS) Non-Fatal Error Status Register.\nFME\nCoreFIM\nRAS_NOFAT_ERR_MASK\nRAS Non-Fatal Error Mask Register. Write 0 to mask error fields in RAS_NOFAT_ERR_STAT Register.\nFME\nCoreFIM\nRAS_CATFAT_ERR_STAT\nRAS Catastrophic and Fatal Errors Status Register.\nFME\nCoreFIM\nRAS_CATFAT_ERR_MASK\nRAS Catastrophic and Fatal Errors Mask Register. Write 0 to mask error fields in the RAS_CATFAT_ERR_STAT Register.\nFME\nCoreFIM\nRAS_ERROR_INJ\nRAS error Injection Register.\nPORT\nCoreFIM\nPORT_ERROR\nPort Error Status Register.\nPORT\nCoreFIM\nPORT_FIRST_ERROR\nPort First Error Register .\nPORT\nCoreFIM\nPORT_MALFORMED_REQ0\nPort Malformed Request Register 0. Provides malformed request header LSBs.\nPORT\nCoreFIM\nPORT_MALFORMED_REQ1\nPort Malformed Request Register 1. Provides malformed request header MSBs."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#141-fme-errors","title":"14.1 FME Errors","text":""},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1411-fme_error0","title":"14.1.1 FME_ERROR0","text":"The FME_ERROR0 register flags CoreFIM FME errors in the Global Error (GLBL_ERROR) private feature. The error bits in this register are sticky bits. You can only clear these bits through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in FME_ERROR0_MASK register masks the error.
\nTable 14-2: FME Error Types
\nError Type\nDescription\nFabric errors\nFIFO underflow/overflow condition in CoreFIM. These errors only occur if you have introduced bugs into the FIM or during very rare single event upset (SEU) or SEU-like events.\nInvalid port access\nA port can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the Port. If it finds a PF is trying to access a port that is mapped to a VF or vice-versa, an error will be reported.\nInvalid AFU access\nAn AFU can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the AFU associated with the Port. If it finds a PF is trying to access an AFU that is mapped to a VF or vice-versa, an error is reported and a fake response is sent back to the requester to avoid a completion timeout on the host."},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1412-pcie0_error","title":"14.1.2 PCIE0_ERROR","text":"The PCIe Avalon-ST to AXI4-Stream bridge monitors the PCIe link for errors and logs any such errors in the PCIE0_ERROR register (in PCIE0 feature region) and PCIE0_ERROR register in the GLBL_ERR private feature. The error bits in the PCIE0_ERROR register are sticky bits that you can only clear through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in PCIE0_ERROR0_MASK masks the error.
\nIf you have other external FME features, you can add similar _ERROR registers to this space. Please refer to the following spreadsheet, bbs/csr/stratix10/pac_d5005/fme_csr_pcie.xls, OFS D5005 FIM Github Branch or the SystemVerilog file the SystemVerilog file src/common/fme/xls/d5005/FME_CSR.xls found OFS FIM_COMMON Github Branch for more details on this register. \n
NOTE
\nThe PCIE0_ERROR register is located in both the Global Error external feature memory space and a separate PCIe external feature memory space. OPAE software supports the PCIe external feature memory space beginning at offset 0x40000 for OFS EA and going forward. PCIe registers beginning at 0x4000 in the Global Error external feature memory space is there for backward compatibility to the Intel FPGA PAC D5005 v2.0.1 Acceleration Stack.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1413-fme_first_error-fme_next_error","title":"14.1.3 FME_FIRST_ERROR, FME_NEXT_ERROR","text":"The FME_FIRST_ERROR register flags which of the FME error reporting registers, such as FME_ERROR0, PCIE0_ERROR0, has reported the first error occurrence. The error fields of the first error register are then continuously logged into the FME_FIRST_ERROR register until a system reset or software clears all the errors in that first error register.\nLikewise, the FME_NEXT_ERROR indicates which of the FME error reporting registers (except the first error register) has reported the next occurrence of error after the first error register. The error fields of the next error register are continuously logged into the FME_NEXT_ERROR register until a system reset or software clears all the errors in the second error register.
\nPlease refer tobbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls, OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv, OFS D5005 FIM Github Branch
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1414-reliability-accessibility-serviceability-ras-error-status","title":"14.1.4 Reliability, Accessibility, Serviceability (RAS) Error Status","text":"The RAS feature in CoreFIM labels errors as non-fatal, fatal or catastrophic based on their impact to the system. \n* A non-fatal error usually originates from software or an AFU. With a non-fatal error, the user application may still be able to recover from the error by performing a soft reset on the AFU, fixing the user application software if necessary, and clearing the error. On the other hand, a fatal or catastrophic error is non-recoverable and requires the platform to be reset.\n* Non-fatal errors are logged in the RAS_NOFAT_ERR_STAT register and fatal or catastrophic errors are logged in the RAS_CATFAT_ERR_STAT register.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14141-non-fatal-errors","title":"14.1.4.1 Non-Fatal Errors","text":"The RAS_NOFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It logs the high-level status of non-fatal errors in the hardware. Unlike the error bits in the PCIE0_ERROR and FME_ERROR0 registers which are RW1C (software can write a 1 to clear the error), the error bits in this register are read-only and can only be cleared by system reset. Software has an option to mask the error using RAS_NOFAT_ERR_MASK.\nPlease refer to bbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls, OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv, OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14142-catastrophic-fatal-errors","title":"14.1.4.2 Catastrophic & Fatal Errors","text":"The RAS_CATFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It captures the high-level status of errors that can only be recovered with a system reset. Therefore, the error bits in the RAS_CATFAT_ERR_STAT register are read-only and can only be cleared by system reset or masked through RAS_CATFAT_ERR_MASK.\nPlease refer to bbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls, OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1415-ras-error-injection","title":"14.1.5 RAS Error Injection","text":"For software testing purposes, you can inject non-fatal, fatal and catastrophic errors into the platform through the RAS_ERROR_INJ register. These errors are reported in the RAS_CATFAT_ERR_STAT and RAS_NOFAT_ERR_STAT registers.\nPlease refer to bbs/csr/stratix10/pac_d5005/s10_iofs_csr_map_update_fme_ral_.xls OFS D5005 FIM Github Branch for individual register field descriptions or the SystemVerilog file: src/fme/fme_csr.sv OFS D5005 FIM Github Branch.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1416-fme-error-interrupts","title":"14.1.6 FME Error Interrupts","text":"In an event of an FME error, the MSI-X module in the FIM generates an interrupt so the host can decide on the next course of action. The FIM does not stall upstream and downstream traffic after the FME error. However, a port error triggered by invalid request from a user AFU stalls all the traffic going from AFU to PCIe.\nThe interrupt capability is discoverable by querying the NumbSuppInterrupt field of the PORT_CAPABILITY register in the Port private feature. The MSI-X vector number is recorded in the InterruptVectorNumber field of the GLBL_ERROR_CAPABILITY register of the Global Error external feature.
\nAn FME error interrupt is generated in response to the FME_ERROR0, PCIE0_ERROR0, RAS_NOFAT_ERR_STAT or RAS_CATFAT_ERR_STAT registers recording a new, unmasked, error per the rules defined by CoreFIM interrupt feature.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14161-msi-x-masking-pending-bit-array-pba-clearing","title":"14.1.6.1 MSI-X Masking & Pending Bit Array (PBA) Clearing","text":"If the MSI-X vector corresponding to the FME error interrupt is masked, events are recorded in the PBA array. Clearing the FME error status registers clears the corresponding MSI-X PBA entries. If only some events are cleared, the normal interrupt triggering rules apply and a new pending interrupt is registered.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1417-fme-error-handling","title":"14.1.7 FME Error Handling","text":"When the host receives an FME error interrupt, it must take the recommended actions described below to bring the system back to its normal operation.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14171-catastrophicfatal-error","title":"14.1.7.1 Catastrophic/Fatal Error","text":"A system reset is mandatory for any catastrophic or fatal error.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#14172-non-fatal-error","title":"14.1.7.2 Non-Fatal Error","text":"When software receives a non-fatal error interrupt which does not require a system reset, it can take the following steps to clear the error after software handles the error:\n1. Set the *_ERROR_MASK register to all 1\u2019s to mask all errors\n2. Clear the *_FIRST_ERROR register\n3. Clear the *_ERROR register\n4. Set *_ERROR_MASK register to all 0\u2019s to enable all errors
\n\n- Result: The *_ERROR & *_FIRST_ERROR registers begin capturing new errors.
\n
\nNOTE
\nA system reset can only clear RAS Error status registers.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#142-mmio-requests","title":"14.2 MMIO Requests","text":"The FIM is designed to gracefully handle MMIO request scenarios.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1421-unsupported-functions-and-bars","title":"14.2.1 Unsupported Functions and BARs","text":"The OFS FIM EA has only one PCIe link and all MMIO requests from the host are sent through this link. The PCIe hard IP in the FIM guarantees that only TLP packets for the functions and BARs supported by the FIM (as configured in PCIe HIP IP instantiation) are sent to the FIM on the PCIe Avalon Streaming interface.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1422-mmio-request-decoding","title":"14.2.2 MMIO Request Decoding","text":"The packet router and memory decoder in the FIM ensure that only legal MMIO requests are forwarded to the targeted MMIO region. Full address and BAR decoding is done both in the packet router and the memory decoder to ensure the requests are forwarded to the designated CSR region as defined in the MMIO Regions chapter. Any unsolicited/illegal MMIO request is dropped, and an error is reported back to host through the FME error register.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1423-unused-fmeport-csr-regions","title":"14.2.3 Unused FME/Port CSR Regions","text":"All the CSR slaves in FIM which are mapped to the FME or Port CSR regions must always respond to MMIO read requests targeting its associated CSR region. A CSR slave must return all 0s for MMIO reads to its unused CSR region such as a reserved space or a region where no CSR register is implemented for the address.\nThe FIM ensures MMIO reads to FME or Port CSR regions that are not mapped to any CSR slave always gets a response of all 0s. The memory decoder module and fake responder module in the FIM provide this guaranteed response.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1424-unsupported-mmio-request","title":"14.2.4 Unsupported MMIO Request","text":"Any MMIO request targeting FME or Port CSR regions with a length or address alignment that are not supported by the FIM is dropped, and an error is logged in PCIE0_ERROR register. The MMIO checker module in the FIM guarantees this response. When an unsupported MMIO read request to the FIM CSR region is detected, the FIM sends back a CPL (completion without data) with error status (UR) back to host.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1425-afu-access-violation","title":"14.2.5 AFU Access Violation","text":"AFU access violations refer to the scenarios where a PF is attempting to access the MMIO region of an AFU bound to a VF (virtualization enabled), or when a VF is trying to access the MMIO region of an AFU bound to a PF (virtualization disabled). When such a violation is detected, the FIM drops the request and logs the error in the FME_ERROR0 register. If the request is an MMIO read request, the FIM returns a fake response to the host.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#1426-afu-mmio-response-timeout","title":"14.2.6 AFU MMIO Response Timeout","text":"An AFU MMIO Response timeout functions in the same manner described in the MMIO Response Timeout section.
"},{"location":"hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_d5005/#15-design-guidance","title":"15 Design Guidance","text":"The OFS FIM is designed with configurability and scalability in mind. At a high level, these are the necessary steps for a user to customize the design. Please refer to the FPGA Interface Manager Developer Guide: Open FPGA Stack for Stratix 10
\nTable 15-1 Features
\nStep\n Description\n Comments \n 1\n Re-configure PCIe HIP for additional VFs (if necessary)\n * PF0 is mandatory for running OPAE software* Only modification of the VFs (added or subtracted) by the user is recommended.\n\u2022 Default configuration supports 1 PF and 3 VFs.\n \n 2\n Update AXI4-Stram PF/VF MUX-DEMUX configuration (if necessary)\n\n * The PF/VF MUX-DEMUX is parameterized for flexibility. You can change, add or delete PF or VF functions by updating the top_cfg_pkg.sv file.\n* You also have the option of keeping the default configuration and tying off the unused VFs if needed.\n\n 3\n Update top level and AFU level as necessary\n\n * If you integrating additional external interfaces, make the edits at the top level (iofs_top.sv) and propagate the interface down to the AFU level (afu_top.sv)\n 4\n Add user implemented function(s) in AFU\n * All of your implemented functions must have the required AXI4-Stream interface for both the data path and the MMIO control path to CSRs. * All CSRs in the user implemented function must have the required DFH layout. * See host exerciser CSRs for reference.\n \n \n\n\nFor more information on modifying the FIM, refer to the [Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs](https://ofs.github.io/ofs-2024.2-1/hw/d5005/dev_guides/fim_dev/ug_dev_fim_ofs_d5005/).\n\n\n\n## Notices & Disclaimers\n\nIntel\u00ae technologies may require enabled hardware, software or service activation.\nNo product or component can be absolutely secure. \nPerformance varies by use, configuration and other factors.\nYour costs and results may vary. \nYou may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.\nNo license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document.\nThe products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.\nIntel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.\nYou are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \n\u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others. \n\nOpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122. \n\n\n\n\n\n\n\n\n*[AFU]: Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region\n*[BBB]: Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID.\n*[BKC]: Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against.\n*[BMC]: Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors.\n*[DFL]: Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\n*[FIM]: FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring.\n*[FME]: FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform.\n*[HEM]: Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc.\n*[JTAG]: Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology.\n*[OFS]: Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs.\n*[OPAE SDK]: Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE.\n*[PIM]: Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols.\n*[PR]: Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page.\n*[RSU]: Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration.\n*[UVM]: Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework.\n*[TB]: Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output.\n*[Intel VT-d]: Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization.\n*[SR-IOV]: Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance.\n*[MMIO]: Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators.\n*[VFIO]: Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace.\n*[IOCTL]: Input/Output Control, System calls used to manipulate underlying device parameters of special files.\n*[AER]: Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting.\n*[PAC]: Programmable Acceleration Card: FPGA based Accelerator card"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/","title":"Platform Evaluation Script: Open FPGA Stack for Stratix 10 FPGA","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#1-overview","title":"1 Overview","text":""},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the checkout and evaluation of an Intel\u00ae FPGA PAC D5005 development platform using Open FPGA Stack (OFS). After reviewing the document, you will be able to:
- Set-up and modify the script to your environment
- Compile and simulate an OFS reference design
- Run hardware and software tests to evaluate the complete OFS flow
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#12-table-software-version-summary","title":"1.2 Table : Software Version Summary","text":"Component Version Description FPGA Platform Intel\u00ae FPGA PAC D5005 Stratix 10 platform you can use for your custom board development OFS FIM Source Code Branch: ofs-d5005, Tag: ofs-2024.1-1 OFS Shell RTL for Stratix 10 FPGA (targeting Intel\u00ae FPGA PAC D5005) OFS FIM Common Branch: ofs-2024.1-1, Tag: ofs-2024.1-1 Common RTL across all OFS-based platforms AFU Examples Branch: examples-afu , Tag: ofs-examples-ofs-2024.1-1 Tutorials and simple examples for the Accelerator Functional Unit region (workload region) OPAE SDK Branch: 2.12.0-5, Tag: 2.12.0-5 Open Programmable Acceleration Engine Software Development Kit Kernel Drivers Branch: ofs-2024.1-6.1-2, Tag: ofs-2024.1-6.1-2 OFS specific kernel drivers OPAE Simulation Branch: opae-sim, Tag: 2.12.0-5 Accelerator Simulation Environment for hardware/software co-simulation of your AFU (workload) Quartus Prime Pro Edition Design Software 23.4 Quartus\u00ae Prime Pro Edition Linux Software tool for FPGA Development Operating System RHEL 8.6 Operating system on which this script has been tested A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae FPGA PAC D5005 can be found on the OFS 2024.1-1 official release drop on GitHub.
Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#2-introduction-to-ofs-evaluation-script","title":"2 Introduction to OFS Evaluation Script","text":"By following the setup steps and using the OFS evaluation script you can quickly evaluate many features that the OFS framework provides and also leverage this script for your own development.
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#21-pre-requisites","title":"2.1 Pre-Requisites","text":"This script uses the following set of software tools which should be installed using the directory structure below. Tool versions can vary.
- Quartus\u00ae Prime Pro Software : The software can be downloaded here. For details on installation of required patches and quartus installation, refer to section 4.1 of the Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
- Synopsys\u00ae VCS Simulator : The simulator can be downloaded in the Synopsys page here
- Siemens\u00ae Questa\u00ae Simulator : The simulator can be downloaded in the Siemens page here
Figure 2-1 Folder Hierarchy for Software Tools
- Download tar(scripts_ofs-2024.1-1_external.tar.gz) from the assets tab of the release page
- Untar to folder using the following command
tar -xvf scripts_ofs-2024.1-1_external.tar.gz\n````\n\n3. The folder structure containing the clone script (ofs-clone_repositories.sh) and evaluation script (ofs_d5005_eval.sh) is as shown in the figure 2-2\n\n**Figure 2-2 Directory Structure for the clone script**\n\n``` sh\n## ofs_scripts\n## -> host_chan_mmio.stp\n## -> ofs_d5005_eval.sh\n## -> README_ofs_d5005_eval.txt\n## ofs-clone_repositories.sh\n
- Open the clone script ofs-clone_repositories.sh in any text editor
- Please enter the location of your proxy server to allow access to external internet to build software packages. (lines 6-8)
Note: Failing to add proxy server will prevent cloning of repositories and the user will be unable to build the OFS framework.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
-
Please enter the license file locations (lines 11-13) for the following tool variables
export LM_LICENSE_FILE=<user_license>\n export DW_LICENSE_FILE=<user_license>\n export SNPSLMD_LICENSE_FILE=<user_license>\n
-
Add the name of the directory where you want the platform repositories to be cloned (line 19)
export OFS_RELEASE=ofs-2024.1-1\n
-
After setting the required variables, source the clone script based on which platform you want to test
source ofs-clone_repositories.sh\n
- The script ofs-clone_repositories.sh has different platforms to choose from the menu to clone as shown in the figure 2-3.
Figure 2-3 Directory Structure for OFS Project
-
Once the repositories are cloned, navigate to the directory where you cloned
cd ofs-2024.1-1\n
-
After cloning, the OFS repositories cloned will look as per Figure 2-4.
Figure 2-4 Directory Structure for OFS Project
## ofs-2024.1-1\n## d5005(OFS platform)\n## -> examples-afu\n## -> ofs-d5005\n## -> oneapi-asp\n## -> oneAPI-samples\n## -> opae-sim\n## -> opae-sdk\n## -> linux-dfl\n## -> ofs_d5005_eval.sh\n## -> README_ofs_d5005_eval.txt\n## -> host_chan_mmio.stp\n
- Once the repositories are cloned, in the platform specific evaluation script, for e.g., in ofs_d5005_eval.sh, please follow the instructions below which explains which line numbers to change to adapt this script to the user environment.
a) Set-Up Proxy Server (lines 67-69)
Please enter the location of your proxy server to allow access to external internet to build software packages.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
b) Tools Location (line 87, 88, 89, 90)
Set Location of Quartus, Synopsys, Questasim and oneAPI Tools
export QUARTUS_TOOLS_LOCATION=/home\nexport SYNOPSYS_TOOLS_LOCATION=/home\nexport QUESTASIM_TOOLS_LOCATION=/home\nexport ONEAPI_TOOLS_LOCATION=/opt\n
In the example above /home is used as the base location of Quartus, Synopsys and Questasim tools, /opt is used for the oneAPI tools
c) PCIe (Bus Number)
The Bus number must be entered by the user after installing the hardware in the chosen server, in the example below \"b1\" is the Bus Number for a single card as defined in the evaluation script.
export OFS_CARD0_BUS_NUMBER=b1\n
The evaluation script uses the bus number as an identifier to interrogate the card. The command below will identify the accelerator card plugged into a server.
lspci | grep acc\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\nb1:00.1 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.2 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\nb1:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
The result identifies the card as being assigned \"b1\" as the bus number so the entry in the script changes to
export OFS_CARD0_BUS_NUMBER=b1\n
The user can also run the following command on the ofs_d5005_eval.sh script to automatically change the bus number to b1 in the ofs_d5005_eval.sh script.
grep -rli 'b1' * | xargs -i@ sed -i 'b1' @
if the bus number is 85 for example
85:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\n85:00.1 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.2 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\n85:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
the command to change to 85 in the evaluation script would be
grep -rli 'b1' * | xargs -i@ sed -i '85' @
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#ofs-platform-example-n6000-n6001-fseries-dk-iseries-dk-custom_board-choice-line-173","title":"OFS Platform (Example:= n6000, n6001, fseries-dk, iseries-dk, custom_board) choice (line 173)","text":"The script is designed to accommodate many OFS platform choices eg, n6000, n6001, fseries-dk, iseries-dk or a custom_board of your choice. By default the script defaults to n6001 as shown below and is set by a variable on line 173 of the ofs_d5005_eval.sh script.
export OFS_PLATFORM=n6001\n
but the user can switch platform by changing the OFS_PLATFORM variable, so for example if the user wants to switch to the i-series development kit the command would be
export OFS_PLATFORM=iseries-dk\n
The ofs_d5005_eval.sh script has now been modified to the server set-up and the user can proceed to build, compile and simulate the OFS stack
Figure 3-1 ofs_d5005_eval.sh Evaluation Menu
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#311-tools-menu","title":"3.1.1 TOOLS MENU","text":"By selecting \"List of Documentation for ADP Intel\u00ae FPGA PAC D5005 Project,\" a list of links to the latest OFS documentation appears. Note that these links will take you to documentation for the most recent release which may not correspond to the release version you are evaluating. To find the documentation specific to your release, ensure you clone the OFS tag that corresponds to your OFS version.
By selecting \"Check Versions of Operating System and Quartus Premier Design Suite\", the tool verifies correct Operating System, Quartus version, kernel parameters, license files and paths to installed software tools.
Menu Option Example Output 1 - List of Documentation for ADP d5005 Project Open FPGA Stack Overview Guides you through the setup and build steps to evaluate the OFS solution https://ofs.github.io 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS) Checking Linux release Linux version 6.1.78-dfl Checking RedHat release Red Hat Enterprise Linux release RHEL 8.6 Checking Ubuntu release cat: /etc/lsb-release: No such file or directory Checking Kernel parameters BOOT_IMAGE=(hd0,msdos1)/vmlinuz-6.1.78-dfl-2023.4-1 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 Checking Licenses LM_LICENSE_FILE is set to port@socket number:port@socket number DW_LICENSE_FILE is set to port@socket number:port@socket number SNPSLMD_LICENSE_FILE is set to port@socket number:port@socket number Checking Tool versions QUARTUS_HOME is set to /home/intelFPGA_pro/23.4/quartus QUARTUS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus IMPORT_IP_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../ip QSYS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../qsys/bin Checking QPDS Patches Quartus Prime Shell Version 23.4"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#312-hardware-menu","title":"3.1.2 HARDWARE MENU","text":"Identifies card by PCIe number, checks power, temperature and current firmware configuration.
Menu Option Example Output 3 - Identify Acceleration Development Platform (ADP) d5005 Hardware via PCIe PCIe card detected as 86:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) Host Server is connected to SINGLE card configuration 4 - Identify the Board Management Controller (BMC) Version and check BMC sensors Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** BMC SENSORS ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e 5 - Identify the FPGA Management Engine (FME) Version Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** FME ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4eBoot Page : user 6 - Check Board Power and Temperature Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** POWER ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e( 1) VCCERAM Voltage : 0.90 Voltsetc ......................Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** TEMP ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e( 1) VCCT Temperature : 57.00 Celsiusetc ...................... 7 - Check Accelerator Port status //****** PORT ******// Object Id : 0xEF00000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00 8 - Check MAC and PHY status Intel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** MAC ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4eMAC address : 64:4c:36:f:44:1fIntel FPGA Programmable Acceleration Card d5005Board Management Controller, MAX10 NIOS FW version: 2.0.8Board Management Controller, MAX10 Build version: 2.0.8//****** PHY ******//Object Id : 0xF000000PCIe s:b:d.f : 0000:86:00.0Vendor Id : 0x8086Device Id : 0xBCCESubVendor Id : 0x8086SubDevice Id : 0x138DSocket Id : 0x00Ports Num : 01Bitstream Id : 288511860124977321Bitstream Version : 4.0.1Pr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#313-fimpr-build-menu","title":"3.1.3 FIM/PR BUILD MENU","text":"Builds FIM, Partial Reconfiguration Region and Remote Signal Tap
Menu Option Description 9 - Check ADP software versions for ADP Intel\u00ae FPGA PAC D5005 Project OFS_ROOTDIR is set to /home/user_area/ofs-2024.1-1/ofs-d5005OPAE_SDK_REPO_BRANCH is set to release/2.12.0-5 OPAE_SDK_ROOT is set to /home/user_area/ofs-2024.1-1/ofs-d5005/../opae-sdk LD_LIBRARY_PATH is set to /home/user_area/ofs-2024.1-1/ofs-d5005/../opae-sdk/lib64: 10 - Build FIM for Intel\u00ae FPGA PAC D5005 Hardware This option builds the FIM based on the setting for the $ADP_PLATFORM, $FIM_SHELL environment variable. Check these variables in the following file ofs_d5005_eval.sh 11 - Check FIM Identification of FIM for Intel\u00ae FPGA PAC D5005 Hardware The FIM is identified by the following file fme-ifc-id.txt located at $OFS_ROOTDIR/$FIM_WORKDIR/syn/syn_top/ 12 - Build Partial Reconfiguration Tree for Intel\u00ae FPGA PAC D5005 Hardware This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the OneAPI build flow 13 - Build Base FIM Identification(ID) into PR Build Tree template This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and OneAPI workloads 14 - Build Partial Reconfiguration Tree for Intel\u00ae FPGA PAC D5005 Hardware with Remote Signal Tap This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development and also for the OneAPI build flow and for the Remote Signal Tap flow 15 - Build Base FIM Identification(ID) into PR Build Tree template with Remote Signal Tap This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree for Remote Signal Tap to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU and OneAPI workloads"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#314-hardware-programmingdiagnostic-menu","title":"3.1.4 HARDWARE PROGRAMMING/DIAGNOSTIC MENU","text":"The following submenu allows you to: * Program and check flash * Perform a remote system update (RSU) of the FPGA image into the FPGA * Bind virtual functions to VFIO PCIe driver * Run host exerciser (HE) commands such as loopback to test interfaces VFIO PCI driver binding * Read the control and status registers (CSRs) for bound modules that are part of the OFS reference design.
Menu Option Description 16 - Program BMC Image into Intel\u00ae FPGA PAC D5005 Hardware The user must place a new BMC flash file in the following directory $OFS_ROOTDIR/bmc_flash_files. Once the user executes this option a new BMC image will be programmed. A remote system upgrade command is initiated to store the new BMC image 17 - Check Boot Area Flash Image from Intel\u00ae FPGA PAC D5005 Hardware This option checks which location area in FLASH the image will boot from, the default is user1 Boot Page : user1 18 - Program FIM Image into user1 area for Intel\u00ae FPGA PAC D5005 Hardware This option programs the FIM image \"d5005_page1_unsigned.bin\" into user1 area in flash 19 - Initiate Remote System Upgrade (RSU) from user1 Flash Image into Intel\u00ae FPGA PAC D5005 Hardware This option initiates a Remote System Upgrade and soft reboots the server and re-scans the PCIe bus for the new image to be loaded 2022-12-13 07:31:33,244 - [[pci_address(0000:86:00.0), pci_id(0x8086, 0xbcce, 0x8086, 0x138d)]] performing RSU operation 2022-12-13 07:31:33,249 - [[pci_address(0000:85:00.0), pci_id(0x8086, 0x2030, 0x1590, 0x00ea)]] removing device from PCIe bus 2022-12-13 07:31:34,333 - waiting 10.0 seconds for boot 2022-12-13 07:31:44,344 - rescanning PCIe bus: /sys/devices/pci0000:85/pci_bus/0000:85 2022-12-13 07:31:44,377 - RSU operation complete 20 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 21 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 22 - Create Virtual Functions (VF) and bind driver to vfio-pci Intel\u00ae FPGA PAC D5005 Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 20 to check that the new drivers are bound 23 - Run HE-LB Test This option runs 5 tests 1) checks and generates traffic with the intention of exercising the path from the AFU to the Host at full bandwidth 2) run a loopback throughput test using one cacheline per request 3) run a loopback read test using four cachelines per request 4) run a loopback write test using four cachelines per request 5) run a loopback throughput test using four cachelines per request 24 - Run HE-MEM Test This option runs 2 tests 1) Checking and generating traffic with the intention of exercising the path from FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host 2) run a loopback throughput test using one cacheline per request 25 - Run HE-HSSI Test This option runs 1 test HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic 1) Send traffic through the 10G AFU 26 - Read from CSR (Command and Status Registers) for Intel\u00ae FPGA PAC D5005 Hardware This option reads from the following CSR's HE-LB Command and Status Register Default Definitions HE-MEM Command and Status Register Default Definitions HE-HSSI Command and Status Register Default Definitions"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#315-hardware-afu-testing-menu","title":"3.1.5 HARDWARE AFU TESTING MENU","text":"This submenu tests partial reconfiguration by building and loading an memory-mapped I/O example AFU/workload, executes software from host, and tests remote signal tap.
Menu Option Description 27 - Build and Compile host_chan_mmio example This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB_EXTERNAL/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) binary file ready for hardware programming 28 - Execute host_chan_mmio example This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test 29 - Modify host_chan_mmio example to insert Remote Signal Tap This option inserts a pre-defined host_chan_mmio.stp Signal Tap file into the OFS code to allow a user to debug the host_chan_mmio AFU example 30 - Build and Compile host_chan_mmio example with Remote Signal Tap This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB_EXTERNAL/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) binary file ready for hardware programming with Remote Signal tap enabled 31 - Execute host_chan_mmio example with Remote Signal Tap This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test. The user must open the Signal Tap window when running the host code to see the transactions in the Signal Tap window"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#316-hardware-afu-bbb-testing-menu","title":"3.1.6 HARDWARE AFU BBB TESTING MENU","text":"This submenu tests partial reconfiguration using a hello_world example AFU/workload, executes sw from the host
Menu Option Description 32 - Build and Compile hello_world example This option builds the hello_ world example from the following repo $FPGA_BBB_CCI_SRC/tutorial/afu_types/01_pim_ifc/$AFU_BBB_TEST_NAME, where AFU_BBB_NAME=hello_world. This produces a GBS (Green Bit Stream) file ready for hardware programming 33 - Execute hello_world example This option builds the host code for hello_world example and programs the GBS file and then executes the test"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#317-adp-oneapi-project-menu","title":"3.1.7 ADP ONEAPI PROJECT MENU","text":"Builds OneAPI kernel, executes the software from host and runs diagnostic tests
Menu Option Result 34 - Check oneAPI software versions for Intel\u00ae FPGA PAC D5005 Project This option checks the setup of the oneAPI software and adds the relevant oneAPI environment variables to the terminal. This option also informs the user to match the oneAPI software version to the oneAPI-samples version 35 - Build and clone shim libraries required by oneAPI host This option builds the oneAPI directory structure 36 - Install OneAPI Host Driver This option Installs the oneAPI Host driver at the following location /opt/Intel/OpenCLFPGA/oneAPI/Boards/, and requires sudo permission 37 - Uninstall One API Host Driver This option Uninstall's the oneAPI Host driver, and requires sudo permissions 38 - Diagnose oneAPI Hardware This option Checks ICD (Intel Client Driver) and FCD (FPGA Client Driver), oneAPI library locations and detects whether oneAPI BSP is loaded into the FPGA 39 - Build oneAPI BSP ofs-d5005 Default Kernel (hello_world) This option Builds the oneAPI BSP using hello_world kernel 40 - Build oneAPI MakeFile Environment This option Builds the oneAPI environment using a Makefile for kernel insertion 41 - Compile oneAPI Sample Application (board_test) for Emulation This option compiles the board_test kernel for Emulation 42 - Run oneAPI Sample Application (board_test) for Emulation This option executes the board_test kernel for Emulation 43 - Generate oneAPI Optimization report for (board_test) This option generates an optimization report for the board_test kernel 44 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 45 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 46 - Create Virtual Function (VF) and bind driver to vfio-pci Intel\u00ae FPGA PAC D5005 Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 45 to check that the new drivers are bound 47 - Program oneAPI BSP ofs-d5005 Default Kernel (hello_world) This option programs the FPGA with a aocx file based on the hello_world kernel 48 - Compile oneAPI Sample Application (board_test) for Hardware This option compiles the board_test kernel for Hardware 49 - Run oneAPI Sample Application (board_test) for Hardware This option builds the host code for board_test kernel and executes the program running through kernel and host bandwidth tests"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#318-unit-test-project-menu","title":"3.1.8 UNIT TEST PROJECT MENU","text":"Builds, compiles and runs standalone simulation block tests. More unit test examples are found at the following location ofs-d5005/sim/unit_test
Menu Option Result 50 - Generate Simulation files for Unit Test This option builds the simulation file set for running a unit test simulation 51 - Simulate Unit Test dfh_walker and log waveform This option runs the dfh_walker based on the environment variable \"UNIT_TEST_NAME=dfh_walker\" in the evaluation script. A user can change the test being run by modifying this variable"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#319-adp-uvm-project-menu","title":"3.1.9 ADP UVM PROJECT MENU","text":"Builds, compiles and runs full chip simulation tests. The user should execute the options sequentially ie 52, 53, 54 and 55
Menu Option Description 52 - Check UVM software versions for Intel\u00ae FPGA PAC D5005 Project DESIGNWARE_HOME is set to /home/synopsys/vip_common/vip_Q-2020.03A UVM_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm VCS_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel VERDIR is set to /home/user_area/ofs-2024.1-1/ofs-d5005/verification VIPDIR is set to /home/user_area/ofs-2024.1-1/ofs-d5005/verification 53 - Compile UVM IP This option compiles the UVM IP 54 - Compile UVM RTL and Testbench This option compiles the UVM RTL and Testbench 55 - Simulate UVM ofs_mmio_test and log waveform This option runs the dfh_walking test based on the environment variable \"UVM_TEST_NAME=dfh_walking_test\" in the evaluation script. A user can change the test being run by modifying this variable 56 - Simulate all UVM test cases (Regression Mode) This option runs the Intel\u00ae FPGA PAC D5005 regression mode, cycling through all UVM tests defined in /ofs-d5005/verification/tests/test_pkg.svh file"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#3110-adp-build-all-project-menu","title":"3.1.10 ADP BUILD ALL PROJECT MENU","text":"Builds the complete OFS flow, good for regression testing and overnight builds
For this menu, a user can run a sequence of tests (compilation, build and simulation) and executes them sequentially. After the script is successfully executed, a set of binary files is produced which a you can use to evaluate your hardware. Log files are also produced which checks whether the tests passed.
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 57 from the main menu the script will execute 23 tests ie (main menu options 2, 9, 10, 11, 12, 13, 14, 15, 27, 29, 30, 32, 34, 35, 39, 40, 48, 50, 51, 52, 53, 54 and 55. These 23 menu options are chosen to build the complete OFS flow covering build, compile and simulation.
Menu Option Result 57 - Build and Simulate Complete Intel\u00ae FPGA PAC D5005 Project Generating Log File with date and timestamp Log file written to /home/user_area/ofs-2024.1-1/log_files/d5005_log_2022_11_10-093649/ofs-d5005_eval.log Definition of Multi-Test Set-up
Menu Option 57 above in the evaluation script can be refined to tailor the number of tests the users runs. The set-up is principally defined by the variable below
MULTI_TEST[A,B]=C
where
A= Total Number of menu options in script B= Can be changed to a number to select the test order C= Menu Option in Script
Example 1 MULTI_TEST[57,0]=2
A= 57 is the total number of options in the script B= 0 indicates that this is the first test to be run in the script C= Menu option in Script ie 2- List of Documentation for ADP Intel\u00ae FPGA PAC D5005 Project
Example 2 MULTI_TEST[57,0]=2 MULTI_TEST[57,1]=9
In the example above two tests are run in order ie 0, and 1 and the following menu options are executed ie 2- List of Documentation for ADP Intel\u00ae FPGA PAC D5005 Project and 9 - Check ADP software versions for ADP Intel\u00ae FPGA PAC D5005 Project
The user can also modify the build time by de-selecting options they do not wish to use, see below for a couple of use-case scenarios.
Default User Case
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 57 from the main menu the script will execute 23 tests ie (main menu options 2, 9, 10, 11, 12, 13, 14, 15, 27, 29, 30, 32, 34, 35, 39, 40, 48, 50, 51, 52, 53, 54 and 55. All other tests with an \"X\" indicates do not run that test
User Case for ADP FIM/PR BUILD MENU
In the example below when the user selects option 57 from the main menu the script will only run options from the ADP FIM/PR BUILD MENU (7 options, main menu options 9, 10, 11, 12, 13, 14 and 15). All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#4-common-test-scenarios","title":"4 Common Test Scenarios","text":"This section will describe the most common compile build scenarios if a user wanted to evaluate an acceleration card on their server. The Pre-requisite column indicates the menu commands that must be run before executing the test eg To run Test 5 then a user needs to have run option 10, 12 and 13 before running options 20, 21, 22, 27 and 28.
Test Test Scenario Pre-Requisite Menu Option Menu Option Test 1 FIM Build - 10 Test 2 Partial Reconfiguration Build 10 12 Test 3 Program FIM and perform Remote System Upgrade 10 18, 19 Test 4 Bind PF and VF to vfio-pci drivers - 20, 21, 22 Test 5 Build, compile and test AFU on hardware 10, 12, 13 20, 21, 22, 27, 28 Test 6 Build, compile and test AFU Basic Building Blocks on hardware 10, 12, 13 20, 21, 22, 32, 33 Test 7 Build, compile and test oneAPI on hardware 10, 12, 13 34, 35, 36, 39, 40, 44, 45, 46, 47, 48, 49 Test 8 Build and Simulate Unit Tests - 50, 51 Test 9 Build and Simulate UVM Tests - 52, 53, 54, 55"},{"location":"hw/d5005/user_guides/ug_eval_ofs_d5005/ug_eval_script_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/","title":"Getting Started Guide: Open FPGA Stack for Intel Stratix 10","text":"Last updated: November 19, 2024
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#10-introduction","title":"1.0 Introduction","text":"This document helps users get started in evaluating Open FPGA Stack (OFS) for Stratix\u00ae 10 FPGA targeting the Intel\u00ae FPGA PAC D5005. After reviewing the document a user shall be able to:
- Set up a development environment with all OFS ingredients
- Build and install the OFS Linux Kernel drivers on the host
- Build and install the Open Programmable Acceleration Engine Software Development Kit (OPAE SDK) on the host
- Flash an OFS FIM binary onto the Intel\u00ae FPGA PAC D5005
- Verify the functionality of OFS on an Intel\u00ae FPGA PAC D5005 board
- Know where to find additional information on all OFS ingredients
The following flow charts show a high level overview of the initial bring-up process, split into three sequential diagrams.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#diagram-1-installing-the-opae-sdk","title":"Diagram 1: Installing the OPAE SDK","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#diagram-2-installing-the-linux-dfl-drivers","title":"Diagram 2: Installing the Linux DFL Drivers","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#diagram-3-bringing-up-the-intel-d5005","title":"Diagram 3: Bringing up the Intel D5005","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#11-intended-audience","title":"1.1 Intended Audience","text":"The information in this document is intended for customers evaluating the Open FPGA Stack for Stratix\u00ae 10 FPGA on the Intel PAC D5005. This document will cover key topics related to initial setup and development, with links for deeper dives on the topics discussed therein.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#12-terminology","title":"1.2 Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#13-reference-documents","title":"1.3 Reference Documents","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#14-component-version-summary","title":"1.4 Component Version Summary","text":"The OFS 2024.1-1 Release targeting the Stratix\u00ae 10 FPGA is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions which comprise this release.
The following table highlights the hardware which makes up the Best Known Configuration (BKC) for the OFS 2024.1-1 release.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-1-2-hardware-bkc","title":"Table 1-2: Hardware BKC","text":"Component 1 x Intel\u00ae FPGA PAC D5005 1 x Supported Server Model 1 x Intel FPGA Download Cable II (Optional, only required if loading images via JTAG) The following table highlights the versions of the software which comprise the OFS stack. The installation of the user-space OPAE SDK on top of the kernel-space linux-dfl drivers is discussed in subsequent sections of this document.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-1-3-software-version-summary","title":"Table 1-3: Software Version Summary","text":"Component Version FPGA Platform Intel\u00ae FPGA PAC D5005 OPAE SDK Tag: 2.12.0-5 Kernel Drivers Tag: ofs-2024.1-6.1-2 OFS FIM Source Code Branch: ofs-2024.1-1 Intel Quartus Prime Pro Edition Design Software 23.4 [Intel\u00ae Quartus\u00ae Prime Pro Edition Linux] Operating System RHEL 8.6 A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae FPGA PAC D5005 can be found on the OFS 2024.1-1 official release drop on GitHub.
Note: If you wish to freeze your Red Hat operating system version on the RHEL 8.6, refer to the following solution provided in the Red Hat customer portal.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#15-host-bios","title":"1.5 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
- Intel VT for Directed I/O (VT-d) must be enabled
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#16-host-server-kernel-and-grub-configuration","title":"1.6 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
- OS: RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.6
- Kernel: 6.1.78
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
OFS is a blanket term which can be used to collectively refer to all ingredients of the OFS reference design, which includes the core hardware components discussed below and software.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM) or 'shell' provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The FIM is implemented in a static region of the FPGA device.
The primary components of the FIM reference design are:
- PCIe Subsystem
- Transceiver Subsystem
- Memory Subsystem
- FPGA Management Engine
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from host or AFU
- Interface to Board Management Controller (BMC)
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration.
For more information on the FIM and its external connections, please refer to the Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs, and the Intel FPGA Programmable Acceleration Card D5005 Data Sheet. Below is a high-level block diagram of the FIM.
Figure 2-1 FIM Overview
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required for partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capabilities to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your AFU resides in a partial reconfiguration (PR) region of the FPGA.
- Your AFU is a part of the static region (SR) and is a compiled flat design.
- Your AFU contains both static and PR regions.
The AFU provided in this release is comprised of the following functions:
- AFU interface handler to verify transactions coming from the AFU region.
- PV/VF Mux to route transactions to and from corresponding AFU components, including the ST2MM module, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), and Memory Host Exerciser (HE-MEM).
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads).
- Port gasket and partial reconfiguration support.
For more information on the Platform Interface Manager (PIM) and AFU development and testing, please refer to the [OFS AFU Development Guide].
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#22-ofs-software-overview","title":"2.2 OFS Software Overview","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#221-kernel-drivers-for-ofs","title":"2.2.1 Kernel Drivers for OFS","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on the DFL Wiki page.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#30-intel-fpga-pac-d5005-card-installation-and-server-requirements","title":"3.0 Intel FPGA PAC D5005 Card Installation and Server Requirements","text":"Currently OFS for Stratix\u00ae 10 FPGA targets the Intel\u00ae FPGA PAC D5005. Because the Intel\u00ae FPGA PAC D5005 is a production card, you must prepare the card in order to receive a new non-production bitstream. For these instructions, please contact an Intel representative.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#40-ofs-dfl-kernel-drivers","title":"4.0 OFS DFL Kernel Drivers","text":""},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#41-ofs-dfl-kernel-driver-installation","title":"4.1 OFS DFL Kernel Driver Installation","text":"All OFS DFL kernel driver code resides in the Linux DFL GitHub repository. This repository is open source and does not require any permissions to access. It includes a snapshot of the latest best-known configuration (BKC) Linux kernel with the OFS driver included in the drivers/fpga/* directory. Downloading, configuration, and compilation will not be discussed in this document. Please refer to the Software Installation Guide: OFS for PCIe Attach FPGAs for guidelines on environment setup and build steps for all OFS stack components.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.1-1 Release Page.
It is recommended you boot into your operating system's native 4.18.x kernel before attempting to upgrade to the dfl enabled 6.1.78 You may experience issues when moving between two dfl enabled 6.1.78 kernels.
This installation process assumes the user has access to an internet connection in order to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#42-ofs-dfl-kernel-driver-installation-environment-setup","title":"4.2 OFS DFL Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.6 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 4.4 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
-
It is recommended you lock your Red Hat release version to RedHatEnterprise Linux\u00ae (RHEL) 8.6 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.6\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
-
Install the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#43-building-and-installing-the-ofs-dfl-kernel-drivers-from-source","title":"4.3 Building and Installing the OFS DFL Kernel Drivers from Source","text":"It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl\ncd /home/OFS/linux-dfl\ngit checkout tags/ofs-2024.1-6.1-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n
2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nofs-2024.1-6.1-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n
3. Copy an existing kernel configuration file from /boot and apply the minimal required settings changes.
```bash\ncd /home/OFS/linux-dfl\ncp /boot/config-`uname -r` .config\ncat configs/dfl-config >> .config\necho 'CONFIG_LOCALVERSION=\"-dfl\"' >> .config\necho 'CONFIG_LOCALVERSION_AUTO=y' >> .config\nsed -i -r 's/CONFIG_SYSTEM_TRUSTED_KEYS=.*/CONFIG_SYSTEM_TRUSTED_KEYS=\"\"/' .config\nsed -i '/^CONFIG_DEBUG_INFO_BTF/ s/./#&/' .config\necho 'CONFIG_DEBUG_ATOMIC_SLEEP=y' >> .config\nexport LOCALVERSION=\n```\n
Note: If you wish to add an identifier to the kernel build, edit .config and make your additions to the line CONFIG_LOCALVERSION=\"\".
4. The above command may report errors resembling symbol value 'm' invalid for CHELSIO_IPSEC_INLINE. These errors indicate that the nature of the config has changed between the currently executing kernel and the kernel being built. The option \"m\" for a particular kernel module is no longer a valid option, and the default behavior is to simply turn the option off. However, the option can likely be turned back on by setting it to 'y'. If the user wants to turn the option back on, change it to 'y' and re-run \"make olddefconfig\":
```bash\ncd /home/OFS/linux-dfl\necho 'CONFIG_CHELSIO_IPSEC_INLINE=y' >> .config\n```\n\n*Note: To use the built-in GUI menu for editing kernel configuration parameters, you can opt to run `make menuconfig`.*\n
5. Linux kernel builds take advantage of multiple processors to parallelize the build process. Display how many processors are available with the nproc command, and then specify how many make threads to utilize with the -j option. Note that number of threads can exceed the number of processors. In this case, the number of threads is set to the number of processors in the system.
```bash\ncd /home/OFS/linux-dfl\nmake -j $(nproc)\n```\n
6. You have two options to build the source:
- Using the built-in install option from the kernel Makefile.\n- Locally building a set of RPM/DEP packages.\n\nThis first flow will directly install the kernel and kernel module files without the need to create a package first:\n\n```bash\ncd /home/OFS/linux-dfl\nsudo make modules_install -j $(nproc)\nsudo make install\n```\n\nIn this second flow, the OFS Makefile contains a few options for package creation:\n\n- rpm-pkg: Build both source and binary RPM kernel packages\n- binrpm-pkg: Build only the binary kernel RPM package\n- deb-pkg: Build both source and binary deb kernel packages\n- bindeb-pkg: Build only the binary kernel deb package\n\nIf you are concerned about the size of the resulting package and binaries, they can significantly reduce the size of the package and object files by using the make variable INSTALL_MOD_STRIP. If this is not a concern, feel free to skip this step. The below instructions will build a set of binary RPM packages:\n\n```bash\ncd /home/OFS/linux-dfl\nmake INSTALL_MOD_STRIP=1 binrpm-pkg -j `nproc`\n```\n\nIf the kernel development package is necessary for other software you plan on installing outside of OFS, you should instead use the build target `rpm-pkg`.\n\nBy default, a directory is created in your home directory called `rpmbuild`. This directory will house all the kernel packages which have been built. You need to navigate to the newly built kernel packages and install them. The following files were generated using the build command executed in the previous step:\n\n```bash\ncd ~/rpmbuild/RPMS/x86_64\nls\nkernel-4.18.0_dfl.x86_64.rpm kernel-headers-6.1.78_dfl.x86_64.rpm\nsudo dnf localinstall kernel*.rpm\n```\n
7. The system will need to be rebooted in order for changes to take effect. After a reboot, select the newly built kernel as the boot target. This can be done pre-boot using the command grub2-reboot, which removes the requirement for user intervention. After boot, verify that the currently running kernel matches expectation.
```bash\nuname -r\n6.1.78-dfl\n```\n
8. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/6.1.78-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\n
If an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n
9. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n
10. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n
11. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n
12. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-6.1.78-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\n
A list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#44-installing-the-ofs-dfl-kernel-drivers-from-pre-built-packages","title":"4.4 Installing the OFS DFL Kernel Drivers from Pre-Built Packages","text":"To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page. You can choose to either install using the SRC RPMs, or to use the pre-built RPM packages targeting the official supported release platform.
tar xf kernel-6.1.78_dfl-1.x86_64-*.tar.gz\n\nsudo dnf localinstall kernel-6.1.78_dfl_*.x86_64.rpm \\\nkernel-devel-4.18.0_dfl_*.x86_64.rpm \\\nkernel-headers-4.18.0_dfl_*.x86_64.rpm\n\n### OR\nsudo dnf localinstall kernel-6.1.78_dfl_*.src.rpm\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#50-opae-software-development-kit","title":"5.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE GitHub. This repository is open source.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#51-opae-sdk-installation","title":"5.1 OPAE SDK Installation","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 5.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#52-opae-sdk-installation-environment-setup","title":"5.2 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package. -
Remove any currently installed OPAE packages.
sudo dnf remove opae*\n
-
Initialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.12.0-5\n
-
Verify that the correct tag/branch have been checkout out.
git describe --tags\n2.12.0-5\n
-
Set up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\n
The following packages will be built in the same directory as create:
-
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\n
-
Check that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.12.0-5.el8.x86_64.rpm\nopae-debuginfo-2.12.0-5.el8.x86_64.rpm\nopae-debugsource-2.12.0-5.el8.x86_64.rpm\nopae-devel-2.12.0-5.el8.x86_64.rpm\nopae-devel-debuginfo-2.12.0-5.el8.x86_64.rpm\nopae-extra-tools-2.12.0-5.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.12.0-5.el8.x86_64.rpm\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#53-installing-the-opae-sdk-with-pre-built-packages","title":"5.3 Installing the OPAE SDK with Pre-Built Packages","text":"You can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.12.0-5.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.12.0-5.x86_64-*.tar.gz\n
For a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#54-opae-tools-overview","title":"5.4 OPAE Tools Overview","text":"The OPAE SDK user-space tools sit upon the kernel-space DFL drivers. In order to use OPAE SDK functionality the user needs to have installed both the OPAE SDK and Linux DFL driver set. You must have at least one D5005 card with the appropriate FIM present in your system. The steps to read and load a new FIM version are discussed in section 7.1 Programming the OFS FIM. After both the DFL kernel-space drivers have been installed and the FIM has been upgraded, you may proceed to test the OPAE commands discussed below.
This section covers basic functionality of the commonly used OPAE tools and their expected results. These steps may also be used to verify that all OFS software installation has been completed successfully. A complete overview of the OPAE tools can be found on the OPAE GitHub and in your cloned GitHub repo at <your path>/opae-sdk/doc/src/fpga_tools. More commands are listed than are defined in the list below - most of these are called by other tools and do not need to be called directly themselves.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#541-fpgasupdate","title":"5.4.1 fpgasupdate","text":"The fpgasupdate tool updates the Intel Max10 BMC image and firmware, root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate will only accept images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then the image will also need to be signed using the correct keys. Please refer to the [Security User Guide: Intel Open FPGA Stack] for information on created signed images and on programming and managing the root entry hash.
The Intel FPGA PAC ships with a factory and user programmed image for both the FIM and BMC FW and RTL on all cards.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-5-1-fpgasupdate-overview","title":"Table 5-1: fpgasupdate Overview","text":"Synopsis:
fpgasupdate [--log-level=<level>] file [bdf]\n
Description: The fpgasupdate command implements a secure firmware update.
Command args (optional) Description --log-level Specifies the log-level which is the level of information output to your command tool. The following seven levels are available: state, ioctl, debug, info, warning, error, critical. Setting --log-level=state provides the most verbose output. Setting --log-level=ioctl provides the second most information, and so on. The default level is info. file Specifies the secure update firmware file to be programmed. This file may be to program a static region (SR), programmable region (PR), root entry hash, key cancellation, or other device-specific firmware. bdf The PCIe address of the PAC to program. bdf is of the form [ssss:]bb:dd:f, corresponding to PCIe segment, bus, device, function. The segment is optional. If you do not specify a segment, the segment defaults to 0000. If the system has only one PAC you can omit the bdf and let fpgasupdate determine the address automatically."},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#542-fpgainfo","title":"5.4.2 fpgainfo","text":"Synopsis:
fpgainfo [-h] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\n{errors,power,temp,fme,port,bmc,mac,phy,security}\n
Description: Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Command args (optional) Description --help, -h Prints help information and exits. --version, -v Prints version information and exits. -S, --segment PCIe segment number of resource. -B, --bus PCIe bus number of resource. -D, --device PCIe device number of resource. -F, --function PCIe function number of resource. errors {fme, port, all} --clear, -c First agument to the errors command specifies the resource type to display in human readable format. The second optional argument clears errors for the given FPGA resource. power Provides total power in watts that the FPGA hardware consumes temp Provides FPGA temperature values in degrees Celsius port Provides information about the port fme Provides information about the FME bmc Provides BMC sensors information mac Provides information about MAC ROM connected to FPGA security Provides information about the security keys, hashes, and flash count, if available. Note: Your Bitstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo.
$ sudo fpgainfo fme\n\nOpen FPGA Stack Platform\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\n
$ sudo fpgainfo bmc\n\nOpen FPGA Stack Platform\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** BMC SENSORS ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\n( 1) VCCERAM Voltage : 0.90 Volts\n( 2) VCCT Temperature : 29.00 Celsius\n( 3) 12v Backplane Voltage : 12.17 Volts\n( 4) VCCERAM Current : 0.18 Amps\n( 5) FPGA Transceiver Temperature : 36.50 Celsius\n( 6) QSFP1 Supply Voltage : 0.00 Volts\n( 7) 3.3v Temperature : 29.00 Celsius\n( 8) 12v Backplane Current : 2.28 Amps\n( 9) RDIMM3 Temperature : 25.50 Celsius\n(10) VCCR Voltage : 1.12 Volts\n(11) Board Inlet Air Temperature : 24.50 Celsius\n(12) 1.8v Temperature : 27.50 Celsius\n(13) 12v AUX Voltage : 12.14 Volts\n(14) VCCR Current : 0.55 Amps\n(15) RDIMM0 Temperature : 24.50 Celsius\n(16) FPGA Core Voltage : 0.88 Volts\n(17) VCCERAM Temperature : 27.50 Celsius\n(18) 12v AUX Current : 1.19 Amps\n(19) QSFP0 Temperature : N/A\n(20) VCCT Voltage : 1.12 Volts\n(21) FPGA Core Current : 11.60 Amps\n(22) FPGA Core Temperature : 42.50 Celsius\n(23) 12v Backplane Temperature : 24.00 Celsius\n(24) VCCT Current : 0.14 Amps\n(25) RDIMM1 Temperature : 24.00 Celsius\n(26) 3.3v Voltage : 3.30 Volts\n(27) VCCR Temperature : 33.50 Celsius\n(28) 1.8v Voltage : 1.80 Volts\n(29) 3.3v Current : 0.32 Amps\n(30) Board Exhaust Air Temperature : 26.00 Celsius\n(31) 12v AUX Temperature : 25.00 Celsius\n(32) QSFP0 Supply Voltage : 0.00 Volts\n(33) QSFP1 Temperature : N/A\n(34) 1.8v Current : 0.54 Amps\n(35) RDIMM2 Temperature : 26.00 Celsius\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#543-rsu","title":"5.4.3 rsu","text":"The rsu performs a Remote System Update operation on a device, given its PCIe address. A rsu operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either the BMC or FPGA.
The Intel FPGA PAC contains a region of flash the user may store their FIM image. After an image has been programmed with fpgasupdate the user may choose to perform rsu to update the image on the device.
Note: The D5005 platform only supports storing and configuring a single user image from flash for the FPGA. It does not include support for the user1/user2 partitions as shown in other OFS related acceleration boards.
rsu Overview
Synopsis
rsu [-h] [-d] {bmc,bmcimg,retimer,sdm,fpgadefault} [PCIE_ADDR]\n
rsu bmc --page=(user) [PCIE_ADDR]\nrsu retimer [PCIE_ADDR]\nrsu sdm [PCIE_ADDR]\n
Perform RSU (remote system update) operation on PAC device given its PCIe address. An RSU operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either BMC, Retimer, SDM, (on devices that support these) or the FPGA.
Note: As a result of using the rsu command, the host rescans the PCI bus and may assign a different Bus/Device/Function (B/D/F) value than the originally assigned value.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#544-pacsign","title":"5.4.4 PACsign","text":"PACSign is an OPAE utility which allows users to insert authentication markers into bitstreams targeted for the platform. All binary images must be signed using PACSign before fpgasupdate can use them for an update. Assuming no Root Entry Hash (REH) has been programmed on the device, the following examples demonstrate how to prepend the required secure authentication data, and specify which region of flash to update. More information, including charts detailing the different certification types and their required options, are fully described in the PACsign python/pacsign/PACSign.md OPAE GitHub on GitHub.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-5-4-pacsign-overview","title":"Table 5-4: PACSign Overview","text":"Synopsis:
PACSign [-h] {FIM,SR,SR_TEST,BBS,BMC,BMC_FW,BMC_FACTORY,AFU,PR,PR_TEST,GBS,FACTORY,PXE,THERM_SR,THERM_PR} ...\n\nPACSign <CMD> [-h] -t {UPDATE,CANCEL,RK_256,RK_384} -H HSM_MANAGER [-C HSM_CONFIG] [-s SLOT_NUM] [-r ROOT_KEY] [-k CODE_SIGNING_KEY] [-d CSK_ID] [-R ROOT_BITSTREAM] [-S] [-i INPUT_FILE] [-o OUTPUT_FILE] [-b BITSTREAM_VERSION] [-y] [-v]\n
Description: The PACSign utility inserts authentication markers into bitstreams.
Command args (optional) Description (required) -t, --cert_type TYPE The following operations are supported: UPDATE, CANCEL, RK_256, RK_348 (required) -H, --HSM_manager MODULE The module name for a module that interfaces to a HSM. PACSign includes both the openssl_manager and pkcs11_manager to handle keys and signing operations. -C, --HSM_config CONFIG The argument to this operation is passed verbatim to the specified HSM. For pkcs11_manager, this option specifies a JSON file describing the PKCS #11 capable HSM\u2019s parameters. -r, --root_key KEY_ID The key identifier that the HSM uses to identify the root key to be used for the selected operation. -k, --code_signing_key KEY_ID The key indentifier that the HSM uses to identify the code signing key to be used for the selected operation -d, --csk_id CSK_NUM Only used for type CANCEL. Specifies the key number of the code signing key to cancel. -s, --slot_num For bitstream types with multiple slots (i.e. multiple ST regions), this option specifies which of the slots to which this bitstream is to be acted upon -b, --bitstream_version VERSION User-formatted version information. This can be any string up to 32 bytes in length. -S, --SHA384 Used to specify that PACSign is to use 384-bit crypto. Default is 256-bit -R, --ROOT_BITSTREAM ROOT_BITSTREAM Valid when verifying bitstreams. The verification step will ensure the generated bitstream is able to be loaded on a board with the specified root entry hash programmed. -i, --input_file FILE Only to be used for UPDATE operations. Specifies the file name containing data to be signed -o, --output_file FILE Specifies the file name for the signed output bitstream. -y, --yes Silently answer all queries from PACSign in the affirmative. -v, --verbose Can be specified multiple times. Increases the verbosity of PACSign. Once enables non-fatal warnings to be displayed. Twice enables progress information. Three or more occurrences enables very verbose debugging information. -h Prints help information and exits {FIM, SR, SR_TEST, BBS, BMC, BMC_FW, BMC_FACTORY, AFU, PR, PR_TEST, GBS, FACTORY, PXE, THERM_SR, THERM_PR} Bitstream type identifier. PACSign can be run on images that have previously been signed. It will overwrite any existing authentication data.
The following example will create an unsigned SR image from an existing signed SR binary update image.
PACSign SR -t UPDATE -s 0 -H openssl_manager -i d5005_page1_unsigned.bin -o new_image.bin\n#output\nNo root key specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo CSK specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo root entry hash bitstream specified. Verification will not be done. Continue? Y = yes, N = no: y\n2022-07-20 10:13:54,954 - PACSign.log - WARNING - Bitstream is already signed - removing signature blocks\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#545-bitstreaminfo","title":"5.4.5 bitstreaminfo","text":"Displays authentication information contained with each provided file on the command line. This includes any JSON header strings, authentication header block information, and a small portion of the payload. The binary is installed by default at /usr/bin/bitstreaminfo.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#546-hssi","title":"5.4.6 hssi","text":"The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode-specific options. Only the hssi_10g MODE is currently supported. An example of this command's output can be found in section 5.2.9 Running the Host Exerciser Modules. The binary is installed by default at /usr/bin/hssi.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#547-opaeio","title":"5.4.7 opae.io","text":"Opae.io is a interactive Python environment packaged on top of libopaevfio.so, which provides user space access to PCIe devices via the vfio-pci driver. The main feature of opae.io is its built-in Python command interpreter, along with some Python bindings that provide a means to access Configuration and Status Registers (CSRs) that reside on the PCIe device. opae.io has two operating modes: command line mode and interactive mode. An example of this command's output can be found in section 5.2.9 Running the Host Exerciser Modules. The binary is installed by default at /usr/bin/opae.io.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#548-host_exerciser","title":"5.4.8 host_exerciser","text":"The host exerciser is used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. An example of this command's output can be found in section 5.2.9 Running the Host Exerciser Modules. The binary is installed by default at /usr/bin/host_exerciser. For more information refer to - Host Exerciser
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#549-running-the-host-exerciser-modules","title":"5.4.9 Running the Host Exerciser Modules","text":"The reference FIM and unchanged compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc.
Note: Before continuing, if huge pages are not set refer to section 4.2, step 7.
There are three HEMs present in the OFS FIM - HE-LPBK, HE-HSSI, and HE-MEM. These exercisers are tied to three different VFs that must be enabled before they can be used. The user should enable the VF for each HEM using the below steps:
1. Determine the BDF of the Intel\u00ae FPGA PAC D5005 card.
The PCIe BDF address is initially determined when the server powers on. The user can determine the addresses of all Intel\u00ae FPGA PAC D5005 boards using lspci:
lspci -d :bcce\n\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
Note: Before continuing, if you updated your OFS installation, please also update your PAC FIM to run HEMs.
2. Enable three VFs.
In this example, the BDF address is 0000:3b:00.0. With this information the user can now enable three VFs with the following:
sudo pci_device 0000:3b:00.0 vf 3\n
3. Verify that all three VFs have been created.
lspci -s 3b:00\n\n3b:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n3b:00.1 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.2 Processing accelerators: Intel Corporation Device bccf (rev 01)\n3b:00.3 Processing accelerators: Intel Corporation Device bccf (rev 01)\n
4. Bind the 3 VFs to the vfio-pci driver.
sudo opae.io init -d 0000:3b:00.1 $USER\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:3b:00.1 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:3b:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.1 is 142\nAssigning /dev/vfio/142 to $USER:$USER\nChanging permissions for /dev/vfio/142 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.2 $USER\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:3b:00.2 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:3b:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.2 is 143\nAssigning /dev/vfio/143 to $USER:$USER\nChanging permissions for /dev/vfio/143 to rw-rw----\n\nsudo opae.io init -d 0000:3b:00.3 $USER\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:3b:00.3 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:3b:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:3b:00.3 is 144\nAssigning /dev/vfio/144 to $USER:$USER\nChanging permissions for /dev/vfio/144 to rw-rw----\n
5. Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
$ sudo fpgainfo port\n\n//****** PORT ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x603B000000000000\nPCIe s:b:d.f : 0000:3B:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0x403B000000000000\nPCIe s:b:d.f : 0000:3B:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x203B000000000000\nPCIe s:b:d.f : 0000:3B:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-5-5-vf-to-hem-mappings","title":"Table 5-5 VF to HEM Mappings","text":"VF BDF HEM BBBB:DD.1 HE-LB BBBB:DD.2 HE-MEM BBBB:DD.3 He-HSSI HE-MEM / HE-LB
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise the DDR interface; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. The following commands are supported by the HE-LB/HE-MEM OPAE driver program. They may need to be run using sudo privileges, depending on your server configuration.
Basic operations:
$ sudo host_exerciser lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5342\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.067 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode lpbk lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5358\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.058 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode write lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 0\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2592\nTotal number of Reads sent: 0\nTotal number of Writes sent: 1024\nBandwidth: 6.321 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode trput lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 3384\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 4.842 GB/s\n Test lpbk(1): PASS\n
Number of cachelines per request 1, 2, and 4. The user may replace --mode lpbk with read, write, trput. The target lpbk can be replaced with mem:
$ sudo host_exerciser --mode lpbk --cls cl_1 lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5475\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 2.993 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode lpbk --cls cl_2 lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5356\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.059 GB/s\n Test lpbk(1): PASS\n\n$ sudo host_exerciser --mode lpbk --cls cl_4 lpbk\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 4481\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.656 GB/s\n Test lpbk(1): PASS\n
Interrupt tests (only valid for mode mem):
$ sudo host_exerciser --interrupt 0 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5140\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.188 GB/s\n Test mem(1): PASS\n\n$ sudo host_exerciser --interrupt 1 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5079\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.226 GB/s\n Test mem(1): PASS\n\n$ sudo host_exerciser --interrupt 2 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 5525\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.439 GB/s\n Test mem(1): PASS\n\n$ sudo host_exerciser --interrupt 3 mem\n\nstarting test run, count of 1\nAPI version: 1\nAFU clock: 250 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\nUsing Interrupts\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1026\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 4735\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.460 GB/s\n Test mem(1): PASS\n
HE-HSSI
HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G ethernet AFU and includes a 10G traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic. Context sensitive information is given by the hssi --help command. Help for the 10G specific test is given by hssi hssi_10g --help Example useage:
$ sudo hssi --pci-address 3b:00.3 hssi_10g --eth-ifc s10hssi0 --eth-loopback on --he-loopback=off --num-packets 100\n10G loopback test\nport: 0\neth_loopback: on\n he_loopback: off\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth: s10hssi0\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#60-compiling-ofs-fim","title":"6.0 Compiling OFS FIM","text":"Pre-Compiled FIM binaries are at OFS 2024.1-1 release page and to compile the OFS FIM for Intel\u00ae FPGA PAC D5005 follow the below steps :
1) Compile OFS FIM manually - Steps are provided in the developer guide to compile FIM and generate binaries. Refer to Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
2) Compile OFS FIM using evaluation script - The script guides you to the steps required for compilation via selecting options from the menu. Refer to Automated Evaluation User Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#70-programming-the-ofs-fim-and-bmc","title":"7.0 Programming the OFS FIM and BMC","text":"Instructions surrounding the compilation and simulation of the OFS FIM have fully moved into the Shell Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#71-programming-the-ofs-fim","title":"7.1 Programming the OFS FIM","text":"In order to program the OFS FIM, both the OPAE SDK and DFL drivers need to be installed on the host system. Please complete the steps in sections 4.0 OFS DFL Kernel Drivers and 5.0 OPAE Software Development Kit. The OFS FIM version can be identified using the OPAE tool fpgainfo. A sample output of this command is included below.
$ sudo fpgainfo fme\n\nIntel FPGA Programmable Acceleration Card D5005\nBoard Management Controller, MAX10 NIOS FW version: 2.0.8\nBoard Management Controller, MAX10 Build version: 2.0.8\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:3B:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x138D\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 288511860124977321\nBitstream Version : 4.0.1\nPr Interface Id : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e\nBoot Page : user\n
Use the value under PR Interface ID to identify that FIM that has been loaded. Refer to the table below for a list of previous FIM releases:
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#table-7-1-previous-fim-releases","title":"Table 7-1 Previous FIM Releases","text":"PR Release PR Interface ID 2023.2 edad864c-99d6-5831-ab67-62bfd81ec654 2022.2 (tag 1.3.0) bf531bcf-a896-5171-ab31-601a4ab754b6 2022.1 Beta (tag: 1.2.0-beta) 2fae83fc-8568-53aa-9157-8f75e9c0ba92 OFS 2.1 Beta (tag: 1.1.0-beta) 99160d37e42a 3f8b586f-c275-594c-92e2-d9f2c23e94d1 OFS 1.0 (tag: ofs-1.0.0) b5f6a71e-daec-59c3-a43a-85567b51fd3f Intel Acceleration Stack for Intel\u00ae FPGA PAC D5005 2.0.1 9346116d-a52d-5ca8-b06a-a9a389ef7c8d If the user's card does not report a PR Interface ID which matches the above table, then a new FIM will need to be programmed.
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#711-programming-the-fim","title":"7.1.1 Programming the FIM","text":"1. Download the file d5005_page1_unsigned.bin from OFS 2024.1-1 release page.
2. Run PACSign to create an unsigned image with added header for use by fpgasupdate
$ PACSign SR -y -v -t UPDATE -s 0 -H openssl_manager -i d5005_page1_unsigned.bin -o d5005_PACsigned_unsigned.bin\n
3. Run fpgasupdate to load the image into the user location of the Intel\u00ae FPGA PAC D5005 FPGA flash, NOTE: use \"sudo fpgainfo fme\" command to find the PCIe address for your card.
$ sudo fpgasupdate d5005_PACsigned_unsigned.bin 3B:00.0\n
4. Run RSU command.
$ sudo rsu bmcimg 0000:3B:00.0\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#72-programming-the-bmc","title":"7.2 Programming the BMC","text":"1. Download intel-fpga-bmc images(To download OFS Stratix 10 BMC binaries contact Intel Technical Sales Representative)
2. The file unsigned_bmc_fw.bin has the newly binary format. This bitstream is programmed with remote system update (RSU) and the bitstream must be signed with PACSign tool to generate.
3. Run PACSign to create an unsigned image with added header for use by fpgasupdate
$ PACSign BMC -y -v -t UPDATE -s 0 -H openssl_manager -i unsigned_bmc_fw.bin -o PACsigned_unsigned_bmc_fw.bin\n\n2022-04-22 03:07:05,626 - PACSign.log - INFO - OpenSSL version \"OpenSSL 1.1.1k FIPS 25 Mar 2021\" matches \"1.1.1\"\n2022-04-22 03:07:05,648 - PACSign.log - INFO - Bitstream not previously signed\n2022-04-22 03:07:05,648 - PACSign.log - INFO - platform value is '688128'\n2022-04-22 03:07:05,745 - PACSign.log - INFO - Starting Block 0 creation\n2022-04-22 03:07:05,745 - PACSign.log - INFO - Calculating SHA256\n2022-04-22 03:07:05,747 - PACSign.log - INFO - Calculating SHA384\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Done with Block 0\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Root Entry creation\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Calculating Root Entry SHA\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Code Signing Key Entry creation\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Calculating Code Signing Key Entry SHA\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Code Signing Key Entry done\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Block 0 Entry creation\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Calculating Block 0 Entry SHA\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Block 0 Entry done\n2022-04-22 03:07:05,749 - PACSign.log - INFO - Starting Block 1 creation\n2022-04-22 03:07:05,750 - PACSign.log - INFO - Block 1 done\n2022-04-22 03:07:05,757 - PACSign.log - INFO - Writing blocks to file\n2022-04-22 03:07:05,758 - PACSign.log - INFO - Processing of file 'PACsigned_unsigned_bmc_fw.bin' complete\n
4. Run fpgasupdate to perform an upgrade of the BMC.
$ sudo fpgasupdate PACsigned_unsigned_bmc_fw.bin 3B:00.0\n\n[2022-04-22 03:08:34.15] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-22 03:08:34.15] [INFO ] updating from file pacsign_unsigned_bmc_fw.bin with size 819968\n[2022-04-22 03:08:34.15] [INFO ] waiting for idle\n[2022-04-22 03:08:34.15] [INFO ] preparing image file\n[2022-04-22 03:09:02.18] [INFO ] writing image file\n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [819968/819968 bytes][Elapsed Time: 0:00:13.01]\n[2022-04-22 03:09:15.20] [INFO ] programming image file\n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:00:29.03]\n[2022-04-22 03:09:44.24] [INFO ] update of 0000:3B:00.0 complete\n[2022-04-22 03:09:44.24] [INFO ] Secure update OK\n[2022-04-22 03:09:44.24] [INFO ] Total time: 0:01:10.08\n
"},{"location":"hw/d5005/user_guides/ug_qs_ofs_d5005/ug_qs_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/","title":"UVM Simulation User Guide: Open FPGA Stack for Intel Stratix\u00ae 10 FPGA","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel FPGA PAC D5005 Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace."},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#1-overview","title":"1 Overview","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the UVM simulation tool using OFS. After reviewing the document, you will be able to:
- Set-up the UVM verification tool suite
- Run pre-existing UVM unit tests and also create new UVM tests for your design
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#2-introduction-to-uvm","title":"2 Introduction to UVM","text":"OFS (Open FPGA Stack) provides a UVM (Universal Verification Methodology) environment for the FIM (FPGA Interface Manager) with a modular, reusable, and scalable testbench structure via an API framework.
The framework consists of a FIM Testbench which is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this Testbench. UVM RAL (Register Abstaction Layer) is used for CSR (Command and Status Registers) verification.
The qualified verification IPs will help to detect incorrect protocol behavior, help to focus on FIM features and accelerate the verification process.
Verification components include:
- FIM monitor to detect correct design behavior
- FIM assertions for signal level integrity testing
- Arm AMBA Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity
- FIM coverage to collect functional data
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#3-testbench-architecture","title":"3 Testbench Architecture","text":"The testbench connects to the full chip that includes major RTL blocks depicted in Figure 1.
Figure 1 Testbench Diagram
The major interface is between the Xeon and FPGA where PCIe Verification IP is connected to PCIe Subsystem. Therefore, as a full chip simulation environment, PCIe host VIP is the sole VIP/BFM used. PCIe host VIP connects to PCIe device which resides in FPGA in serial mode.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#4-testbench-infrastructure","title":"4 Testbench Infrastructure","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#41-traffic-flow","title":"4.1 Traffic Flow","text":"PCIe Host, as the master of FPGA, initiates MMIO read/write requests to FPGA to program registers. The PCIe host also passively receives memory requests from FPGA to read from or write to host memory.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#42-link-up-and-enumeration","title":"4.2 Link Up and Enumeration","text":"With serial mode connection between PCIe host and device, link training and enumeration has to be done before the regular traffic starts.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#421-link-up","title":"4.2.1 Link Up","text":"Linkup sequence(pcie_device_bring_up_link_sequence) is part of configure sequence(ofs_config_seq), which is started in UVM configure phase.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#422-enumeration","title":"4.2.2 Enumeration","text":"PCIe host driver needs to retrieve information from the device hard IP and program necessary configuration space registers, such as PF/VF BAR values. This is done in enumerate_seq, which follows link up sequence in configure sequence(ofs_config_seq).
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#423-pfvf-bar","title":"4.2.3 PF/VF BAR","text":"PF0 BAR0 is set in the base sequence and can be randomized. During enumeration, PF0 BAR0, along with PCIe device hard IP configuration, derives other PF and VF BAR values. These BAR values are stored into base sequence variables and can be used throughout any test sequences that extend the base sequence.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#43-mmio-apis","title":"4.3 MMIO APIs","text":"The base sequence provides APIs for 32-bit and 64-bit MMIO read/write accesses, as well as blocking or non-blocking for MMIO read as described in Table 1. The users can use MMIO APIs without knowing the underlining PCIe sequence items.
Name API 32-bit MMIO Write task mmio_write32(input bit [63:0] addr_, input bit [31:0] data_); 64-bit MMIO Write task mmio_write64(input bit [63:0] addr_, input bit [63:0] data_); 32-bit MMIO Read task mmio_read32(input bit [63:0] addr_, output bit [31:0] data_, input blocking_ = 1); 64-bit MMIO Read task mmio_read64(input bit [63:0] addr_, output bit [63:0] data_, input blocking_ = 1); Table 1 MMIO APIs
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#44-ral","title":"4.4 RAL","text":"UVM RAL is integrated in the testbench providing alternative ways of accessing CSRs in test sequences. RAL is generated from an excel format CSR specification where register name, field, offset, bitmap, attribute, and description are specified.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#45-vip-dut-connection","title":"4.5 VIP DUT Connection","text":"PCIe host verification IP and DUT connection is achieved by connecting 16 bits lanes. The module for connection from VIP is svt_pcie_device_agent_serdes_x16_x8g_hdl.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#5-test-plan","title":"5 Test Plan","text":"The test plan consists of four major categories: MMIO path, HE-LB, HE-MEM, HE-HSSI and interrupt tests.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#51-mmio-path","title":"5.1 MMIO Path","text":"The tests under this category exercise MMIO path including all destination functions or blocks as well as PF/VF mux and different fabrics.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#52-he-lb","title":"5.2 HE-LB","text":"The tests under this category target HE-LB function only. Software which test sequences needs to configure HE-LB CSRs before starting it. These CSRs include SRC_ADDR, DST_ADDR, DSM_ADDR, NUM_LINES, CFG etc.
If HE-LB is configured to have memory read transactions, PCIe host memory has to be initialized before HE-LB is started. This is done by svt_pcie_mem_target_service sequence. In other words, PCIe host VIP programs its internal memory model entries in backdoor way. The same process applies to DSM memory entry.
Once HE-LB is started, HE-LB will function based on what it is programmed to do. When HE-LB is done with all necessary memory transactions, it will perform a final memory write to DSM memory entry. Since the software does not know when hardware is done, software polls DSM memory entry periodically until the DSM status bit is asserted.
For loopback mode, data is compared between source buffer and destination buffer in host memory.
RTL statistic counters are also compared against the corresponding variables inside the test sequence at the end of the simulation.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#53-he-mem","title":"5.3 HE-MEM","text":"HE-MEM tests are duplicates from HE-LB with MMIO to CSRs targeting HE-MEM instead of HE-LB.
The DDR simulation model is inside memory controller IP when being generated.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#54-he-hssi","title":"5.4 HE-HSSI","text":"HE-HSSI has indirect registers that are associated with HSSI subsystem, MMIO for indirect registers is different from other functions.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#541-indirect-registers","title":"5.4.1 Indirect Registers","text":"To obtain access to indirect registers, either reading or writing, a MMIO write must be performed.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#542-tx-loopback","title":"5.4.2 TX Loopback","text":"In TX loopback, HE-HSSI initiates ethernet packets to HSSI subsystem and the packets are looped back to HE-HSSI. The loopback is achieved by hard-wiring HSSI TX and RX lanes. This is done inside RTL and for simulation purposes only.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#55-interrupt-test","title":"5.5 Interrupt Test","text":"The test plan covers the basic interrupt flow for FME error, PORT error and user AFU interrupts. The MSI-X table must be programmed in PF0 BAR4. Corresponding PBA bit is expected to be asserted.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#56-performance-test","title":"5.6 Performance Test","text":"Performance tests are derived from HE-LB tests and they are directed tests. At the end of the simulation, performance number is calculated and printed to terminal and a log file.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#57-csr-test","title":"5.7 CSR Test","text":"CSR consists of two parts.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#571-reset-value-check","title":"5.7.1 Reset Value Check","text":"Front-door MMIO read data is compared against RAL register reset value out of reset.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#572-rw-attribute-csr","title":"5.7.2 RW Attribute CSR","text":"MMIO write-read-compare is performed after reset value check for RW attribute CSRs.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#6-checking-mechanism","title":"6 Checking Mechanism","text":"Since there is only PCIe host verification component in testbench, data checking is done by a self-checking mechanism in the test sequence.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#61-protocol-violation","title":"6.1 Protocol Violation","text":"PCIe host VIP has built-in protocol checking on TLP received from FPGA. Abnormal responses are also flagged by the VIP.
Internal AXI Streaming interface has integrated RTL assertion to check AXI Streaming protocol violations.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#62-data-checking","title":"6.2 Data Checking","text":"Data checking is done by self-checking inside a test sequence. MMIO write/read/compare to read-writable CSRs is done inside a sequence.
For memory transactions initiated by functions, backdoor reads from host memory on source buffer and destination buffer is done inside a sequence. Data is compared in case of loopback mode.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#63-counter-checking","title":"6.3 Counter Checking","text":"RTL statistic counters records the number of transactional information that can be read at the end of the simulation and compared against the test expected number.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#64-afu-error-csr","title":"6.4 AFU Error CSR","text":"AFU interface handler provides an error log for illegal transactions that can be read at the end of the simulation.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#7-uvm-set-up","title":"7 UVM set-up","text":"To run the tutorial steps in this guide requires the following development environment:
Item Version Intel Quartus Prime Pro Intel Quartus Prime Pro 23.4 Simulator (VCS) Synopsys VCS S-2021.09-SP1 or newer for UVM simulation of top level FIM Simulator (Questasim) Questasim 2023.4 or newer for UVM simulation of top level FIM"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#71-uvm-prerequisite","title":"7.1 UVM Prerequisite","text":"Retrieve OFS repositories.
The OFS FIM source code is included in the GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories. Cloning the repo using the HTTPS method requires a personal access token. Please see this blog post for information about obtaining a personal access token Token authentication requirements for Git operations.
Navigate to the location for storage of OFS source, create the top-level source directory and clone OFS repositories.
$ mkdir ofs-2024.1-1\n$ cd ofs-2024.1-1\n$ export OFS_BUILD_ROOT=$PWD\n$ git clone --recurse-submodules https://github.com/OFS/ofs-d5005\n\nCloning into 'ofs-d5005' ...\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\n\n$ cd ofs-d5005\n$ git checkout tags/ofs-2024.1-1\n
Verify that the correct tag/branch have been checked out
$ git describe --tags\n\n$ ofs-2024.1-1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#72-license-requirements","title":"7.2 License Requirements","text":"The FIM Testbench is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this TB. UVM RAL (Register Abstraction Layer) is used for CSR Verification.
The Qualified Verification IPs will help to detect incorrect protocol behavior easily, help to focus on BBS features and accelerate the verification process.
- VCS & DVE
- SNPS-Assertions
- Verdi
- VerdiCoverage
- VerdiSimDB
- VerdiTransactionDebugUltra
- VIP-AMBA-AXI-SVT
- VIP-AMBA-STREAM-SVT
- VIP-PCIE-SVT
- VIP-PCIE-TS-SVT
- VIP-PCIE-G3-OPT-SVT
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#73-software-tools-requirements","title":"7.3 Software Tools Requirements","text":"The following tools are required for successful UVM set-up
- Python 3.6.8
- Synopsys PCIE and AMBA AXI UVM VIP Q-2020.03A License
- Synopsys Verdi R-2020.12-SP2 License Note: Makefile can be modified to use DVE instead of Verdi
- VCS R-2020.12-SP2 License
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#74-creating-a-software-tools-script","title":"7.4 Creating a Software Tools Script","text":"The UVM tool set-up is best done by creating a simple set-up script so all applicable tools are sourced before running the tests.
The following environment variables can be pasted into a script and used prior to running the UVM verification environment
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#license-files","title":"License Files","text":"export LM_LICENSE_FILE=\nexport SNPSLMD_LICENSE_FILE=\n
The license environment variables LM_LICENSE_FILE and SNPSLMD_LICENSE_FILE can point to a server license on your system.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#general-environment-variables","title":"General Environment Variables","text":"export OFS_BUILD_ROOT=$PWD\nexport OFS_ROOTDIR=<user_path>/ofs-d5005\nexport WORKDIR=$OFS_ROOTDIR\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#quartus-tools","title":"Quartus Tools","text":"export QUARTUS_HOME=<user_path>/intelFPGA_pro/23.4/quartus\nexport QUARTUS_ROOTDIR=$QUARTUS_HOME\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-verification-tools","title":"Synopsys Verification Tools","text":"export DESIGNWARE_HOME=<user_path>/synopsys/vip_common/vip_Q-2020.03A\nexport PATH=$DESIGNWARE_HOME/bin:$PATH\nexport UVM_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm\nexport VCS_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel\nexport PATH=$VCS_HOME/bin:$PATH\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim-verification-tools","title":"QuestaSIM Verification Tools","text":"export MTI_HOME=<user_path>/mentor/questasim/2023.4/linux64\nexport PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\nexport QUESTA_HOME=$MTI_HOME\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#8-running-a-uvm-simulation-test-and-analysing-results","title":"8 Running a UVM Simulation Test and Analysing Results","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#81-simulation","title":"8.1 Simulation","text":"The default simulator used in this document is Synopsys VCS-MX but there will be references to Questasim. Users can refer to the options and adopt the options for other simulators.
The script is a makefile that calls vlogan, vcs and simv for compilation, elaboration and simulation, respectively
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#82-file-structure","title":"8.2 File Structure","text":"After cloning the repo, the verification and ofs-common directories contain all UVM verification related files. The directory structure is shown in Figure 2 below.
Figure 2 UVM Verification Directory File Structure
ofs-d5005/testbench has a testbench, uvm env, virtual sequencer, RAL etc.
ofs-d5005/verification/tests contains all uvm tests and sequences.
Users can run the simulation under \"ofs-d5005/verification/scripts\" directory and the simulation result is outputted to a \"sim\" directory for Synopsys VCS or \"sim_msim\" for Questasim.
The simulation result folder is named after the test name with increasing suffix number. If user runs the same test multiple times, the suffix is incremented by 1 each time.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#83-uvm-test-suite","title":"8.3 UVM Test Suite","text":"The UVM environment contains a variety of tests that have been developed to test out the FIM portion of OFS.
The table below lists out the \"Test Name\" which will be used on the command line to execute the test, the \"Test Scenario\" and the \"Checking Criteria\".
Tests are located at ofs-d5005/verification/tests
Test Name DUT Scope Test Scenario Checking Criteria dfh_walking_test DFH DHF walking offset checking, eol checking flr_reset_test FLR Reset FLR reset to all PFs Reset checking flr_vf0_reset_test FLR Reset FLR reset to VF0 Reset checking flr_vf1_reset_test FLR Reset FLR reset to VF1 Reset checking flr_vf2_reset_test FLR Reset FLR reset to VF2 Reset checking fme_csr_test FME CSR CSR accesses data checking fme_hemem_intr_test Interrupt FME and HE MEM interrupt Interrupts assertion, PBA bits check fme_intr_test Interrupt FME error interrupt Interrupts assertion, PBA bits check he_hssi_csr_test HE-HSSI CSR accesses for HSSI data checking he_hssi_err_test HE-HSSI Error Cases counter checking he_hssi_rx_lpbk_test HE-HSSI RX loopback data checking he_hssi_tx_lpbk_test HE-HSSI TX loopback counter checking he_lpbk_cont_test HE-LPBK Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_lpbk_long_rst_test HE-MEM Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_lpbk_long_test HE-MEM Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_lpbk_port_rst_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len with port rst data checking he_lpbk_rd_cont_test HE-LPBK Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_rd_test HE-LPBK Read only mode. Randomize num_lines, addresses, req_len counter checking he_lpbk_reqlen1_test HE-LPBK Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_lpbk_reqlen2_test HE-LPBK Loopback mode. 128 CLs, req_len = 2CL, random addresses. data checking, counter checking he_lpbk_reqlen4_test HE-LPBK Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_lpbk_reqlen8_test HE-LPBK Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_lpbk_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_thruput_contmode_test HE-LPBK Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses req_len data checking, counter checking he_lpbk_thruput_test HE-LPBK Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_lpbk_wr_cont_test HE-LPBK Write only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_wr_test HE-LPBK Write only mode. Randomize num_lines, addresses, req_len counter checking he_mem_cont_test HE-MEM Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_mem_lpbk_long_rst_test HE-LPBK Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_mem_lpbk_long_test HE-LPBK Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_mem_lpbk_reqlen1_test HE-MEM Loopback mode. 128 CLs, req_len = 1CL, random addresses. data checking, counter checking he_mem_lpbk_reqlen2_test HE-MEM Loopback mode. 128 CLs, req_len = 2CL, random addresses. data checking, counter checking he_mem_lpbk_reqlen4_test HE-MEM Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_mem_lpbk_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_cont_test HE-MEM Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_test HE-MEM Read only mode. Randomize num_lines, addresses, req_len counter checking he_mem_thruput_contmode_test HE-MEM Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_thruput_test HE-MEM Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_mem_wr_cont_test HE-MEM Write only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_wr_test HE-MEM Write only mode. Randomize num_lines, addresses, req_len counter checking he_random_long_test All HE's Enable all HEs and randomize modes for multiple iterations data checking if in lpbk mode, counter checking he_random_test All HEs Enable all HEs and randomize modes data checking if in lpbk mode, counter checking hehssi_csr_test HE-HSSI CSR accesses for Traffic Control Mail box registers data checking helb_csr_test HE-LPBK CSR accesses data checking helb_rd_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Read only mode data checking, counter checking helb_rd_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Read only mode data checking, counter checking helb_rd_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Read only mode data checking, counter checking helb_thruput_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Thruput mode data checking, counter checking helb_thruput_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Thruput mode data checking, counter checking helb_thruput_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Thruput mode data checking, counter checking helb_wr_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Write only mode data checking, counter checking helb_wr_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Write only mode data checking, counter checking helb_wr_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Write only mode data checking, counter checking hemem_csr_test HE-MEM CSR accesses data checking hemem_intr_test Interrupt HE MEMN Interrupt Interrupts assertion, PBA bits check malformedtlp_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. maxpayloaderror_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. MaxTagError_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3.Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. mini_smoke_test All HEs shorter simpler version of random test for turn-in sanity check data checking if in lpbk mode, counter checking mmio_64b_bar_test PCIe MMIO Path 64-bit bar addess for MMIO data checking mmio_stress_nonblocking_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux with non-blocking MMIO reads data checking mmio_stress_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux data checking mmio_test PCIe MMIO Path MMIO targeting PF0(ST2MM, FME, PMCI, HSSI SS), PF1, PF1.VF1, PF1.VF2 data checking mmio_unimp_test PCIe MMIO Path MMIO acccess to unimplemented addresses MMIO checking MMIODataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. MMIOInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. MMIOTimedout_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing. msix_csr_test MSIX CSR CSR accesses data checking pmci_csr_test PMCI CSR CSR accesses data checking port_gasket_csr_test PORT GASKET Port Gasket CSR test port csr checking TxMWrDataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing TxMWrInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing UnexpMMIORspErr_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be retuened on resds. Write a 0x5 to set and a 0x4 to clear) 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. After clearing the error register,check if normal transcation are completing The next section describes how to compile and build the UVM environment prior to running each UVM test and analyzing the results in the log files
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#84-ip-compile","title":"8.4 IP Compile","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs","title":"Synopsys VCS","text":"To compile all IPs for the Synopsys VCS simulater:
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk cmplib\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim","title":"Questasim","text":"To compile all IPs for the Questasim simulater:
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk cmplib
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#85-rtl-test-bench-compile","title":"8.5 RTL & Test Bench Compile","text":"The RTL file list for compilation is located here: verification/scripts/rtl_comb.f
The TB file list for compilation is located here: verification/scripts/ver_list.f
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_1","title":"Synopsys VCS","text":"To compile RTL and Testbench for the Synopsys VCS simulater
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk build DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_1","title":"Questasim","text":"To compile RTL and Testbench for the Questasim simulater
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk build DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#86-ip-and-rtl-test-bench-compile","title":"8.6 IP and RTL & Test Bench Compile","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_2","title":"Synopsys VCS","text":"If the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS then follow the procedure below
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk build_all DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_2","title":"Questasim","text":"If the user wants to compile all IPs and RTL Testbench in one command for Questasim then follow the procedure below
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk build_all DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_3","title":"Synopsys VCS","text":"To run a simulation for Synopsys VCS:
cd $VERDIR/scripts\n\ngmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_3","title":"Questasim","text":"To run a simulation for Questasim:
cd $VERDIR/scripts\n\ngmake -f Makefile_MSIM.mk run TESTNAME=mmio_test DUMP=1
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_4","title":"Synopsys VCS","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation.
ofs-d5005/verification/scripts gmake -f Makefile_VCS.mk build DUMP=1\nofs-d5005/verification/scripts gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> DUMP=1\n
Or ofs-d5005/verification/scripts gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> DUMP=1\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_4","title":"Questasim","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Questasim build and simulation.
ofs-d5005/verification/scripts gmake -f Makefile_MSIM.mk build DUMP=1\nofs-d5005/verification/scripts gmake -f Makefile_MSIM.mk run TESTNAME=<test_case_name> DUMP=1\n
Or ofs-d5005/verification/scripts gmake -f Makefile_MSIM.mk build_run TESTNAME=<test_case_name> DUMP=1\n
There are some optimizations in the Table below for convenience if you want to bypass some commands for both Synopsys VCS and Questasim:
Command (Synopsys VCS) Command (Questasim) Details gmake -f Makefile_VCS.mk build_all DUMP=1 gmake -f Makefile_MSIM.mk build_all DUMP=1 compile IP + compile RTL gmake -f Makefile_VCS.mk build_run TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk build_run TESTNAME= DUMP=1 compile RTL + run test gmake -f Makefile_VCS.mk do_it_all TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk do_it_all TESTNAME= DUMP=1 compile IP, RTL and run test gmake -f Makefile_VCS.mk rundb TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk rundb TESTNAME= DUMP=1 run test in sim dir + over-writes content"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#87-uvm-regression-test","title":"8.7 UVM Regression Test","text":"cd $VERDIR/scripts\n\nFor Regression in VCS with top/test package, execute the following command python uvm_regress.py -l -n 8 -s vcs -c\n\nResults are created in a sim directory ($VERDIR/sim) with individual testcase log dir\n\nFor Regression in MSIM with top/test package, execute the following command python uvm_regress.py -l -n 8 -s msim -c\n
Results are created in a sim directory ($VERDIR/sim_msim) with individual testcase log dir
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#88-uvm-waveform-and-transcript-analysis","title":"8.8 UVM Waveform and Transcript Analysis","text":""},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#synopsys-vcs_5","title":"Synopsys VCS","text":"Running Synopsys VCS UVM tests will generate a ofs-d5005/verification/sim directory
- All build time logs are at ofs-d5005/verification/sim
- Each testcase will have separate directory inside sim ofs-d5005/verification/sim/
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Synopsys VCS. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 3
Figure 3 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 4
Figure 4 trans.log
The waveform generated is named as \"inter.vpd\". To open the waveform, go to simulation result directory and run
dve -full64 -vpd inter.vpd &\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#questasim_5","title":"Questasim","text":"Running Questasim UVM tests will generate a ofs-d5005/verification/sim_msim directory
- All build time logs are at ofs-d5005/verification/sim_msim
- Each testcase will have separate directory inside sim ofs-d5005/verification/sim_msim/
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Questasim. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 4
Figure 4 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 5
Figure 5 trans.log
The waveform generated is named as \"vsim.wlf\". To open the waveform, go to simulation result directory and run
vsim -view vsim.wlf &
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#9-modifying-uvm-testbench","title":"9 Modifying UVM Testbench","text":"The next section describe what needs to be considered when modifying the UVM, targeting a different device, adding a new interface to the testbench and creating a new UVM test for a customized OFS Accelerator platform.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#91-modifying-uvm-environment-when-targeting-different-device","title":"9.1 Modifying UVM environment when targeting different device","text":"A new device may have different design feature or flow. The base address must be allocated for the new device. The MMIO targeting the new device must be based on the base address. If it is a new PF or VF, PCIe HIP must be regenerated and enumeration sequence must be updated accordingly.
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#92-modifying-uvm-environment-when-adding-a-new-interface","title":"9.2 Modifying UVM environment when adding a new interface","text":"Adding a new interface requires signal connections in the testbench. An additional BFM or verification IP is needed to drive the new interface. The main testbench file tb_top.sv is found at the following location
$OFS_ROOTDIR/verification/testbench\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#93-adding-a-new-uvm-test","title":"9.3 Adding a new UVM test","text":"In the following example we will modify an existing test \"he_lpbk\" and name it \"he_lpbk_new\", and rebuild the test to check it. Please follow the steps below
-
Create a new test sequence file under ofs-d5005/verification/tests/sequences
he_lpbk_seq_new.svh\n
-
Modify ifndef, define and endif statements in new test sequence case i.e he_lpbk_seq_new.svh file
`ifndef HE_LPBK_SEQ_NEW_SVH `define HE_LPBK_SEQ_NEW_SVH\n`endif // HE_LPBK_SEQ_NEW_SVH\n
also replace all occurences of he_lpbk_seq with he_lpbk_seq_new in the he_lpbk_seq_new.svh file
-
Append the new sequence name into ofs-d5005/verification/tests/sequences/seq_lib.svh file
`include \"he_lpbk_seq_new.svh\"\n
-
Create a new test under ofs-d5005/verification/tests
he_lpbk_test_new.svh\n
-
Modify ifndef, define and endif statements in new test case i.e he_lpbk_test_new.svh file
`ifndef HE_LPBK_TEST_NEW_SVH `define HE_LPBK_TEST_NEW_SVH\n`endif // HE_LPBK_TEST_NEW_SVH\n
also replace all occurences of he_lpbk_test with he_lpbk_test_new in the he_lpbk_test_new.svh file
-
Append the new test name into ofs-d5005/verification/tests/test_pkg.svh file
`include \"he_lpbk_test_new.svh\"\n
-
Rebuild UVM test suite for either Synopsys VCS or Questasim simulater
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk build_all\n
or
cd $VERDIR/scripts\ngmake -f Makefile_MSIM.mk build_all\n
-
Execute new test for either Synopsys VCS or Questasim simulater
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk run TESTNAME=he_lpbk_test_new\n
or
cd $VERDIR/scripts\ngmake -f Makefile_MSIM.mk run TESTNAME=he_lpbk_test_new\n
9) Check new test and log files cd ofs-d5005/verification/sim/he_lpbk_test_new
```sh\nopen runsim.log\n```\n
"},{"location":"hw/d5005/user_guides/ug_sim_ofs_d5005/ug_sim_ofs_d5005/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/doc_modules/Glossary/","title":"Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/doc_modules/Notices_%26_Disclaimers/","title":"Notices & Disclaimers","text":""},{"location":"hw/doc_modules/Notices_%26_Disclaimers/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/doc_modules/afu_flr_disable/","title":"Afu flr disable","text":"The vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\n
Enable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\n
If you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\n
Sample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\n
"},{"location":"hw/doc_modules/agx7_pcie_attach_build_command_gen_tool/","title":"Agx7 pcie attach build command gen tool","text":"OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command."},{"location":"hw/doc_modules/agx7_pcie_attach_sec_base_ofss_file/","title":"Agx7 pcie attach sec base ofss file","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_build_work_directory/","title":"Agx7 pcie attach sec build work directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_fim_build_script/","title":"Agx7 pcie attach sec fim build script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_fim_simulation/","title":"Agx7 pcie attach sec fim simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_hssi_ofss_file/","title":"Agx7 pcie attach sec hssi ofss file","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_iopll_ofss_file/","title":"Agx7 pcie attach sec iopll ofss file","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_memory_ofss_file/","title":"Agx7 pcie attach sec memory ofss file","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_null_host_exercisers/","title":"Agx7 pcie attach sec null host exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_ofss_file_usage/","title":"Agx7 pcie attach sec ofss file usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_pcie_ofss_file/","title":"Agx7 pcie attach sec pcie ofss file","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/doc_modules/agx7_pcie_attach_sec_top_level_ofss_file/","title":"Agx7 pcie attach sec top level ofss file","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/doc_modules/contents_agx7_pcie_attach/","title":"Documentation Overview for Agilex 7 PCIe Attach Reference Designs","text":"Use this page as a guide to understand how to navigate the documentation depending on your role:
Document Description Board Developer Shell Developer Workload Developer Software Developer Board Installation Guide: OFS for Acceleration Development Platforms Describes how to install the N6000/N60001 platform in a server with required cabling and configure BIOS X X Board Installation Guide: OFS for Agilex\u00ae 7 PCIe Attach Development Kits Describes how to install evaluation kits in a server with required cabling and configure BIOS X X Software Installation Guide: OFS for PCIe Attach FPGAs Provides steps for installing Linux kernel packages and OPAE software development kit package X X X Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) Provides steps for loading a pre-packaged PCIe attach shell binary unto the I-Series Development Kit and using the OPAE user space tools X X Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) Provides steps for loading a pre-packaged PCIe attach shell binary unto the F-Series Development Kit and using the OPAE user space tools X X Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) Provides steps for loading a pre-packaged PCIe attach shell binary unto the Intel FPGA SmartNIC N6000/1-PL platform and using the OPAE user space tools X X Automated Evaluation User Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs Describes how to use provided automated script to step through setup, build flows, simulation and OneAPI tasks. Script can be leveraged for your own custom builds. X X PCIe Attach Shell Technical Reference Manual Describes features and functions of the Agilex 7 PCIe attach reference design for the Intel FPGA SmartNIC N6000-PL Platform X Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs Provides explanation of the shell reference designs targeting the I-Series Development Kit and steps for building and modifying the design X Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs Provides explanation of the shell reference design targeting the F-tile Development Kit and steps for building and modifying the design X Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs Provides explanation of the shell reference design targeting the Intel FPGA SmartNIC N6000-PL platform and steps for building and modifying the design X Hard Processor System Software Developer Guide: OFS for Agilex\u00ae FPGAs Provides steps for building or using pre-compiled ITB images for exploration and evaluation of the HPS bring-up flow on the Intel FPGA SmartNIC N6000/1-PL platforms. X X UVM Simulation User Guide: OFS for Agilex\u00ae 7 PCIe Attach Describes how to use the provided UVM environment files and test suite to simulate the default reference designs X Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs Describes how to build, test and debug an example workload for a PCIe attach shell. X PIM Based AFU Developer Guide Describes how to create a an AFU/workload using the Platform Interface Manager (PIM) shims X AFU Host Software Developer Guide Provides instructions on managing data transfer between the host and the AFU X X AFU Simulation Environment User Guide Provides steps for using the AFU Simulation Environment for hardware/software co-simulation of your workload X oneAPI Accelerator Support Package (ASP): Getting Started User Guide Provides steps to use and test with the default OneAPI Accelerator Support Package X oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack Provides details about the hardware and software components required for enabling OneAPI in OFS designs. X Software Reference Manual: Open FPGA Stack Provides details of OPAE SDK user-space software stack and the kernel-space linux-dfl drivers X KVM User Guide: Open FPGA Stack Describes how to setup and configure a virtual machine in an OFS-enabled design X X Docker User Guide: Open FPGA Stack Provides steps for implementing a docker container to evaluate and develop with OFS X X X Document Description Board Developer Shell Developer Workload Developer Software Developer"},{"location":"hw/doc_modules/contents_agx7_soc_attach/","title":"Documentation Overview for Agilex 7 SoC Attach Reference Designs","text":"Use this page as a guide to understand how to navigate the documentation depending on your role:
Document Description Board Developer Shell Developer Workload Developer Software Developer Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL Describes how to install the Intel FPGA IPU F2000X-PL Platform in a server with required cabling and configure BIOS X X Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Provides steps for installing Linux kernel packages and OPAE software development kit package X X X Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Provides steps for loading a pre-packaged SoC attach shell binary unto the Intel FPGA IPU F2000x-PL and using the OPAE user space tools X X Automated Evaluation User Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Describes how to use provided automated script to step through setup, build flows, simulation and OneAPI tasks. Script can be leveraged for your own custom builds. X X Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs Describes features and functions of the provided Agilex 7 SoC Attach Reference design X Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Provides explanation of the shell reference design targeting the Intel FPGA IPU F2000X-PL Platform and steps for building and modifying the design X UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach Describes how to use the provided UVM environment files and test suite to simulate the default reference designs X Workload Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs Describes how to build, test and debug an example workload for a SoC attach shell. X PIM Based AFU Developer Guide Describes how to create a an AFU/workload using the Platform Interface Manager (PIM) shims X AFU Simulation Environment User Guide Provides steps for using the AFU Simulation Environment for hardware/software co-simulation of your workload X Software Reference Manual: Open FPGA Stack Provides details of OPAE SDK user-space software stack and the kernel-space linux-dfl drivers X KVM User Guide: Open FPGA Stack Describes how to setup and configure a virtual machine in an OFS-enabled design X X Docker User Guide: Open FPGA Stack Provides steps for implementing a docker container to evaluate and develop with OFS X X X Document Description Board Developer Shell Developer Workload Developer Software Developer"},{"location":"hw/doc_modules/contents_s10_pcie_attach/","title":"Documentation Overview for Stratix 10 PCIe Attach Reference Designs","text":"Use this page as a guide to understand how to navigate the documentation depending on your role:
Document Description Board Developer Shell Developer Workload Developer Software Developer Board Installation Guide: OFS for Acceleration Development Platforms Describes how to install the Intel FPGA PAC D5005 Platform in a server with required cabling and configure BIOS X X Software Installation Guide: OFS for PCIe Attach FPGAs Provides steps for installing Linux kernel packages and OPAE software development kit package X X X Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs Provides steps for loading a pre-packaged shell binary unto the Intel FPGA PAC D5005 platform and using the OPAE user space tools X X Automated Evaluation User Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs Describes how to use provided automated script to step through setup, build flows, simulation and OneAPI tasks. Script can be leveraged for your own custom builds. X X Shell Technical Reference Manual: OFS for Stratix\u00ae 10 PCIe Attach FPGAs Provides explanation of the SoC attach shell reference design targeting the Intel FPGA PAC D5005 Platform and steps for building and modifying the design X UVM Simulation User Guide: OFS for Stratix\u00ae 10 PCIe Attach Describes how to use the provided UVM environment files and test suite to simulate the default reference designs X Workload Developer Guide: OFS for Stratix\u00ae 10 PCIe Attach FPGAs Describes how to build, test and debug an example workload for a PCIe attach shell. X PIM Based AFU Developer Guide Describes how to create a an AFU/workload using the Platform Interface Manager (PIM) shims X AFU Simulation Environment User Guide Provides steps for using the AFU Simulation Environment for hardware/software co-simulation of your workload X oneAPI Accelerator Support Package (ASP): Getting Started User Guide Provides steps to use and test with the default OneAPI Accelerator Support Package X oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack Provides details about the hardware and software components required for enabling OneAPI in OFS designs. X Software Reference Manual: Open FPGA Stack Provides details of OPAE SDK user-space software stack and the kernel-space linux-dfl drivers X KVM User Guide: Open FPGA Stack Describes how to setup and configure a virtual machine in an OFS-enabled design X X Docker User Guide: Open FPGA Stack Provides steps for implementing a docker container to evaluate and develop with OFS X X X Document Description Board Developer Shell Developer Workload Developer Software Developer"},{"location":"hw/doc_modules/links/","title":"OFS Collateral","text":""},{"location":"hw/doc_modules/links/#top-level-collateral-contents-pages","title":"Top Level Collateral Contents Pages","text":""},{"location":"hw/doc_modules/links/#automated-evaluation-guides","title":"Automated Evaluation Guides","text":""},{"location":"hw/doc_modules/links/#board-installation-guides","title":"Board Installation Guides","text":""},{"location":"hw/doc_modules/links/#software-installation-guides","title":"Software Installation Guides","text":""},{"location":"hw/doc_modules/links/#getting-started-guides","title":"Getting Started Guides","text":""},{"location":"hw/doc_modules/links/#shell-technical-reference-manuals","title":"Shell Technical Reference Manuals","text":""},{"location":"hw/doc_modules/links/#shell-developer-guides","title":"Shell Developer Guides","text":""},{"location":"hw/doc_modules/links/#workload-developer-guides","title":"Workload Developer Guides","text":""},{"location":"hw/doc_modules/links/#oneapi","title":"oneAPI","text":""},{"location":"hw/doc_modules/links/#uvm-simulation-user-guides","title":"UVM Simulation User Guides","text":""},{"location":"hw/doc_modules/links/#common","title":"Common","text":""},{"location":"hw/doc_modules/links/#ofs-repositories","title":"OFS Repositories","text":""},{"location":"hw/doc_modules/links/#oneapi-sites","title":"oneAPI Sites","text":""},{"location":"hw/doc_modules/links/#software","title":"Software","text":""},{"location":"hw/doc_modules/links/#afu-dev","title":"AFU Dev","text":""},{"location":"hw/doc_modules/links/#misc","title":"Misc","text":""},{"location":"hw/doc_modules/quartus_installation/","title":"Quartus installation","text":"Intel ${{ env.QUARTUS_PRO_VER_L }} is verified to work with the latest OFS release ${{ env.OFS_FIM_RELEASE }}. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use ${{ env.HOST_OS_L }} for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are ${{ env.QUARTUS_PATCHES }}.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }} then the new line is:
export PATH=/home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }}/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }}/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/${{ env.QUARTUS_PRO_VER_S }}/quartus/bin/quartus\n
"},{"location":"hw/doc_modules/wt_clone_fim_repo/","title":"Wt clone fim repo","text":"Perform the following steps to clone the OFS ${{ env.DESIGN }} FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules ${{ env.OFS_FIM_REPO_URL }}.git\n
-
Check out the correct tag of the repository
cd ${{ env.OFS_FIM_REPO }}\ngit checkout --recurse-submodules tags/${{ env.OFS_FIM_TAG }}\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (${{ env.OFS_FIM_TAG }})\n
"},{"location":"hw/doc_modules/wt_compile_fim_in_prep_for_afu/","title":"Wt compile fim in prep for afu","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the [Set Up Development Environment] Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the [Clone FIM Repository] section for step-by-step instructions.
-
Set development environment variables. Refer to the [Set Development Environment Variables] section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/${{ env.MODEL }}.ofss ${{ env.MODEL }}:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_${{ env.MODEL }}\n
"},{"location":"hw/doc_modules/wt_install_git_lfs_rhel/","title":"Wt install git lfs rhel","text":"To install the Git Large File Storage (LFS) extension, execute the following commands:
- Obtain Git LFS package
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh\n
- Install Git LFS package
sudo dnf install git-lfs\n
- Install Git LFS
git lfs install\n
"},{"location":"hw/doc_modules/wt_run_individual_unit_level_sim/","title":"Wt run individual unit level sim","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the [Walkthrough: Set Up Development Environment] Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the [Walkthrough: Clone FIM Repository] section for step-by-step instructions.
-
Set development environment variables. Refer to the [Walkthrough: Set Development Environment Variables] section for step-by-step instructions.
-
Generate the simulation files for the ${{ env.MODEL }}
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\n./gen_sim_files.sh --ofss=$OFS_ROOTDIR/tools/ofss_config/${{ env.MODEL }}.ofss ${{ env.MODEL }}\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/${{ env.OFS_FIM_REPO }}/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/doc_modules/wt_set_fim_dev_env_vars/","title":"Wt set fim dev env vars","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ${{ env.OFS_FIM_REPO }}\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=${{ env.OPAE_BRANCH }}\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/doc_modules/wt_set_up_development_environment/","title":"Wt set up development environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that ${{ env.QUARTUS_PRO_VER_L }} for Linux with Intel Agilex FPGA device support is installed on your development machine. Refer to the [Walkthrough: Install Quartus Prime Pro Software] section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion ${{ env.QUARTUS_PRO_VER_S }} SC Pro Edition\nCopyright (C) 2024 Intel Corporation. All rights reserved.\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python ${{ env.PYTHON_VER }} or later
-
Verify version number
python --version\n
Example Output:
Python ${{ env.PYTHON_VER }}\n
-
GCC ${{ env.GCC_VER }} or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) ${{ env.GCC_VER }}\n
-
cmake ${{ env.CMAKE_VER }} or later
-
Verify version number
cmake --version\n
Example output:
cmake version ${{ env.CMAKE_VER }}\n
-
git ${{ env.GIT_VER }} or later.
-
Verify version number
git --version\n
Example output:
git version ${{ env.GIT_VER }}\n
-
Clone the ${{ env.OFS_FIM_REPO }} repository. Refer to the [Walkthrough: Clone FIM Repository] section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches ${{ env.QUARTUS_PATCHES }}. All required patches are provided in the Assets of the OFS FIM Release: ${{ env.OFS_FIM_RELEASE_PAGE_URL }}
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the [Walkthrough: Set Environment Variables] section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/f2000x/","title":"Index","text":"Location for SoC Attach Collateral for OFS.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/","title":"AFU Developer Guide: OFS for Agilex\u00ae 7 FPGA SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#1-introduction","title":"1. Introduction","text":"This document is a design guide for the creation of an Accelerator Functional Unit (AFU) using Open FPGA Stack (OFS) for Agilex\u00ae 7 FPGAs SoC Attach. The AFU concept consists of separating out the FPGA design development process into two parts, the construction of the foundational FPGA Interface Manager (FIM), and the development of the Acceleration Function Unit (AFU), as shown in the diagram below.
This diagram shows the separation of FPGA board interface development from the internal FPGA workload creation. This separation starts with the FPGA Interface Manager (FIM) which consists of the external interfaces and board management functions. The FIM is the base system layer and is typically provided by board vendors. The FIM interface is specific to a particular physical platform. The AFU makes use of the external interfaces with user defined logic to perform a specific application. By separating out the lengthy and complicated process of developing and integrating external interfaces for an FPGA into a board allows the AFU developer to focus on the needs of their workload. OFS for Agilex\u00ae 7 FPGAs SoC Attach provides the following tools for rapid AFU development:
- Scripts for both compilation and simulation setup
- Optional Platform Interface Manager (PIM) which is a set of SystemVerilog shims and scripts for flexible FIM to AFU interfacing
- Acceleration Simulation Environment (ASE) which is a hardware/software co-simulation environment scripts for compilation and Acceleration
- Integration with Open Programmable Acceleration Engine (OPAE) SDK for rapid software development for your AFU application
Please notice in the above block diagram that the AFU region consists of static and partial reconfiguration (PR) regions where the PR region can be dynamically reconfigured while the remaining FPGA design continues to function. Creating AFU logic for the static region is described in Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs. This guide covers logic in the AFU Main region.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#11-document-organization","title":"1.1. Document Organization","text":"This document is organized as follows:
- Description of design flow
- Interfaces and functionality provided in the Agilex\u00ae 7 FPGAs SoC Attach FIM
- Setup of the AFU Development environment
- Synthesize the AFU example
- Testing the AFU example on the IPU Platform F2000X-PL card
- Hardware/Software co-simulation using ASE
- Debugging an AFU with Remote Signal Tap
This guide provides theory followed by tutorial steps to solidify your AFU development knowledge.
NOTE:
This guide uses the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL as the platform for all tutorial steps. Additionally, this guide and the tutorial steps can be used with other Agilex\u00ae 7 FPGAs SoC Attach platforms..
Some of the document links in this guide are specific to the IPU Platform F2000X-PL . If using a different platform, please use the associated documentation for your platform as there could be differences in building the FIM and downloading FIM images.
If you have worked with previous Altera\u00ae Programmable Acceleration products, you will find out that OFS for Agilex\u00ae 7 FPGAs SoC Attach is similar. However, there are differences and you are advised to carefully read and follow the tutorial steps to fully understand the design tools and flow.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#12-prerequisite","title":"1.2. Prerequisite","text":"This guide assumes you have the following FPGA logic design-related knowledge and skills:
- FPGA compilation flows including the Quartus\u00ae Prime Pro Edition design flow
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition software, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL and coding practices to create synthesizable logic.
- Understanding of AXI and Avalon memory mapped and streaming interfaces.
- Simulation of complex RTL using industry standard simulators (Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae).
- Signal Tap Logic Analyzer tool in the Quartus\u00ae Prime Pro Edition software.
You are strongly encouraged to review the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#13-acceleration-functional-unit-afu-development-flow","title":"1.3. Acceleration Functional Unit (AFU) Development Flow","text":"The AFU development flow is shown below:
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#131-understanding-platform-capabilities","title":"1.3.1. Understanding Platform Capabilities","text":"The block diagram of the F2000x Board is shown below:
The F2000x FIM provided with this release is shown below:
This release F2000x FIM provides the following features:
- Host interface
- PCIe Gen4 x 16
- 2 - PFs
- MSI-X interrupts
- Logic to demonstrate simple PCIe loopback
- Network interface
- 2 - QSFP28/56 cages
- 8 X 25 GbE with exerciser logic demonstrating traffic generation/monitoring
- External Memory - DDR4 - 2400
- 4 Banks - 4 GB organized as 1 Gb x 32 with 1 Gb x 8 ECC
- Memory exerciser logic demonstrating external memory operation
- Board Management
- SPI interface
- FPGA management and configuration
- Example logic showing DFH operation
- Partial reconfiguration control logic
- SoC - Xeon Icelake-D subsystem with embedded Linux
- PCIe Gen4 x 16 interface to FPGA
- 1 - PF, 3 - VF, AXI-S TLP packets
- DDR Memory
- 2 Banks - 8 GB organized as 1 Gb x 64 with 1 Gb x 8 ECC
- NVMe SSD - 64 GB
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#132-high-level-data-flow","title":"1.3.2. High Level Data Flow","text":"The OFS high level data flow is shown below:
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#133-considerations-for-pim-usage","title":"1.3.3. Considerations for PIM Usage","text":"When creating an AFU, a designer needs to decide of what type of interfaces the AFU will provide to the platform (FIM). The AFU can use the native interfaces (i.e. PCIe TLP commands) provided by the FIM or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
The following resources are available to assist in creating an AFU:
PIM Core Concepts provides details on using the PIM and its capabilities.
PIM Based AFU Developer Guide provides details on interfacing your AFU to the FIM using the PIM.
The examples-afu repo provides several AFU examples:
Example Description PIM-based Hybrid Native clocks Example AFU using user configurable clocks. X copy_engine Example AFU moving data between host channel and a data engine. X dma Example AFU moving data between host channel and local memory with a DMA. X hello_world Example AFU sending \"Hello World!\" to host channel. X X X local_memory Example AFU reading and writing local memory. X X These examples can be run with the current OFS FIM package. There are three AFU types of examples provided (PIM based, hybrid and native). Each example provides the following:
- RTL, which includes the following interfaces:
- Host Channel:
- Host memory, providing a DMA interface.
- MMIO, providing a CSR interface.
- Local Memory
- Host software example interfacing to the CSR interface and host memory interface, using the OPAE C API.
- Accelerator Description File .json file
- Source file list
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#134-afu-interfaces-included-with-ipu-platform-f2000x-pl","title":"1.3.4. AFU Interfaces Included with IPU Platform F2000X-PL","text":"The figure below shows the interfaces available to the AFU in this architecture. It also shows the design hierarchy with module names from the fim (top.sv) to the PR region AFU (afu_main.sv). One of the main differences from the Stratix 10 PAC OFS architecture to this one is the presence of the static port gasket region (port_gasket.sv) that has components to facilitate the AFU and also consists of the PR region (afu_main.sv) via the PR slot. The Port Gasket contains all the PR specific modules and logic, e.g., PR slot reset/freeze control, user clock, remote STP etc. Architecturally, a Port Gasket can have multiple PR slots where user workload can be programmed into. However, only one PR slot is supported for OFS Release for Agilex. Everything in the Port Gasket until the PR slot should be provided by the FIM developer. The task of the AFU developer is to add their desired application in the afu_main.sv module by stripping out unwanted logic and instantiating the target accelerator. As shown in the figure below, here are the interfaces connected to the AFU (highlighted in green) via the SoC Attach FIM:
- AXI Streaming (AXI-S) interface to the Host via PCIe Gen4x16
- AXI Memory Mapped Channels (4) to the DDR4 EMIF interface
- AXI Streaming (AXI-S) interface to the HSSI 25 Gb Ethernet
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#2-set-up-afu-development-environment","title":"2. Set Up AFU Development Environment","text":"This section covers the setup of the AFU development environment.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#21-prepare-afu-development-environment","title":"2.1. Prepare AFU development environment","text":"A typical development and hardware test environment consists of a development server or workstation with FPGA development tools installed and a separate server with the target OFS compatible FPGA PCIe card installed. The typical usage and flow of data between these two servers is shown below:
Note: both development and hardware testing can be performed on the same server if desired.
This guide uses IPU Platform F2000X-PL as the target OFS compatible FPGA PCIe card for demonstration steps. The IPU Platform F2000X-PL must be fully installed following the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL. If using a different OFS FPGA PCIe card, contact your supplier for instructions on how to install and operate user developed AFUs.
The following is a summary of the steps to set up for AFU development:
- Install Quartus Prime Pro Version 23.4 for Linux with Agilex device support and required Quartus patches.
- Make sure support tools are installed and meet version requirements.
- Install OPAE SDK.
- Download the Basic Building Blocks repository.
- Build or download the relocatable AFU PR-able build tree based on your Agilex FPGA PCIe Attach FIM.
- Download FIM to the Agilex FPGA PCIe Attach platform.
Building AFUs with OFS for Agilex requires the build machine to have at least 64 GB of RAM.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#22-installation-of-quartus-and-required-patches","title":"2.2. Installation of Quartus and required patches","text":"Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use Ubuntu 22.04 LTS for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.17.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#23-installation-of-support-tools","title":"2.3. Installation of Support Tools","text":"Make sure support tools are installed and meet version requirements.
The OFS provided Quartus build scripts require the following tools. Verify these are installed in your development environment.
Item Version Python 3.6.8 GCC 8.5.0 cmake 3.15 git 1.8.3.1 perl 5.8.8"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#24-installation-of-opae-sdk","title":"2.4. Installation of OPAE SDK","text":"Working with the IPU Platform F2000X-PL card requires opae-2.12.0-5. Follow the instructions in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs to build and install the required OPAE SDK for the IPU Platform F2000X-PL. Make sure to check out the cloned repository to tag 2.12.0-5 and branch release/2.12.0.
$ git checkout tags/2.12.0-5 -b release/2.12.0\n
Note: The tutorial steps provided in the next sections assume the OPAE SDK is installed in default system locations, under the directory, /usr. In most system configurations, this will allow the OS and tools to automatically locate the OPAE binaries, scripts, libraries and include files required for the compilation and simulation of the FIM and AFUs.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#25-download-the-basic-building-blocks-repositories","title":"2.5. Download the Basic Building Blocks repositories","text":"The ofs-platform-afu-bbb repository contains the PIM files as well as example PIM-based AFUs that can be used for testing and demonstration purposes. This guide will use the host_chan_mmio AFU example in the ofs-platform-afu-bbb repository and the hello_world sample in the examples-afu repository to demonstrate how to synthesize, load, simulate, and test a PIM-based AFU using the IPU Platform F2000X-PL card with the SoC Attach FIM.
Execute the next commands to clone the BBB repositories.
# Create top level directory for AFU development\n$ mkdir OFS_BUILD_ROOT\n$ cd OFS_BUILD_ROOT\n$ export OFS_BUILD_ROOT=$PWD\n# Clone the ofs-platform-afu-bbb repository.\n$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/ofs-platform-afu-bbb.git\n$ cd ofs-platform-afu-bbb\n$ git checkout tags/ofs-2024.1-1\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n# Verify retrieval\n$ ls\nLICENSE plat_if_develop plat_if_release plat_if_tests README.md\n
The documentation in the ofs-platform-afu-bbb repository further addresses - The PIM concept. - The structure of the PIM-based AFU examples. - How to generate a release and configure the PIM. - How to connect an AFU to an FIM.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#26-build-or-download-the-relocatable-pr-build-tree","title":"2.6. Build or download the relocatable PR build tree","text":"A relocatable PR build tree is needed to build the AFU partial reconfiguration area for the intended FIM. The tree is relocatable and may be copied to a new location. It does not depend on files in the original FIM build.
You can use the IPU Platform F2000X-PL release package and download the PR build tree and FIM images, to develop your AFU. These are located at OFS-F2000X-PL release
Or you can build your own FIM and generate the PR build tree during the process.
To download and untar the pr_build_template:
$ cd $OFS_BUILD_ROOT\n$ wget https://github.com/OFS/ofs-f2000x-pl/releases/download/ofs-2024.1-1/f2000x-images_ofs-2024-2-1.tar.gz\n$ tar -zxvf f2000x-images_ofs-2024-2-1.tar.gz\n$ cd f2000x-images_ofs-2024-2-1/\n$ mkdir pr_build_template\n$ tar -zxvf pr_build_template.tar.gz -C ./pr_build_template\n$ cd pr_build_template\n$ export OPAE_PLATFORM_ROOT=$PWD\n
To build your own FIM and generate the PR build tree for the IPU Platform F2000X-PL platform, refer the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs and follow the Out-of-Tree PR FIM build flow. If you are using a different platform, refer to the Shell Developer Guide for your platform and follow the Out-of-Tree PR FIM build flow.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#27-download-fim-to-fpga","title":"2.7. Download FIM to FPGA","text":"The AFU requires that the FIM from which the AFU is derived be loaded onto the FPGA.
If you are using the IPU Platform F2000X-PL release package downloaded in the previous section, copy the associated FIM files to the SoC:
# On Development Host\n$ cd $OFS_BUILD_ROOT/f2000x-images_ofs-${{ env.COMMON_OFS_RELEASE_TAR }}/\n$ scp ofs_top_page1_unsigned_user1.bin <user>@<SoC IP address>:</remote/directory>\n$ scp ofs_top_page2_unsigned_user2.bin <user>@<SoC IP address>:</remote/directory>\n
If you are generating your own FIM, use the unsigned FPGA binary images from your FIM build and copy over to the SoC.
To downlaod the FIM to the IPU Platform F2000X-PL platform:
# On SoC\n$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin $ sudo fpgasupdate ofs_top_page2_unsigned_user2.bin $ sudo rsu fpga --page=user1
If you are using a different platform, refer to the documentation for your platform to download the FIM images onto your Agilex\u00ae SoC Attach Platform.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#28-set-up-required-environment-variables","title":"2.8. Set up required Environment Variables","text":"Set the required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks. You can create a simple script to set these variables and save time going forward.
# If not already done, export OFS_BUILD_ROOT to the top level directory for AFU development\n$ export OFS_BUILD_ROOT=<path to ofs build directory>\n\n# If not already done, export OPAE_PLATFORM_ROOT to the PR build tree directory\n$ export OPAE_PLATFORM_ROOT=<path to pr build tree>\n\n# If not already done, export OFS_PLATFORM_AFU_BBB to the clone of ofs-platform-afu-bbb repository which contains PIM files and AFU examples.\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu-bbb> # Quartus Tools\n# Note, QUARTUS_HOME is your Quartus installation directory, e.g. $QUARTUS_HOME/bin contains Quartus executable.\n$ export QUARTUS_HOME=<user_path>/intelFPGA_pro/23.4/quartus\n$ export QUARTUS_ROOTDIR=$QUARTUS_HOME\n$ export QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\n$ export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\n$ export IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\n$ export QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys\n$ export PATH=$QUARTUS_HOME/bin:$QSYS_ROOTDIR/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\n# OPAE SDK release\n$ export OPAE_SDK_REPO_BRANCH=release/2.12.0\n\n# OPAE and MPF libraries must either be on the default linker search paths or on both LIBRARY_PATH and LD_LIBRARY_PATH. \n$ export OPAE_LOC=/usr\n$ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH\n$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#3-compiling-an-afu","title":"3. Compiling an AFU","text":"In this section, you will use the relocatable PR build tree created in the previous steps from the FIM to compile an example PIM-based AFU. This section will be developed around the host_chan_mmio and hello_world AFU examples to showcase the synthesis of a PIM-based AFU.
The build steps presented below demonstrate the ease in building and running an actual AFU on the board. To successfully execute the instructions in this section, you must have set up your development environment and have a relocateable PR Build tree as instructed in section 2 of this document.
The build steps presented below demonstrate the ease in building and running an actual AFU on the board. To successfully execute the instructions in this section, you must have set up your development environment and have a relocateable PR Build tree as instructed in section 2 of this document.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#31-creating-the-afu-synthesis-environment","title":"3.1. Creating the AFU Synthesis Environment","text":"The PIM flow provides the script afu_synth_setup to create the synthesis environment to build the AFU examples. See how to use it below.
usage: afu_synth_setup [-h] -s SOURCES [-p PLATFORM] [-l LIB] [-f] dst\n\nGenerate a Quartus build environment for an AFU. A build environment is\ninstantiated from a release and then configured for the specified AFU. AFU\nsource files are specified in a text file that is parsed by rtl_src_config,\nwhich is part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA platform name.\n -l LIB, --lib LIB FPGA platform release hw/lib directory. If not\n specified, the environment variables OPAE_FPGA_HW_LIB\n and then BBS_LIB_PATH are checked.\n -f, --force Overwrite target directory if it exists.\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#32-building-and-running-host_chan_mmio-example-afu","title":"3.2. Building and Running host_chan_mmio example AFU","text":"The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using Avalon and AXI interfaces. However, this guide will use the AXI version of the host_chan_mmio AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the actual AFU hardware.
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\n
Execute afu_synth_setup as follows to create the synthesis environment for a host_chan_mmio AFU that fits the SoC Attach FIM previously constructed.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_dev\n
Now, move into the synthesis environment afu_dev directory just created. From there, execute the afu_synth command. The successful completion of the command will produce the host_chan_mmio.gbs file under the synthesis environment directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev.
$ cd afu_dev\n$ $OPAE_PLATFORM_ROOT/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating host_chan_mmio.gbs\n==================================\n...\n...\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'host_chan_mmio.gbs'\nDesign meets timing\n===========================================================================\n
The previous output indicates the successful compilation of the AFU and the compliance with the timing requirements. Analyze the reports generated in case the design does not meet timing. The timing reports are stored in the directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev/build/syn/syn_top/output_files/timing_report.
Once the compilation finishes successfully, load the new host_chan_mmio.gbs bitstream file into the partial reconfiguration region of the target IPU Platform F2000X-PL board. Keep in mind, that the loaded image is dynamic - this image is not stored in flash and if the card is power cycled, then the PR region is re-loaded with the default AFU.
To load the image, perform the following steps:
# On Development Host\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev\n# Copy FIM files to SoC\n$ scp host_chan_mmio.gbs <user>@<SoC IP address>:</remote/directory>\n
# On SoC\n$ cd </remote/directory>\n$ fpgasupdate host_chan_mmio.gbs [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\n
Set up your board to work with the newly loaded AFU.
# On SoC\n# Verify PCIe b.d.f\n# For the following example, the F2000x SKU2 PCIe b:d.f is 15:00.0,\n# however this may be different in your system\n$ fpgainfo fme\nIntel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.2.4\nBoard Management Controller Build version: 1.2.4\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n...\n\n# Create the Virtual Functions (VFs) provided by the FIM, the default FIM has 3 VFs\n$ pci_device 15:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s 15:00\n15:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n15:00.1 Processing accelerators: Intel Corporation Device bccf\n15:00.2 Processing accelerators: Intel Corporation Device bccf\n15:00.3 Processing accelerators: Intel Corporation Device bccf\n\n# Bind VFs to VFIO driver. \n$ opae.io init -d 0000:15:00.1 opae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.1 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.1 is 52\n$ opae.io init -d 0000:15:00.2\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.2 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.2 is 53\n$ opae.io init -d 0000:15:00.3\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.3 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.3 is 54\n# Verify the new AFU is loaded. The host_chan_mmio AFU GUID is \"76d7ae9c-f66b-461f-816a-5428bcebdbc5\".\n$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x6015000000000000\nPCIe s:b:d.f : 0000:15:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0x4015000000000000\nPCIe s:b:d.f : 0000:15:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0x2015000000000000\nPCIe s:b:d.f : 0000:15:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : 76d7ae9c-f66b-461f-816a-5428bcebdbc5\n
Now, navigate to the directory of the host_chan_mmio AFU containing the host application's source code, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw. Once there, compile the host_chan_mmio host application and execute it on the host server to excercise the functionality of the AFU.
Note: If OPAE SDK libraries were not installed in the default systems directories under /usr, you need to set the OPAE_LOC, LIBRARY_PATH, and LD_LIBRARY_PATH environment variables to the custom locations where the OPAE SDK libraries were installed.
# On Development Host, move to the sw directory of the the host_chan_mmio AFU. This directory holds the source for the host application.\n$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw\n$ make\n# Copy application to SoC\n$ scp host_chan_mmio <user>@<SoC IP address>:</remote/directory>\n
# On SoC, Run the application\n$ cd </remote/directory>\n$ ./host_chan_mmio\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#33-building-and-running-the-hello_world-example-afu","title":"3.3. Building and running the hello_world example AFU","text":"The platform-independent examples-afu repository also provides some interesting example AFUs. In this section, you will compile and execute the PIM based hello_world AFU. The RTL of the hello_world AFU receives from the host application an address via memory mapped I/O (MMIO) write and generates a DMA write to the memory line at that address. The content written to memory is the string \"Hello world!\". The host application spins, waiting for the memory line to be updated. Once available, the software prints out the string.
The hello_world example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using CCIP, Avalon, and AXI interfaces. However, this guide will use the AXI version of the AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the AFU hardware.
hello_world\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u251c\u2500\u2500 ccip\n\u2502 \u2502 \u251c\u2500\u2500 hello_world_ccip.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu.sv\n\u2502 \u2502 \u2514\u2500\u2500 sources.txt\n\u2502 \u2514\u2500\u2500 hello_world.json\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 hello_world\n \u251c\u2500\u2500 hello_world.c\n \u251c\u2500\u2500 Makefile\n \u2514\u2500\u2500 obj\n \u251c\u2500\u2500 afu_json_info.h\n \u2514\u2500\u2500 hello_world.o\n
The following instructions can be used to compile other AFU samples accompanying this repository.
If not done already, download and clone the examples-afu repository.
$ cd $OFS_BUILD_ROOT $ git clone https://github.com/OFS/examples-afu.git\n$ cd examples-afu\n$ git checkout tags/ofs-2024.1-1\n
Compile the hello_word sample AFU.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_synth_setup --source hw/rtl/axi/sources.txt afu_dev\n$ cd afu_dev\n$ ${OPAE_PLATFORM_ROOT}/bin/afu_synth\nCompiling ofs_top ofs_pr_afu\nGenerating hello_world.gbs\n==================================\n.\n.\n.\n===========================================================================\nPR AFU compilation complete\nAFU gbs file is 'hello_world.gbs'\nDesign meets timing\n===========================================================================\n
To test the AFU in actual hardware, load the hello_world.gbs to the IPU Platform F2000X-PL card. For this step to be successful, the SoC Attach FIM must have already been loaded to the IPU Platform F2000X-PL card following the steps described in Section 2 of this document.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_dev/\n# Copy FIM files to SoC\n$ scp hello_world.gbs <user>@<SoC IP address>:</remote/directory>\n
# On SoC\n$ cd </remote/directory>\n$ fpgasupdate hello_world.gbs [2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.\n[2022-04-15 20:22:19.75] [INFO ] Partial Reconfiguration OK\n[2022-04-15 20:22:19.75] [INFO ] Total time: 0:00:00.90\n
Set up your IPU Platform F2000X-PL board to work with the newly loaded hello_world.gbs file.
# On SoC\n# Verify PCIe b.d.f\n# For the following example, the F2000x SKU2 PCIe b:d.f is 15:00.0,\n# however this may be different in your system\n$ fpgainfo fme\nIntel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.2.4\nBoard Management Controller Build version: 1.2.4\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n...\n\n# Create the Virtual Functions (VFs) provided by the FIM, the default FIM has 3 VFs\n$ pci_device 15:00.0 vf 3\n# Verify the VFs have been added (device id: bccf)\n$ lspci -s 15:00\n15:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n15:00.1 Processing accelerators: Intel Corporation Device bccf\n15:00.2 Processing accelerators: Intel Corporation Device bccf\n15:00.3 Processing accelerators: Intel Corporation Device bccf\n\n# Bind VFs to VFIO driver.\n$ opae.io init -d 0000:15:00.1\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.1 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.1 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.1 is 52\n$ opae.io init -d 0000:15:00.2\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.2 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.2 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.2 is 53\n$ opae.io init -d 0000:15:00.3\nopae.io 0.2.5\nUnbinding (0x8086,0xbccf) at 0000:15:00.3 from dfl-pci\nBinding (0x8086,0xbccf) at 0000:15:00.3 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:15:00.3 is 54\n# Verify the new AFU is loaded. The hello_world AFU GUID is \"c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\".\n$ fpgainfo port\n//****** PORT ******//\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0x6015000000000000\nPCIe s:b:d.f : 0000:15:00.3\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0210-000000000000\n//****** PORT ******//\nObject Id : 0x4015000000000000\nPCIe s:b:d.f : 0000:15:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : d15ab1ed-0000-0000-0110-000000000000\n//****** PORT ******//\nObject Id : 0x2015000000000000\nPCIe s:b:d.f : 0000:15:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nAccelerator GUID : c6aa954a-9b91-4a37-abc1-1d9f0709dcc3\n
Compile and execute the host application of the hello_world AFU. You should see the application outputs the \"Hello world!\" message in the terminal.
# On Development Host, move to the sw directory of the hello_world AFU and build application\n$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n$ make\n# Copy application to SoC\n$ scp hello_world <user>@<SoC IP address>:</remote/directory>\n
# On SoC, Run the application\n$ cd </remote/directory>\n$ ./hello_world\nHello world!\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#34-modify-the-afu-user-clocks-frequency","title":"3.4. Modify the AFU user clocks frequency","text":"AFU user clocks are currently not supported in F2000x base FIM configuration.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#4-simulating-an-afu-using-ase","title":"4. Simulating an AFU using ASE","text":"The Application Simulation Environment (ASE) is a hardware/software co-simulation environment for your AFU. See diagram below illustrating ASE operation:
ASE uses the simulator Direct Programming Interface (DPI) to provide HW/SW connectivity. The PCIe connection to the AFU under testing is emulated with a transactional model.
The following list describes ASE operation:
- Attempts to replicate the transactions that will be seen in real system.
- Provides a memory model to AFU, so illegal memory accesses can be identified early.
- Not a cache simulator.
- Does not guarantee synthesizability or timing closure.
- Does not model system latency.
- No administrator privileges are needed to run ASE. All code is user level.
The remainder of this section is a tutorial providing the steps on how to run ASE with either Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae using an example AFU and the AFU build tree previously created in this guide.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#41-set-up-steps-to-run-ase","title":"4.1. Set Up Steps to Run ASE","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#411-install-opae-sdk","title":"4.1.1. Install OPAE SDK","text":"The F2000x SKU2 card requires 2.12.0-5. Follow the instructions in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs to build and install the required OPAE SDK for the IPU Platform F2000X-PL. Make sure to check out the cloned repository to tag 2.12.0-5 and branch release/2.12.0.
$ git checkout tags/2.12.0-5 -b release/2.12.0\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#412-install-ase-tools","title":"4.1.2 Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK.
-
If not done already, set the environment variables as described in section, Set Up AFU Development Environment.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n$ git checkout tags/2.12.0-1 -b release/2.12.0
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#413-setup-required-ase-environment-variables","title":"4.1.3. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ cd /usr/bin\n$ export PATH=$PWD:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ cd /usr/lib\n$ export LIBRARY_PATH=$PWD\n$ cd /usr/lib64\n$ export LD_LIBRARY_PATH=$PWD\n$ cd $OFS_BUILD_ROOT/ofs-platform-afu-bbb\n$ export OFS_PLATFORM_AFU_BBB=$PWD\n## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to Modelsim installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#42-simulating-the-host_chan_mmio-afu","title":"4.2. Simulating the host_chan_mmio AFU","text":"The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files:
host_chan_mmio\n\u251c\u2500\u2500 hw\n\u2502 \u2514\u2500\u2500 rtl\n\u2502 \u251c\u2500\u2500 avalon\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_avalon.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_avalon_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_avalon.sv\n\u2502 \u251c\u2500\u2500 axi\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 afu_axi.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi512.sv\n\u2502 \u2502 \u251c\u2500\u2500 ofs_plat_afu_axi_from_ccip.sv\n\u2502 \u2502 \u2514\u2500\u2500 ofs_plat_afu_axi.sv\n\u2502 \u251c\u2500\u2500 host_chan_mmio.json\n\u2502 \u251c\u2500\u2500 test_mmio_avalon0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon1.txt\n\u2502 \u251c\u2500\u2500 test_mmio_avalon2_512rw.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi0_from_ccip.txt\n\u2502 \u251c\u2500\u2500 test_mmio_axi1.txt\n\u2502 \u2514\u2500\u2500 test_mmio_axi2_512rw.txt\n\u2514\u2500\u2500 sw\n \u251c\u2500\u2500 main.c\n \u251c\u2500\u2500 Makefile\n
This example AFU contains examples using both Avalon and AXI interface buses. This guide will use the AXI version of the host_chan_mmio AFU.
ASE uses client-server application architecture to deliver hardware/software co-simulation. You require one shell for the hardware based simulation and another shell where the software application is running. The hardware is started first with a simulation compilation and simulator startup script, once the simulator has loaded the design, it will wait until the software process starts. Once the software process starts, the simulator proceeds. Transaction logging and waveform capture is performed.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#421-set-up-and-run-the-hw-simulation-process","title":"4.2.1 Set Up and Run the HW Simulation Process","text":"You will run the afu_sim_setup script to create the scripts for running the ASE environment. The afu_sim_setup script has the following usage:
usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n [-f] [--ase-mode ASE_MODE] [--ase-verbose]\n dst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n -s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\n list into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\n Default simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
Run afu_sim_setup to create the ASE simulation environment for the host_chan_mmio example AFU. The '-t VCS' option indicates to prepare the ASE simulation environment for Synopsys\u00ae VCS\u00ae.
$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio\n$ afu_sim_setup -s ./hw/rtl/test_mmio_axi1.txt -t VCS afu_sim\n\nCopying ASE from /opae-sdk/install-opae-sdk/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /ofs-f2000x-pl/work_pr/build_tree/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup creates the ASE scripts in the directory host_chan_mmio_sim where the afu_sim_setup script was run. Start the simulator as shown below:
$ cd afu_sim\n$ make\n$ make sim\n
This process launches the AFU hardware simulator. Before moving to the next section, pay attention to the simulator output highlighted in the image below.
The simulation artifacts are stored in host_chan_mmio/work and consist of:
log_ase_events.tsv\nlog_ofs_plat_host_chan.tsv \nlog_ofs_plat_local_mem.tsv \nlog_pf_vf_mux_A.tsv \nlog_pf_vf_mux_B.tsv \n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#422-set-up-and-run-the-sw-process","title":"4.2.2 Set Up and Run the SW Process","text":"Open an additional shell to build and run the host application that communicates with the actual AFU hardware. Set up the same environment variable you have set up in the shell you have been working on until this point.
Additionally, as indicated by the hardware simulator output that is currently executing in the \"simulator shell\", copy and paste the line \"export ASE_WORKDIR=...\", into the new \"software shell\". See the last image of the previous section.
$ export ASE_WORKDIR=$OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_sim/work\n
Then, go to the sw directory of the host_chan_mmio AFU example to compile the host application. $ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/sw $ make\n\nafu_json_mgr json-info --afu-json=../hw/rtl/host_chan_mmio.json --c-hdr=obj/afu_json_info.h\nWriting obj/afu_json_info.h\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c main.c -o obj/main.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c test_host_chan_mmio.c -o obj/test_host_chan_mmio.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/connect.c -o obj/connect.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/csr_mgr.c -o obj/csr_mgr.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/hash32.c -o obj/hash32.o\ncc -g -O2 -std=gnu99 -fstack-protector -fPIE -fPIC -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I../../common/sw -I./obj -c ../../common/sw/test_data.c -o obj/test_data.o\ncc -o host_chan_mmio obj/main.o obj/test_host_chan_mmio.o obj/connect.o obj/csr_mgr.o obj/hash32.o obj/test_data.o -z noexecstack -z relro -z now -pie -luuid -lopae-c-ase\n
Now, launch the host application to exercise the AFU hardware running on the simulator shell. The next image shows the AFU hardware simulation process on the left side shell. The right hand shell shows the host application's output of a successful simulation.
$ with_ase ./host_chan_mmio\n [APP] Initializing simulation session ...\nRunning in ASE mode\nAFU ID: 76d7ae9cf66b461f 816a5428bcebdbc5\nAFU MMIO interface: AXI Lite\nAFU MMIO read bus width: 64 bits\n512 bit MMIO write supported: yes\nAFU pClk frequency: 470 MHz\n\nTesting 32 bit MMIO reads:\n PASS - 4 tests\n\nTesting 32 bit MMIO writes:\n PASS - 5 tests\n\nTesting 64 bit MMIO writes:\n PASS - 5 tests\n\nTesting 512 bit MMIO writes:\n PASS\n [APP] Deinitializing simulation session\n [APP] Took 1,003,771,568 nsec\n [APP] Session ended\n
Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
$ make wave\n
This brings up the Synopsys\u00ae VCS\u00ae simulator GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | afu , as shown below.
Right click on the afu (afu) entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#43-simulating-the-hello_world-afu","title":"4.3 Simulating the hello_world AFU","text":"In this section you will quickly simulate the PIM-based hello_world sample AFU accompanying the examples-afu repository.
-
Set the environment variables as described in section 4.1. Set Up Steps to Run ASE.
-
Prepare an RTL simulation environment for the AXI version of the hello_world AFU.
Simulation with ASE requires two software processes, one to simulate the AFU RTL and the other to run the host software that excercises the AFU. To construct an RTL simulation environment under the directory simulation, execute the following.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world\n$ afu_sim_setup -s ./hw/rtl/axi/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/<user_area>/ofs-f2000x-pl/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
The afu_sim_setup script constructs an ASE environment in the hello_world_sim subdirectory. If the command fails, confirm that the path to the afu_sim_setup is on your PATH environment variable (in the OPAE SDK bin directory) and that your Python version is at least 2.7.
- Build and execute the AFU RTL simulator.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_sim\n$ make\n$ make sim
The previous commands will build and run the Synopsys\u00ae VCS\u00ae RTL simulator, which prints a message saying it is ready for simulation. The simulation process also prints a message instructing you to set the ASE_WORKDIR environment variable in a second shell.
-
Open a second shell where you will build and execute the host software. In this new \"software shell\", set up the environment variables you have set up so far in the \"hardware simulation\" shell.
-
Also, set the ASE_WORKDIR environment variable following the instructions given in the \"hardware simulation\" shell.
$ export ASE_WORKDIR=$OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/afu_sim/work\n
-
Then, move to the sw directory of the hello_world AFU sample to build the host software.
$ cd $OFS_BUILD_ROOT/examples-afu/tutorial/afu_types/01_pim_ifc/hello_world/sw\n$ make
-
Run the hello_world host application to resume the work of the RTL simulation. The host software process and the RTL simulation execute in lockstep. If successful, you should see the Hello world! output.
$ with_ase ./hello_world\n\n[APP] Initializing simulation session ...\nHello world!\n [APP] Deinitializing simulation session\n [APP] Took 43,978,424 nsec\n [APP] Session ended\n
The image below shows the simulation of the AFU hardware and the execution of the host application side-by-side.
- Finally, on the hardware simulation shell, you can view the wave forms by invoking the following command.
make wave\n
This brings up the DVE GUI and loads the simulation waveform files. Use the Hierarchy window to navigate to the afu instance located under, ase_top | ase_top_plat | ase_afu_main_pcie_ss | ase_afu_main_emul | afu_main | port_afu_instances | ofs_plat_afu | hello_afu, as shown below.
Right click on the hello_afu entry to display the drop-down menu. Then, click on Add to Waves | New Wave View to display the following waveforms window.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#5-adding-remote-signal-tap-logic-analyzer-to-debug-the-afu","title":"5. Adding Remote Signal Tap Logic Analyzer to debug the AFU","text":"Remote Signal Tap is currently not supported in F2000x base FIM configuration.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#6-disabling-the-flr-function-level-reset-during-afu-debugging","title":"6. Disabling the FLR (Function Level Reset) during AFU Debugging","text":"The vfio-pci driver implementation will automatically issue an FLR (Function Level Reset) signal every time a new host application is executed. This signal is triggered whenever an application opens a /dev/vfio* file and is expected behavior for the vfio driver architecture.
You may also encounter issues while debugging an AFU when executing the OPAE SDK tool opae.io with peek/poke subcommands, which will automatically set register values if they are connected to a reset. The OPAE SDK function fpgaReset() will also not accept devices bound to the vfio-pci driver. Both of these behaviors can be worked around if they are not desired.
You can use the following steps to enable / disable FLR for a specific device bound to the vfio-pci driver. In this example we will use an OFS enabled PCIe device at BDF af:00.0, and will disable FLR on a VF at address af:00.5.
Disable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"\" > reset_method\ncat reset_method\n
Enable FLR:
cd /sys/bus/pci/devices/0000:ae:00.0/0000:af:00.5\necho \"flr\" > reset_method\ncat reset_method\n
If you wish to manually reset your currently configured AFU without resetting the entire FIM, you can use the OPAE SDK function fpgaEnumerate(). This will issue a reset on the AFU's VFIO DEVICE_GROUP. To avoid issuing an FLR to the entire FIM, you need to call this function after disabling FLR as shown above.
If you wish to debug your AFU's register space without changing any of its register values using opae.io, you need to execute a opae.io compatible python script. An example application is shown below:
opae.io --version\nopae.io 1.0.0\n\nsudo opae.io init -d BDF $USER\nopae.io script sample.py\nValue@0x0 = 0x4000000010000000\nValue@0x12060 = 100\n
Sample.py contents:
import sys\ndef main():\n# Check opae.io initialization\nif the_region is None :\nprint(\"\\'opae.io\\' initialization has not been performed, please bind the device in question to vfio-pci.\")\nsys.exit(1)\nv = the_region.read64(0x0)\nprint(\"Value@0x0 = 0x{:016X}\".format(v))\nthe_region.write32(0x12060,100)\nv = the_region.read32(0x12060)\nprint(\"Value@0x12060 = {:d}\".format(v))\n####################################\nif __name__ == \"__main__\":\nmain()\n
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#7-how-to-modify-the-pfvf-mux-configuration","title":"7. How to modify the PF/VF MUX configuration","text":"For information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/dev_guides/afu_dev/ug_dev_afu_ofs_f2000x/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/","title":"Shell Developer Guide: Agilex\u00ae 7 SoC Attach: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1-introduction","title":"1 Introduction","text":""},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#11-about-this-document","title":"1.1 About This Document","text":"This document serves as a guide for OFS Agilex\u00ae 7 SoC Attach developers targeting the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL. The following topics are covered in this guide:
- Compiling the OFS Agilex\u00ae 7 SoC Attach FIM design
- Simulating the OFS Agilex\u00ae 7 SoC Attach FIM design
- Customizing the OFS Agilex\u00ae 7 SoC Attach FIM design
- Configuring the FPGA with an OFS Agilex\u00ae 7 SoC Attach FIM design
This document uses the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL as the platform to illustrate key points and demonstrate how to extend the capabilities provided in OFS. The demonstration steps serve as a tutorial for the development of your OFS knowledge.
This document covers OFS architecture lightly. For more details on the OFS architecture, please see Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#111-knowledge-prerequisites","title":"1.1.1 Knowledge Prerequisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex\u00ae 7 SoC Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus Prime Pro Version 23.4, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Signal Tap Logic Analyzer tool software.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex\u00ae 7 SoC Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex\u00ae 7 SoC Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":"Agilex\u00ae 7 SoC Attach OFS supports the following features.
FIM BASE IPU Platform F2000X-PL f2000x PCIe Configuration Host: PCIe Gen4x16SoC: PCIe Gen4x16 SR-IOV support Host: 2 PFs, No VFsSoC: 1 PFs, 3 VFs AXI ST datapath 512b @ 470MHz Transceiver Subsystem Configuration 2x4x25G The FIM also integrates:
- SoC AFU and Host AFU
- Exercisers demonstrating PCIe, external memory, and Ethernet interfaces
- FME CSR
- Remote Signal Tap
- Partial Reconfiguration of the SoC AFU
The Host exercisers are provided for the quick evaluation of the FIM and can be leveraged for the verification of the platform's functionality and capabilities. The host exercisers can be removed by the designer to release FPGA real estate to accommodate new workload functions. To compile the FIM without host exercisers go to How to compile the FIM in preparation for designing your AFU.
OFS is extensible to meet the needs of a broad set of customer applications. The general use cases listed below are examples where the OFS base design is easily extended to build a custom FIM:
- Use OFS design example as-is
- Porting the code to another platform that is identical to OFS reference platform changing targeted FPGA device and pinout
- Change I/O assignments without changing design
- Update the configuration of peripheral IP in OFS design example, not affecting FIM architecture
- External memory settings
- Ethernet Subsystem analog settings
- Remove/update peripheral feature in OFS design example, not affecting FIM architecture
- External memory speed/width change
- Change number of VFs supported
- Add new features as an extension to OFS design example, not affecting the FIM architecture
- Add/remove external memory interface to the design
- Add/remove user clocks for the AFU
- Add/remove IP to the design with connection to the AFU
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1211-top-level-fpga","title":"1.2.1.1 Top Level FPGA","text":"OFS separates the FPGA design into two areas: FPGA Interface Manager (FIM) and workload (or Acceleration Function Unit) as shown in the figure below:
As can be seen in this diagram, the OFS FPGA structure has a natural separation into two distinct areas:
- FPGA Interface Manager (FIM or sometimes called the \"the shell\") containing:
- FPGA external interfaces and IP cores (e.g. Ethernet, DDR-4, PCIe, etc)
- PLLs/resets
- FPGA - Board management infrastructure
- Interface to Acceleration Function Unit (AFU)
- Acceleration Function Unit (\"the workload\")
- Uses the FIM interfaces to perform useful work inside the FPGA
- Contains logic supporting partial reconfiguration
- Remote Signal Tap core for remote debugging of SoC AFU PR region
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex\u00ae 7 SoC Attach design are listed below.
- Host interface
- PCIe Gen4 x 16
- SoC Interface
- PCIe Gen4 x 16
- Network interface
- 2 - QSFP28/56 cages
- Eight Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels of 25G Ethernet interfacing to an E-tile Ethernet Subsystem.
- External Memory - DDR4
- Four Fabric DDR4-2400 banks - 4 GB organized as 1Gb x 32 with 1 Gb x 8 ECC (ECC login not implemented in default FIM)
- Board Management
- SPI interface
- FPGA configuration
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex Agilex\u00ae 7 SoC Attach f2000x FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem Intel FPGA PCI Express Subsystem IP User Guide N/A Memory Subsystem Intel FPGA Memory Subsystem IP User Guide 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
The host control and data flow is shown in the diagram below:
The control and data paths are composed of the following:
- Host Interface Adapter (PCIe)
- SoC Interface Adapter (PCIe)
- Low Performance Peripherals
- Slow speed peripherals (JTAG, I2C, Smbus, etc)
- Management peripherals (FME)
- High Performance Peripherals
- Memory peripherals
- Acceleration Function peripherals (eg. AFUs)
- HPS Peripheral
- Fabrics
- Peripheral Fabric (multi drop)
- AFU Streaming fabric (point to point)
Peripherals are connected to one another using AXI, either:
- Via the peripheral fabric (AXI4-Lite, multi drop)
- Via the AFU streaming fabric (AXI-S, point to point)
Peripherals are presented to software as:
- OFS managed peripherals that implement DFH CSR structure.
- Native driver managed peripherals (i.e. Exposed via an independent PF, VF)
The peripherals connected to the peripheral fabric are primarily OPAE managed resources, whereas the peripherals connected to the AFU are \u201cprimarily\u201d managed by native OS drivers. The word \u201cprimarily\u201d is used since the AFU is not mandated to expose all its peripherals to OPAE.
OFS uses a defined set of CSRs to expose the functionality of the FPGA to the host software. These registers are described in Open FPGA Stack Reference Manual - MMIO Regions section.
If you make changes to the FIM that affect the software operation, then OFS provides a mechanism to communicate that information to the proper software driver that works with your new hardware. The Device Feature Header (DFH) structure is followed to provide compatibility with OPAE software. Please see FPGA Device Feature List (DFL) Framework Overview for a description of DFL operation from the driver perspective.
In the default design, the SoC and Host AFUs are isolated from each other. You must develop mechanisms for Host - SoC communication if desired.
Note: The default configuration of the Board Peripheral Fabric, there is a connection from the Host Interface to the PMCI-SS, however the PMCI-SS is not in the Host DFL, and is not discovered by Host SW by default. If you want to guarantee that the Host can not access the PMCI-SS, and by extension the Board BMC, you must implement a filtering mechanism, for example, in the Host ST2MM module to prevent access to the PMCI-SS address space.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workloads in the OFS Agilex\u00ae 7 SoC Attach f2000x FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI f2000x: Used to exercise and characterize HSSI interfaces. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex Agilex\u00ae 7 SoC Attach f2000x FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the SoC/Host software and AFU and board peripherals. The following tables describe the address mapping of the APF and BPF for both the SoC and the Host.
Table: SoC APF Address Map
Address Size (Bytes) Feature 0x00_0000 - 0x0F_FFFF 1024K Board Peripherals (See SoC BPF Address Map table) 0x10_0000 - 0x10_FFFF 64K ST2MM 0x11_0000 - 0x12_FFFF 128K Reserved 0x13_0000 - 0x13_FFFF 64K PR Gasket: 4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x14_0000 - 0x14_FFFF 64K AFU Error Reporting Table: Host APF Address Map
Address Size (Bytes) Feature 0x00_0000 - 0x0F_FFFF 1024K Board Peripherals (See Host BPF Address Map table) 0x10_0000 - 0x10_FFFF 64K ST2MM 0x11_0000 - 0x13_FFFF 192K Reserved 0x14_0000 - 0x14_FFFF 64K AFU Error Reporting Table: SoC BPF Address Map
Address Size (Bytes) Feature 0x0_0000 - 0x0_FFFF 64K FME 0x1_0000 - 0x1_0FFF 4K SoC PCIe IF 0x1_1000 - 0x1_1FFF 4K Reserved 0x1_2000 - 0x1_2FFF 4K QSFP0 0x1_3000 - 0x1_3FFF 4K QSFP1 0x1_4000 - 0x1_4FFF 4K Ethernet Sub-System 0x1_5000 - 0x1_5FFF 4K Memory Sub-System 0x8_0000 - 0xF_FFFF 512K PMCI Controller Table: Host BPF Address Map
Address Size (Bytes) Feature 0x0_0000 - 0x0_0FFF 4K Host PCIe IF 0x8_0000 - 0xF_FFFF 512K PMCI Controller"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customization Examples Table lists the general user flows for OFS Agilex\u00ae 7 SoC Attach f2000x FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customization Examples
Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Modify and run UVM tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Memory Sub-System Using IP Presets With OFSS Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Modify the Ethernet Sub-System Without HSSI OFSS Set up JTAG Program the FPGA via JTAG Program the FPGA via RSU"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Refer to the following guides for instructions on setting up an OFS deployment environment.
- Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL
- Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
The recommended Best Known Configuration (BKC) operating system for development of the OFS FIM is RedHatEnterprise Linux\u00ae (RHEL) 8.6, which is the assumed operating system for this developer guide.
The recommended development environment requires the following:
- Workstation or server with a Quartus Prime Pro Version 23.4 installed on a Quartus Prime Pro-supported Linux distribution. See Operating System Support. The Linux distribution known to work with this version of RedHatEnterprise Linux\u00ae (RHEL) 8.6 . Note, Windows is not supported.
- Compilation targeting Agilex\u00ae 7 FPGA devices requires a minimum of 64 GB of RAM.
- Simulation of lower level functionality (not chip level) is supported by Synopsys\u00ae VCS and Mentor Graphics\u00ae QuestaSim SystemVerilog simulators.
- Simulation of chip level requires Synopsys VCS and VIP
You may modify the build scripts and pin files to target different boards with Agilex\u00ae 7 FPGA devices.
Testing the Agilex\u00ae 7 SoC Attach on the IPU Platform F2000X-PL hardware requires a deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Development Operating System RedHatEnterprise Linux\u00ae (RHEL) 8.6 N/A Quartus Prime Software Quartus Prime Pro Version 23.4 for Linux + Patches 0.17 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A git 1.8.3.1 PERL 5.8.8 FIM Source Files ofs-2024.1-1 Section 1.3.2.1"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use Ubuntu 22.04 LTS for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.17.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/23.4/quartus/bin/quartus\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"OFS provides a framework of FPGA synthesizable code, simulation environment, and synthesis/simulation scripts. FIM designers can use the provided code as-is, modify the provided code, or add new code to meet your specific product requirements. Instructions for compiling the existing design is given in the FIM Compilation section, while instructions for customizing the default design can be found in the FIM Customization section.
The source files for the OFS Agilex\u00ae 7 SoC Attach FIM are provided in the following repository: https://github.com/OFS/ofs-f2000x-pl/releases/tag/ofs-2024.1-1.
Some essential directories in the repository are described as follows:
| ipss // Contains files for IP Sub-Systems\n| | hssi // Contains source files for HSSI Sub-System\n| | mem // Contains source files for HSSI Sub-System\n| | pcie // Contains source files for PCIe Sub-System\n| | pmci // Contains source files for PMCI Sub-System\n| | qsfp // Contains source files for QSFP Sub-System\n| ofs-common // Contains files which are common across OFS platforms\n| | scripts // Contains common scripts\n| | src // Contains common source files, including host exercisers\n| | tools // Contains common tools files\n| | verification // Contains common UVM files\n| sim // Contains simulation files\n| | bfm\n| | common\n| | scripts\n| | unit_test // Contains files for all unit tests\n| src // Contains source files for Agilex Agilex\u00ae 7 SoC Attach FIM\n| | afu_top // Contains top-level source files for AFU\n| | includes // Contains source file header files\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | top // Contains top-level source files, including design top module\n| syn // Contains files related to design synthesis\n| | scripts\n| | setup // Contains setup files, including pin constraints and location constraints\n| | syn_top // Contains Quartus project files\n| tools\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| verification // Contains files for UVM testing\n| | scripts\n| | testbench\n| | tests\n| | unit_tb\n| | verifplan\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 SoC Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-f2000x-pl.git\n
-
Check out the correct tag of the repository
cd ofs-f2000x-pl\ngit checkout --recurse-submodules tags/ofs-2024.1-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.1-1)\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-f2000x-pl\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.F2000X_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.F2000X_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.12.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 23.4 for Linux with Agilex\u00ae 7 FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions. Quartus Prime Pro Version 23.4 is the currently verified version of Quartus Prime Pro used for building the FIM and AFU images. Porting to newer versions of Quartus Prime Pro may be performed by developers, however, you will need to verify operation.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 23.4 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
Clone the ofs-f2000x-pl repository if not already cloned. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $OFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.17. All required patches are provided in the Assets of the OFS FIM Release: https://github.com/OFS/ofs-f2000x-pl/releases/tag/ofs-2024.1-1
-
Extract and unzip the patch-agx7-2024-1.tar.gz file.
tar -xvzf patch-agx7-2024-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-23.4-0.17-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This section describes the theory behind FIM compilation.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config> Used to modify IP, such as the PCIe SS, using .ofss configuration files. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk | f2000x Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
If you wish to compile the f2000x FIM using the Quartus Prime Pro GUI, you must at least run the setup portion of the build_top.sh script before compiling with the GUI. For instructions on compiling the FIM using the Quartus GUI, refer to the Compiling the OFS FIM Using Quartus GUI section.
The build scripts included with OFS are verified to run in a bash shell. Other shells have not been tested. The full build script typically takes around 3 hours to complete.
The build script copies the ipss, ofs-common, sim, src,syn and tools directories to the specified work directory and then these copied files are used in the Quartus Prime Pro compilation process.
Some key output directories are described in the following table:
Directory Description $OFS_ROOTDIR/<WORK_DIR>/syn/syn_top Quartus Prime Pro project (ofs_top.qpf) and other Quartus Prime Pro specific files $OFS_ROOTDIR/<WORK_DIR>/syn/syn_top/output_files Directory with build reports and FPGA programming files The build script will run PACSign (if installed) and create an unsigned FPGA programming files for both user1 and user2 locations of the f2000x FPGA flash. Please note, if the f2000x has the root entry hash key loaded, then PACsign must be run to add the proper key to the FPGA binary file.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script can use OFSS files to easily customize design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building FIMs; it is not required if the source files are already configured as desired. The Provided OFSS Files table below describes the pre-made OFSS files for the f2000x that can be found in the $OFS_ROOTDIR/tools/ofss_config directory.
Table: Provided OFSS Files
OFSS File Name Location Type Description f2000x.ofss $OFS_ROOTDIR/tools/ofss_config Top Top level OFSS file which includes OFS, PCIe Host, PCIe SoC, IOPLL, and Memory OFSS files. f2000x_base.ofss $OFS_ROOTDIR/tools/ofss_config ofs Defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the f2000x Host pcie_soc.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the f2000x SoC iopll.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as f2000x hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE CAUI-4 hssi_4x100_caui2.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE CAUI-2 There can typically be three sections contained within an OFSS file.
-
[include]
- This section of an OFSS file contains elements separated by a newline, where each element is the path to an OFSS file that is to be included for configuration by the OFSS Configuration Tool. Ensure that any environment variables (e.g.
$OFS_ROOTDIR) is set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter
-
[ip]
- This section of an OFSS file contains a key value pair that allows the OFSS Config tool to determine which IP configuration is being passed in. The currently supported values of IP are
ofs, iopll, pcie, memory, and hssi.
-
[settings]
- This section of an OFSS file contains IP specific settings. Refer to an existing IP OFSS file to see what IP settings are set. For the IP type
ofss, the settings will be information of the OFS device (platform, family, fim, part #, device_id)
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2121-platform-ofss-file","title":"2.1.2.1 Platform OFSS File","text":"The <platform>.ofss file (e.g. $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss) is the platform level OFSS wrapper file. This is typically the OFSS file that is provided to the build script. It only contains an include section which lists all other OFSS files that are to be used when the <platform>.ofss file is passed to the build script.
The generic structure of a <platform>.ofss file is as follows:
[include]\n<PATH_TO_PLATFORM_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2122-ofs-ip-ofss-file","title":"2.1.2.2 OFS IP OFSS File","text":"An OFSS file with IP type ofs (e.g. $OFS_ROOTDIR/tools/ofss_config/f2000x_base.ofss) contains board specific information for the target board.
The default configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Defaults
Section Parameter f2000x Default Value [ip] type ofs [settings] platform f2000x family agilex fim base_x16 part AGFC023R25A2E2VR0 device_id 6100"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2123-pcie-ip-ofss-file","title":"2.1.2.3 PCIe IP OFSS File","text":"An OFSS file with IP type pcie (e.g. $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss) is used to configure the PCIe-SS in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, the Host must have at least 1 PF. The SoC must have at least 1 PF with 1 VF. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations tables below describe the supported number of PFs and VFs for the Host and SoC.
Table: Host PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 0 Max # of VFs 2000 distributed across all PFs Table: SoC PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 1 (on PF0) Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description [ip] type pcie Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss | soc_pcie_ss Specifies the output name of the PCIe-SS IP preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF bar0_address_width Integer bar4_address_width Integer vf_bar0_address_width Integer ats_cap_enable 0 | 1 vf_ats_cap_enable 0 | 1 prs_ext_cap_enable 0 | 1 pasid_cap_enable 0 | 1 pci_type0_vendor_id 32'h Value pci_type0_device_id 32'h Value revision_id 32'h Value class_code 32'h Value subsys_vendor_id 32'h Value subsys_dev_id 32'h Value sriov_vf_device_id 32'h Value exvf_subsysid 32'h Value The default values for all PCIe-SS parameters (that are not defined in the PCIe IP OFSS file) are defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2124-iopll-ip-ofss-file","title":"2.1.2.4 IOPLL IP OFSS File","text":"An OFSS file with IP type iopll (e.g. $OFS_ROOTDIR/tools/ofss_config/iopll/iopll.ofss) is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description [ip] type iopll Specifies that this OFSS file configures the IOPLL [settings] output_name sys_pll Specifies the output name of the IOPLL. instance_name iopll_0 Specifies the instance name of the IOPLL. [p_clk] freq Integer: 250 - 470 Specifies the IOPLL clock frequency in MHz. Note: The following frequencies have been tested on reference boards: 350MHz, 400MHz, 470MHz.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2125-memory-ip-ofss-file","title":"2.1.2.5 Memory IP OFSS File","text":"An OFSS file with IP type memory (e.g. $OFS_ROOTDIR/tools/ofss_config/memory/memory.ofss) is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description [ip] type memory Specifies that this OFSS file configures the Memory-SS [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. preset f2000x | String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi (e.g. $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_8x25.ofss) is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS num_channels Integer Specifies the number of channels. data_rate 10GbE | 25GbE | 100GCAUI-4 | 100GAUI-2 Specifies the data rate preset None | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. [1] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets should be stored in a $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is: $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/syn_top/output_files
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes some of the essential images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Description ofs_top.bin This is an intermediate, raw binary file. This intermediate raw binary file is produced by taking the Quartus Prime Pro generated *.sof file, and converting it to *.pof using quartus_pfg, then converting the *.pof to *.hexout using quartus_cpf, and finally converting the *.hexout to *.bin using objcopy. ofs_top.bin - Raw binary image of the FPGA. ofs_top_page0_unsigned_factory.bin This is the unsigned PACSign output generated for the Factory Image. Unsigned means that the image has been signed with an empty header. ofs_top_page1_user1.bin This is an input file to PACSign to generate ofs_top_page1_unsigned_user1.bin. This file is created by taking the ofs_top.bin file and assigning the User1 or appending factory block information. Unsigned means that the image has been signed with an empty header. ofs_top_page1_unsigned_user1.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User1 Image. This file is used to load the FPGA flash User1 Image using the fpgasupdate tool. Unsigned means that the image has been signed with an empty header. ofs_top_page2_user2.bin This is an input file to PACSign to generate ofs_top_page2_unsigned_user2.bin. This file is created by taking the ofs_top.bin file and assigning the User2 or appending factory block information. ofs_top_page2_unsigned_user2.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User2 Image. This file is used to load the FPGA flash User2 Image using the fpgasupdate tool. Unsigned means that the image has been signed with an empty header. ofs_top.sof This image is used to generate ofs_top.bin, and can also be used to program the Agilex\u00ae 7 FPGA device directly through JTAG Note: The build/output_files/timing_report directory contains clocks report, failing paths and passing margin reports.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <PATH_OF_RELOCATABLE_PR_TREE> <BOARD_TARGET> <WORK_DIRECTORY>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <PATH_OF_RELOCATABLE_PR_TREE> String Specifies the location of the relocatable PR directory tree to be created. <BOARD_TARGET> n6001 | n6000 | fseries-dk | iseries-dk Specifies the name of the board target. <WORK_DIRECTORY> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_page0_unsigned_factory.bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_page1_unsigned_user1.bin\n\u2514\u2500\u2500 \u2514\u2500\u2500 \u2514\u2500\u2500 ofs_top_page2_unsigned_user2.bin\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex\u00ae 7 SoC Attach FIM for the f2000x:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. Some examples are provided:
-
Out-of-Tree PR FIM using OFSS (Standard Flow)
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_oot_pr\n
Refer to the Create a Relocatable PR Directory Tree section for more information on out-of-tree PR builds.
-
Flat FIM using OFSS
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x_flat\n
-
In-Tree PR FIM using OFSS
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_in_tree_pr\n
-
Flat FIM using OFSS with Host Exercisers Removed
bash ./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat,null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_f2000x_flat_null_he
-
In-Tree PR FIM using Source
./ofs-common/scripts/common/syn/build_top.sh f2000x work_f2000x\n
The build takes ~3 hours to complete. A successful build will report the following:
Compile work directory: <OFS_ROOTDIR>/agx7-soc-attach/rel/default/ofs-f2000x-pl/work_f2000x/syn/syn_top\nCompile artifact directory: <OFS_ROOTDIR>/agx7-soc-attach/rel/default/ofs-f2000x-pl/work_f2000x/syn/syn_top/output_files\n\n***********************************\n***\n*** OFS_PROJECT: f2000x\n*** OFS_FIM: base\n*** OFS_BOARD: adp\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 1\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#226-creating-a-relocatable-pr-directory-tree","title":"2.2.6 Creating a Relocatable PR Directory Tree","text":"If you are developing a FIM to be used by another team developing AFU workload(s), scripts are provided that create a relocatable PR directory tree. ODM and board developers will make use of this capability to enable a broad set of AFUs to be loaded on a board using PR.
You can create this relocatable PR directory tree by either:
- Build FIM and AFU using ofs-common/scripts/common/syn/build_top.sh followed by running /ofs-common/scripts/common/syn/generate_pr_release.sh
- Build FIM and AFU using ofs-common/scripts/common/syn/build_top.sh with optional -p switch included
The generate_pr_release.sh has the following command structure:
ofs-common/scripts/common/syn/generate_pr_release.sh -t <work_directory>/build_tree <target_board> <work_directory>\n
Where: <work_directory>/build_tree is the location for your relocatable PR directory tree in the work directory<target_board> is the name of the board target/FIM e.g. f2000x <work_directory> is the work directory
Here is an example of running the generate_pr_release.sh script:
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/generate_pr_release.sh -t work_f2000x/build_tree f2000x work_f2000x
The resulting relocatable build tree has the following structure:
.\n\u251c\u2500\u2500 bin\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 afu_synth\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build_env_config\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 run.sh -> afu_synth\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 update_pim\n\u251c\u2500\u2500 hw\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 blue_bits\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ofs_top_page0_unsigned_factory.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ofs_top_page1_unsigned_user1.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ofs_top_page2_unsigned_user2.bin\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 ofs_top.sof -> ../lib/build/syn/syn_top/output_files/ofs_top.sof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 lib\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 build\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-ifc-id.txt\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 fme-platform-class.txt\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 platform\n\u2514\u2500\u2500 README\n
This build tree can be moved to a different location and used for AFU development of PR-able AFU to be used with this board."},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2261-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6.1 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM.
Note: This can be automatically done for you if you run the build script with the -p option.
Note: This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the f2000x OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_f2000x/pr_build_template f2000x work_f2000x\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
vim $OFS_ROOTDIR/syn/syn_top/ofs_top.qsf\n
set_global_assignment -name SEED 2\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#228-compiling-the-ofs-fim-using-quartus-gui","title":"2.2.8 Compiling the OFS FIM Using Quartus GUI","text":"Perform the following steps to compile the OFS FIM using the Quartus GUI:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Set the environment variables as described in the Setting Up Required Environment Variables section.
-
Run the setup portion of the build script. This takes a few seconds to complete.
./ofs-common/scripts/common/syn/build_top.sh --stage setup f2000x work_dir\n
-
Open the OFS FIM project using the Quartus Prime Pro GUI. The project is located at $OFS_ROOTDIR/work_dir/syn/syn_top/ofs_top.qpf.
-
Ensure the checkbox next to Assembler (Generate programming files) is marked.
-
Click the arrow next to Compile Design in the Compilation Flow window to start full compilation.
-
After compilation is complete, check the $OFS_ROOTDIR/work_dir/syn/syn_top/output_files directory. This directory will contain the output SOF image generated by Quartus, named ofs_top.sof.
-
Run the finish portion of the build script. This will generate the images that can be programmed to f2000x FPGA flash. Use the optional -p argument to create an out-of-tree PR build.
./ofs-common/scripts/common/syn/build_top.sh --stage finish f2000x work_dir\n
-
Check the $OFS_ROOTDIR/work_dir/syn/syn_top/output_files directory again to see that all output files have been generated.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#3-unit-level-simulation","title":"3 Unit Level Simulation","text":"Unit level simulation of key components in the FIM is provided. These simulations provide verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS-MX or Mentor Graphics Questasim simulators. Readme files are provided explaining how to run the simulation of each component.
The scripts to run the unit level simulations are located in $OFS_ROOTDIR/sim/unit_test, which contains subdirectories soc_tests and host_tests. The table below lists the tests that are provided.
Test Type Test Name SoC Tests csr_test dfh_walker flr he_lb_test he_mem_test hssi_kpi_test hssi_test mem_ss_csr_test mem_ss_rst_test mem_tg_test pf_vf_access_test qsfp_test regress_run.py Host Tests csr_test he_lb_test pcie_csr_test pf_vf_access_test pmci_csr_test pmci_mailbox_test pmci_rd_default_value_test pmci_ro_mailbox_test pmci_vdm_b2b_drop_err_scenario_test pmci_vdm_len_err_scenario_test pmci_vdm_mctp_mmio_b2b_test pmci_vdm_multipkt_error_scenario_test pmci_vdm_multipkt_tlp_err_test pmci_vdm_tlp_error_scenario_test pmci_vdm_tx_rx_all_random_lpbk_test pmci_vdm_tx_rx_all_toggle_test pmci_vdm_tx_rx_lpbk_test regress_run.py"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config> Used to modify IP, such as the PCIe SS, using .ofss configuration files. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. Platform Dependent[1] <build_target> n6001 | n6000 | fseries-dk | iseries-dk | f2000x Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional [1] Using OFSS is required for the N6000, F-Series Development Kit (2xF-Tile), and the I-Series Development Kit (2xR-Tile, 1xF-Tile).
Refer to the Run Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files; refer to the Run Regression Unit Level Simulation section for step-by-step instructions.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#321-walkthrough-run-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Run Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test on either the SoC or Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the target design.
./gen_sim_files.sh --ofss=$OFS_ROOTDIR/tools/ofss_config/f2000x.ofss f2000x\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
```bash sh run_sim.sh TEST=dfh_walker VCSMX=1
-
Once the test has completed, the test summary will be shown. The following is an example test summary after running the SoC DFH Walker Test:
Test status: OK\n\n********************\nTest summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/applications.fpga.ofs.fim-f2000x-pl/sim/unit\n_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 355393750\nV C S S i m u l a t i o n R e p o r t\nTime: 355393750 ps\nCPU Time: 59.870 seconds; Data structure size: 91.2Mb\nSun May 14 16:54:20 2023\n
-
The log for the test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_TYPE>/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the SoC DFH Walker test using VCS can be found in:
$OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker/sim_vcs/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review. You are encouraged to run the additional simulation examples to learn about each key area of the OFS shell.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"The regress_run.py script is provided to automatically run all unit tests for either the SoC or the Host. Note that running all tests will take around three hours for the SoC tests and around 2.5 hours for the Host tests to complete.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#331-walkthrough-run-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Run Regression Unit Level Simulation","text":"Perform the following steps to run comprehensive regression unit tests.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the test directory you wish to run from.
-
SoC Tests
cd $OFS_ROOTDIR/sim/unit_test/soc_tests\n
-
Host Tests
cd $OFS_ROOTDIR/sim/unit_test/host_tests\n
-
Run the tests with the regress_run.py. Use the -h argument to display the help menu. For example, to run all tests locally, generate the simulation files, using VCS, with 8 processors:
python regress_run.py -g -l -n 8 -k all -s vcs\n
-
Once all tests have completed, the comprehensive test summary will be shown. The following is an example test summary after running the SoC tests:
2023-05-14 19:10:55,404: Passing Unit Tests:12/12 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n2023-05-14 19:10:55,404: csr_test:......... PASS -- Time Elapsed:0:03:57.713154\n2023-05-14 19:10:55,404: dfh_walker:....... PASS -- Time Elapsed:0:02:46.025067\n2023-05-14 19:10:55,404: flr:.............. PASS -- Time Elapsed:0:03:26.289900\n2023-05-14 19:10:55,404: he_lb_test:....... PASS -- Time Elapsed:0:06:41.142643\n2023-05-14 19:10:55,404: he_mem_test:...... PASS -- Time Elapsed:1:39:01.824096\n2023-05-14 19:10:55,404: hssi_kpi_test:.... PASS -- Time Elapsed:2:21:33.007328\n2023-05-14 19:10:55,404: hssi_test:........ PASS -- Time Elapsed:2:16:36.821034\n2023-05-14 19:10:55,404: mem_ss_csr_test:.. PASS -- Time Elapsed:0:38:45.836540\n2023-05-14 19:10:55,404: mem_ss_rst_test:.. PASS -- Time Elapsed:0:40:51.065354\n2023-05-14 19:10:55,404: mem_tg_test:...... PASS -- Time Elapsed:0:54:00.210146\n2023-05-14 19:10:55,404: pf_vf_access_test: PASS -- Time Elapsed:0:03:25.561919\n2023-05-14 19:10:55,404: qsfp_test:........ PASS -- Time Elapsed:0:39:29.192304\n2023-05-14 19:10:55,404: Failing Unit Tests: 0/12 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n2023-05-14 19:10:55,404: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n2023-05-14 19:10:55,404: Number of Unit test results captured: 12\n2023-05-14 19:10:55,404: Number of Unit test results passing.: 12\n2023-05-14 19:10:55,404: Number of Unit test results failing.: 0\n2023-05-14 19:10:55,404: End Unit regression running at date/time................: 2023-05-14 19:10:55.404641\n2023-05-14 19:10:55,404: Elapsed time for Unit regression run....................: 2:22:39.310455\n
-
Output logs for each individual test are saved in the respective test's directory
$OFS_ROOTDIR/sim/unit_test/<TEST_TYPE>/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the SoC DFH Walker test using VCS can be found in:
$OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker/sim_vcs/transcript\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4-fim-customization","title":"4 FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Table: FIM Customization Walkthroughs
Walkthrough Name How to add a new module to the FIM How to debug the FIM with Signal Tap How to compile the FIM in preparation for designing your AFU How to resize the Partial Reconfiguration Region How to modify the Memory Subsystem How to compile the FIM with no HSSI How to change the PCIe Device ID and Vendor ID How to migrate to a different Agilex 7 device number How to change the Ethernet interface from 8x25 GbE to 8x10 GbE How to change the Ethernet interface from 8x25 GbE to 2x100 GbE How to add more transceiver channels to the FIM How to modify the PF/VF MUX configuration How to create a minimal FIM"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a walkthrough for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the SoC. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the f2000x card. The process for these are described in this section.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can only be mastered by the SoC. The BPF is an interconnect generated by Platform Designer. The figure below shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
You can create, modify, and/or generate the BPF manually in Platform Designer or more conveniently by executing a provided script.
We will add the Hello FIM module to an un-used address space in the SoC MMIO region. The table below shows the MMIO region for the SoC with the Hello FIM module added at base address 0x16000.
Offset Feature CSR set 0x00000 FME AFU 0x01000 Thermal Management 0x03000 Global Performance 0x04000 Global Error 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 HSSI Interface 0x15000 EMIF Interface 0x16000 Hello FIM 0x80000 PMCI Controller 0x100000 ST2MM (Streaming to Memory-Mapped) 0x130000 PR Control & Status (Port Gasket) 0x131000 Port CSRs (Port Gasket) 0x132000 User Clock (Port Gasket) 0x133000 Remote SignalTap (Port Gasket) 0x140000 AFU Errors (AFU Interface Handler) Refer to the FIM Technical Reference Manual: Interconnect Fabric for more information on the default MMIO region.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4113-files-to-edit-to-support-hello-fim","title":"4.1.1.3 Files to Edit to Support Hello FIM","text":"The table below shows all files in $OFS_ROOTDIR that will be modified or created in order to implement the Hello FIM module. The build_top.sh script copies files from $OFS_ROOTDIR into the target work directory and then the copied files are used in the Quartus build process. Details on editing these files is given in subsequent sections.
Category Status Path File Description Source Modify src/top top.sv Top RTL Modify src/pd_qsys bpf_top.sv BPF top RTL New src/hello_fim hello_fim_top.sv Hello FIM top RTL New src/hello_fim hello_fim_com.sv Hello FIM common RTL Design Files Modify syn/syn_top ofs_top.qsf Quartus setting file Modify syn/syn_top ofs_top_sources.tcl Define sources for top level design New syn/setup hello_fim_design_files.tcl Define RTLs of Hello FIM Modify syn/setup fabric_design_files.tcl Define IPs for fabric Platform Designer Modify src/pd_qsys/fabric bpf.txt Text definition of BPF interconnect Modify src/pd_qsys/fabric bpf.qsys BPF Qsys file Simulation Modify sim/unit_test/soc_tests/dfh_walker test_csr_defs.sv Define CSRs for simulation Verification Modify /ofs-common/verification/fpga_family/agilex/tests/sequences mmio_seq.svh MMIO testbench Modify /ofs-common/verification/fpga_family/agilex/tests/sequences dfh_walking_seq.svh DFH Walking testbench Modify ofs-common/verification/fpga_family/agilex/scripts Makefile_VCS.mk Makefile for VCS"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the SoC.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify syn/syn_top/ofs_top.qsf
-
Define the INCLUDE_HELLO_FIM Verilog macro to the Verilog Macros section. This will enable instantiation of the Hello FIM module. If this is not set, a dummy register will be instantiated instead.
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify syn/syn_top/ofs_top_sources.tcl
-
Add hello_fim_design_files.tcl to the list of subsystems in the Design Files section.
############################################\n# Design Files\n############################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE ../setup/hello_fim_design_files.tcl\n
-
Create syn/setup/hello_fim_design_files.tcl
-
Create hello_fim_design_files.tcl with the following contents:
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify /src/pd_qsys/fabric/fabric_design_files.tcl
-
Add bpf_hello_fim_slv.ip to the list of files in the BPF section.
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE ../ip_lib/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify src/top/top.sv
-
Add bpf_hello_fim_slv_if to AXI4-Lite Interfaces:
// AXI4-lite interfaces\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(12), .ARADDR_WIDTH(12)) bpf_hello_fim_slv_if();\n
-
Modify the value of NEXT_DFH_OFFSET of mem_ss_top from 24'h6B000 to 24'h1000
//*******************************\n// Memory Subsystem\n//*******************************\n`ifdef INCLUDE_DDR4\nmem_ss_top #(\n.FEAT_ID (12'h009),\n.FEAT_VER (4'h1),\n.NEXT_DFH_OFFSET (24'h1000),\n.END_OF_LIST (1'b0)\n) mem_ss_top (\n
-
Modify the value of NEXT_DFH_OFFSET of the Memory Subsystem dummy_csr from 24'h6B000 to 24'h1000
// Placeholder logic if no mem_ss\ndummy_csr #(\n.FEAT_ID (12'h009),\n.FEAT_VER (4'h1),\n.NEXT_DFH_OFFSET (24'h1000),\n.END_OF_LIST (1'b0)\n) emif_dummy_csr (\n
-
Add Hello FIM instance and dummy CSR after the Memory Subsystem. Set the NEXT_DFH_OFFSET to 24'h6A000 for both.
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (12),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (24'h6A000),\n.END_OF_LIST (1'b0)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if) );\n`else\ndummy_csr #( .FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (24'h6A000),\n.END_OF_LIST (1'b0)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif
-
Modify /src/pd_qsys/bpf_top.sv
-
Add bpf_hello_fim_slv_if to the interface descriptions
...\nmodule bpf_top (\n...\n//BPF funtions\n...\nofs_fim_axi_lite_if.master bpf_hello_fim_slv_if\n
-
Add bpf_hello_fim_slv_if to the module
module bpf_top (\n...\n);\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\nendmodule\n
-
Create src/hello_fim
-
Create src/hello_fim directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create src/hello_fim/hello_fim_top.sv
-
Create hello_fim_top.sv with the following contents:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : Nov 2021\n// Module Name : hello_fim_top.sv\n// Project : IOFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create src/hello_fim/hello_fim_com.sv
-
Create hello_fim_com.sv with the following contents:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Modify src/pd_qsys/fabric/bpf.txt
-
Add hello_fim as a slave in the BPF, and enable the SoC as a master for it.
#### - '#' means comment\n# NAME TYPE BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 21 fme,soc_pcie,hssi,qsfp0,qsfp1,emif,pmci,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute the helper script to re-generate the BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\nsh gen_fabrics.sh\n
-
After the shell script finishes, you can find the generated bpf_hello_fim_slv.ip file in $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/. This is the ip variant of the axi4lite shim that bridges the Hello FIM module with the BPF. The updated bpf.qsys file is located in $OFS_ROOTDIR/src/pd_qsys/fabric. You can view the updated bpf file in Platform designer as follows.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\nqsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/syn_top/ofs_top.qpf\n
The image below shows the BPF that integrates the bpf_hello_fim_slv axi4lite shim at address 0x16000, generated through the helper script gen_fabrics.sh.
-
Compile the Hello FIM design:
-
Return to the OFS root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_hello_fim\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker/test_csr_defs.sv
-
Add a HELLO_FIM_IDX entry to the t_dfh_idx enumeration:
typedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX,\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n
-
Add an entry for HELLO_FIM_IDX into the get_dfh_names() function:
function automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\ndfh_name[MAX_DFH_IDX-1:0] dfh_names;\ndfh_names[FME_DFH_IDX] = \"FME_DFH\";\ndfh_names[THERM_MNGM_DFH_IDX] = \"THERM_MNGM_DFH\";\ndfh_names[GLBL_PERF_DFH_IDX] = \"GLBL_PERF_DFH\";\ndfh_names[GLBL_ERROR_DFH_IDX] = \"GLBL_ERROR_DFH\";\ndfh_names[QSFP0_DFH_IDX] = \"QSFP0_DFH\";\ndfh_names[QSFP1_DFH_IDX] = \"QSFP1_DFH\";\ndfh_names[HSSI_DFH_IDX] = \"HSSI_DFH\";\ndfh_names[EMIF_DFH_IDX] = \"EMIF_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\";\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\ndfh_names[PG_PR_DFH_IDX] = \"PG_PR_DFH\";\ndfh_names[PG_PORT_DFH_IDX] = \"PG_PORT_DFH\";\ndfh_names[PG_USER_CLK_DFH_IDX] = \"PG_USER_CLK_DFH\";\ndfh_names[PG_REMOTE_STP_DFH_IDX] = \"PG_REMOTE_STP_DFH\";\ndfh_names[AFU_ERR_DFH_IDX] = \"AFU_ERR_DFH\";\nreturn dfh_names;\nendfunction\n
-
Modify the expected DFH value of the EMIF from 64'h3_00000_06B000_1009 to 64'h3_00000_001000_1009 and add the expected value for HELLO_FIM as 64'h3_00000_06A000_0100:
function automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\nlogic[MAX_DFH_IDX-1:0][63:0] dfh_values;\ndfh_values[FME_DFH_IDX] = 64'h4000_0000_1000_0000;\ndfh_values[THERM_MNGM_DFH_IDX] = 64'h3_00000_002000_0001;\ndfh_values[GLBL_PERF_DFH_IDX] = 64'h3_00000_001000_0007;\ndfh_values[GLBL_ERROR_DFH_IDX] = 64'h3_00000_00e000_1004;\ndfh_values[QSFP0_DFH_IDX] = 64'h3_00000_001000_0013;\ndfh_values[QSFP1_DFH_IDX] = 64'h3_00000_001000_0013;\ndfh_values[HSSI_DFH_IDX] = 64'h3_00000_001000_100f;\ndfh_values[EMIF_DFH_IDX] = 64'h3_00000_001000_1009;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_06A000_0100;\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_080000_1012;\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_030000_0014;\ndfh_values[PG_PR_DFH_IDX] = 64'h3_00000_001000_1005;\ndfh_values[PG_PORT_DFH_IDX] = 64'h4_00000_001000_1001;\ndfh_values[PG_USER_CLK_DFH_IDX] = 64'h3_00000_001000_1014;\ndfh_values[PG_REMOTE_STP_DFH_IDX] = 64'h3_00000_00d000_2013;\ndfh_values[AFU_ERR_DFH_IDX] = 64'h3_00001_000000_2010;\nreturn dfh_values;\nendfunction\n
-
Regenerate the simulation files
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\nsh gen_sim_files.sh f2000x
-
Run the DFH Walker test
cd $OFS_ROOTDIR/sim/unit_test/soc_tests/dfh_walker\nsh run_sim.sh\n
-
Check the output for the presence of the HELLO_FiM module at address 0x16000:
********************************************\nRunning TEST(0) : test_dfh_walking\n********************************************\n\n...\n\nREAD64: address=0x00015000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000010001009\n\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nREAD64: address=0x00016000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x30000006a0000100\n\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000006a0000100)\nREAD64: address=0x00080000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000800001012\n\nPMCI_DFH\n Address (0x80000)\nDFH value (0x3000000800001012)\n...\n\nTest status: OK\n\n********************\nTest summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#414-walkthrough-modify-and-run-uvm-tests-for-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Modify and run UVM tests for a FIM that has a new module","text":"Perform the following steps to modify the UVM simulation files to support a design that has a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Modify $OFS_ROOTDIR/verification/tests/sequences/dfh_walking_seq.svh
-
Modify the dfh_offset_array to insert the Hello FIM.
dfh_offset_array = new[16];\ndfh_offset_array[ 0] = tb_cfg0.PF0_BAR0; // FME_DFH 0x8000_0000\ndfh_offset_array[ 1] = dfh_offset_array[ 0] + 64'h0_1000; // THERM_MNGM_DFH 0x8000_1000\ndfh_offset_array[ 2] = dfh_offset_array[ 1] + 64'h0_2000; // GLBL_PERF_DFH 0x8000_3000\ndfh_offset_array[ 3] = dfh_offset_array[ 2] + 64'h0_1000; // GLBL_ERROR_DFH 0x8000_4000\ndfh_offset_array[ 4] = dfh_offset_array[ 3] + 64'h0_E000; // QSFP0_DFH 0x8001_2000\ndfh_offset_array[ 5] = dfh_offset_array[ 4] + 64'h0_1000; // QSFP1_DFH 0x8001_3000\ndfh_offset_array[ 6] = dfh_offset_array[ 5] + 64'h0_1000; // HSSI_DFH 0x8001_4000\ndfh_offset_array[ 7] = dfh_offset_array[ 6] + 64'h0_1000; // EMIF_DFH 0x8001_5000\ndfh_offset_array[ 8] = dfh_offset_array[ 7] + 64'h0_1000; // HELLO_FIM_DFH 0x8001_6000\ndfh_offset_array[ 9] = dfh_offset_array[ 8] + 64'h6_a000; // PMCI_DFH 0x8008_0000\ndfh_offset_array[ 10] = dfh_offset_array[ 9] + 64'h8_0000; // ST2MM_DFH 0x8010_0000\ndfh_offset_array[ 11] = dfh_offset_array[10] + 64'h3_0000; // PG_PR_DFH_IDX 0x8013_0000\ndfh_offset_array[ 12] = dfh_offset_array[11] + 64'h0_1000; // PG_PORT_DFH_IDX 0x8013_1000\ndfh_offset_array[ 13] = dfh_offset_array[12] + 64'h0_1000; // PG_USER_CLK_DFH_IDX 0x8013_2000\ndfh_offset_array[ 14] = dfh_offset_array[13] + 64'h0_1000; // PG_REMOTE_STP_DFH_IDX 0x8013_3000\ndfh_offset_array[ 15] = dfh_offset_array[14] + 64'h0_D000; // PG_AFU_ERR_DFH_IDX 0x8014_0000\n
-
Modify $OFS_ROOTDIR/verification/tests/sequences/mmio_seq.svh
-
Add test code related to the Hello FIM. This code will verify the scratchpad register at 0x16030 and read only the register at 0x16038.
// HELLO_FIM_Scratchpad 64 bit access\n`uvm_info(get_name(), $psprintf(\"////Accessing PF0 HELLO_FIM_Scratchpad Register %0h+'h16030////\", tb_cfg0.PF0_BAR0), UVM_LOW)\nassert(std::randomize(wdata));\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h30;\nmmio_write64(.addr_(addr), .data_(wdata));\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h38;\nwdata = 64'h6626_0701_5000_0034;\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\n
Note: uvm_info and uvm_error statements will put a message into log file.
-
Modify $OFS_ROOTDIR/verification/scripts/Makefile_VCS.mk
-
Add INCLUDE_HELLO_FIM define option to enable Hello FIM on UVM
VLOG_OPT += +define+INCLUDE_HELLO_FIM\n
-
Re-generate the UVM files
-
Navigate to the verification scripts directory
cd $VERDIR/scripts\n
-
Clean the output of previous builds
gmake -f Makefile_VCS.mk clean\n
-
Compile the IP files
gmake -f Makefile_VCS.mk cmplib_adp\n
-
Build the RTL and Test Benches
gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1
Note: Using the DEBUG option will provide more detail in the log file for the simulation.
-
Run the UVM DFH Walker simulation
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk run TESTNAME=dfh_walking_test DUMP=1 DEBUG=1\n
Note: Using the DEBUG option will provide more detail in the log file for the simulation.
-
Verify the DFH Walker test results
-
The output logs are stored in the $VERDIR/sim/dfh_walking_test directory. The main files to note are described in the table below:
File Name Description runsim.log A log file of UVM trans.log A log file of transactions on PCIe bus inter.vpd A waveform for VCS -
Run the following command to quickly verify- that the Hello FIM module was successfully accessed. In the example below, the message DFH offset Match! Exp = 80016000 Act = 80016000 shows that the Hello FIM module was successfully accessed.
cd $VERDIR/sim/dfh_walking_test\ncat runsim.log | grep \"DFH offset\"\n
Expected output:
UVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 111950000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp = 80000000 Act = 80000000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 112586000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80001000 Act = 80001000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113222000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80003000 Act = 80003000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113858000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80004000 Act = 80004000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 114494000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80012000 Act = 80012000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115147000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80013000 Act = 80013000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115801000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80014000 Act = 80014000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 116628000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80015000 Act = 80015000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117283000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80016000 Act = 80016000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117928000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80080000 Act = 80080000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 118594000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80100000 Act = 80100000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119248000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80130000 Act = 80130000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119854000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80131000 Act = 80131000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 120460000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80132000 Act = 80132000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121065000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80133000 Act = 80133000\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121672000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80140000 Act = 80140000\n
-
Run UVM MMIO Simulation
cd $VERDIR/scripts\ngmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1\n
-
Verify the MMIO test results. Run the following commands to show the result of the scratchpad register and Hello FIM ID register. You can see the \"Data match\" message indicating that the registers are successfuly verified.
cd $VERDIR/sim/mmio_test\ncat runsim.log | grep \"Data\" | grep 1603\n
Expected output:
UVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/mmio_seq.svh(68) @ 115466000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016030, data = 880312f9558c00e1\nUVM_INFO /home/applications.fpga.ofs.fim-f2000x-pl/verification/tests/sequences/mmio_seq.svh(76) @ 116112000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016038, data = 6626070150000034\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#415-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.5 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
Start in your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Run the fpgainfo fme command to determine the PCIe B:D.F of your board. In this example, the PCIe B:D.F is 15:00.0.
Example Output:
Intel IPU Platform f2000x\nBoard Management Controller NIOS FW version: 1.2.4 Board Management Controller Build version: 1.2.4 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : fb25ff1d-c31a-55d8-81d8-cbcedcfcea17\nBoot Page : user1\nUser1 Image Info : 81d8cbcedcfcea17fb25ff1dc31a55d8\nUser2 Image Info : a566ceacaed810d43c60b0b8a7145591\nFactory Image Info : None\n
-
Check that the driver software on 0000:15:00.0 is dfl-pci.
opae.io ls\n
Example output:
[0000:15:00.0] (0x8086:0xbcce 0x8086:0x17d4) Intel IPU Platform f2000x (Driver: dfl-pci)\n
-
Initialize the opae.io tool
opae.io init -d 15:00.0\n
-
Confirm the driver software on 0000:15:00.0 has been updated to vfio-pci.
opae.io ls\n
Example Output:
[0000:15:00.0] (0x8086:0xbcce 0x8086:0x17d4) Intel IPU Platform f2000x (Driver: vfio-pci)\n
-
Run the DFH Walker test to verify there is a module at offset 0x16000
opae.io walk -d 15:00.0\n
Example output:
offset: 0x0000, value: 0x4000000010000000\n dfh: id = 0x0, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x1000, value: 0x3000000020000001\n dfh: id = 0x1, rev = 0x0, next = 0x2000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x3000, value: 0x3000000010000007\n dfh: id = 0x7, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x4000, value: 0x30000000e0001004\n dfh: id = 0x4, rev = 0x1, next = 0xe000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x12000, value: 0x3000000010000013\n dfh: id = 0x13, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x13000, value: 0x3000000010000013\n dfh: id = 0x13, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x14000, value: 0x3000000010002015\n dfh: id = 0x15, rev = 0x2, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000006a0000100\n dfh: id = 0x100, rev = 0x0, next = 0x6a000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x80000, value: 0x3000000800002012\n dfh: id = 0x12, rev = 0x2, next = 0x80000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x100000, value: 0x3000000300000014\n dfh: id = 0x14, rev = 0x0, next = 0x30000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x130000, value: 0x3000000010001005\n dfh: id = 0x5, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x131000, value: 0x4000000010001001\n dfh: id = 0x1, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4\noffset: 0x132000, value: 0x3000000010001014\n dfh: id = 0x14, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x133000, value: 0x30000000d0002013\n dfh: id = 0x13, rev = 0x2, next = 0xd000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x140000, value: 0x3000010000002010\n dfh: id = 0x10, rev = 0x2, next = 0x0, eol = 0x1, reserved = 0x0, feature_type = 0x3\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d 15:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d 15:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d 0000:15:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d 15:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d 15:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:15:00.0] (0x8086:0xbcce 0x8086:0x17d4) Intel IPU Platform f2000x (Driver: dfl-pci)\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#416-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.6 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
For more detailed information on Signal Tap please see refer to Quartus Prime Pro Edition User Guide: Debug Tools (RDC Document ID 683819).
Signal Tap uses the FPGA Download Cable II USB device to provide access. Please see Intel FPGA Download Cable II for more information. This device is widely available via distributors for purchase.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Steps:
-
The design must be synthesized before adding Signal Tap.
-
If you are using the previously built Hello FIM design, copy the work directory and rename it so that we have a work directory dedicated to the Hello FIM Signal Tap design.
cp -r $OFS_ROOTDIR/work_hello_fim $OFS_ROOTDIR/work_hello_fim_with_stp\n
-
If you are adding signal tap to a new design that has not yet been synthesized, perform the following steps to synthesize the design.
-
Set the environment variables as described in the Setting Up Required Environment Variables section.
-
Run the build script with the -e option to synthesize the design.
./ofs-common/scripts/common/syn/build_top.sh -e --ofss tools/ofss_config/f2000x.ofss f2000x work_hello_fim_with_stp\n
-
Open the Hello FIM Signal Tap project in the Quartus Prime Pro GUI. The Quartus Prime Pro project is named ofs_top.qpf and is located in the work directory $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top/ofs_top.qpf.
-
Select Tools > Signal Tap Logic Analyzer to open the Signal Tap GUI.
-
Accept the \"Default\" selection and click \"Create\".
-
This opens the Signal Tap Logic Analyzer window.
-
Set up the clock for the STP instance. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Prime Pro Project Navigator to find the block of interest and open the design instance for review. For example, see the image below using Project Navigator to open the top module where hello_fim_top_inst is instantiated.
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note, that the clock selected should be associated with the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. Ensure that all signals that are to be sampled are on the same clock domain as the clock you select here. In the middle right of the Signal Tap window, under Signal Configuration, Clock:, select \"\u2026\" as shown below:
-
In the Node Finder tool that popped up, input \"hello_fim_top_inst|clk\" into the \"Named:\" textbox and click \"Search\". Select \"clk\" in the Matching Nodes list and click the \">\" button to select this clock as shown below. Click \"OK\" to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM. Select the signals that appear from the search patterns hello_fim_top_inst|reset and hello_fim_top_inst|csr_lite_if*. Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, \"STP_For_Hello_FIM\".
-
Save the newly created Signal Tap file, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will aurtomatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE STP_For_Hello_FIM.stp\nset_global_assignment -name SIGNALTAP_FILE STP_For_Hello_FIM.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
ofs-common/scripts/common/syn/build_top.sh -p -k f2000x work_hello_fim_with_stp\n
Alternatively, you can copy the ofs_top.qsf and STP_For_Hello_FIM.stp files from the Hello FIM with STP work directory to replace the original files in the cloned OFS repository. In this scenario, all further FIM compilation projects will include the Signal Tap instance integrated into the design. Execute the following commands for this alternative flow:
Copy the modified file \"work_hello_fim_with_stp/syn/syn_top/ofs_top.qsf\" over to the source OFS repository, into \"syn/syn_top/\".
cd $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top\ncp ofs_top.qsf $OFS_ROOTDIR/syn/syn_top\ncp STP_For_Hello_FIM.stp $OFS_ROOTDIR/syn/syn_top\n
Compile the FIM using the files from the OFS repository to create a new work directory.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh -p f2000x work_hello_fim_with_stp_from_src_repo\n
-
Ensure that the compile completes successfully and meets timing:
***********************************\n***\n*** OFS_PROJECT: f2000x\n*** OFS_FIM: base\n*** OFS_BOARD: adp\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 0\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the f2000x. Refer to Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the f2000x via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/syn_top/output_files/ofs_top.sof image to the f2000x FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open Quartus Signal Tap GUI.
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap GUI, open your STP file. Your STP file settings will load. In this example we used STP_For_Hello_FIM.stp.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable USB-BlasterII. In the Device: selection box select the Agilex\u00ae 7 FPGA device.
-
If the Agilex\u00ae 7 FPGA is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
If not already set, you can create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'STP_For_Hello_FIM' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, go back to the terminal and walk the DFH list or peek/poke the Hello FIM registers as was done during the creation of the Hello FIM design example.
opae.io init -d 0000:15:00.0\nopae.io walk -d 0000:15:00.0\nopae.io release -d 0000:15:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
-
The PCIe AER feature is automatically re-enabled by rebooting the server.
This concludes the example on how to instrument an OFS FIM with the Quartus Prime Signal Tap Logic Analyzer.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" block during compile-time. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination, order or all can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#421-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
-
Navigate to the OFS root directory
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the null host exerciser options.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_f2000x\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to reduce the size of the PR region to fit the additional logic into the static region. Similiarly, if you reduce logic in the FIM region, then you can increase the size of the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions is reported in the Logic Lock Region Usage Summary sections of following two files:
$OFS_ROOTDIR/<YOUR_WORK_DIRECTORY>/syn/syn_top/output_files/ofs_top.fit.place.rpt-
$OFS_ROOTDIR/<YOUR_WORK_DIRECTORY>/syn/syn_top/output_files/ofs_top.fit.rpt
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to comfortably hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to customize the resources allocated to the AFU in the PR regions:
-
The $OFS_ROOTDIR/syn/setup/pr_assignments.tcl TCL file defines the Logic Lock Regions in the design, including the PR partition where the AFU is allocated.
The default design uses the the following Logic Lock Regions:
set TOP_MEM_REGION \"X115 Y310 X219 Y344\"\nset BOTTOM_MEM_REGION \"X0 Y0 X294 Y20\"\nset SUBSYSTEM_REGION \"X0 Y0 X60 Y279; X0 Y0 X300 Y39; X261 Y0 X300 Y129;\"\nset AFU_PLACE_REGION \"X61 Y40 X260 Y309; X220 Y130 X294 Y329; X12 Y280 X114 Y329;\"\nset AFU_ROUTE_REGION \"X0 Y0 X294 Y329\"\n
Each region is made up of rectangles defined by the origin (X0,Y0) and the top right corner (X1,Y1).
-
Use Quartus Chip Planner to identify the locations of the resources available within the Agilex\u00ae 7 FPGA for placement and routing your AFU. The image below shows the default floorplan for the f2000x Agilex\u00ae 7 FPGA.
-
Make changes to the $OFS_ROOTDIR/syn/setup/pr_assignments.tcl file based on your findings in Quartus Chip Planner. You can modify the size and location of existing Logic Lock Regions or add new ones and assign them to the AFU PR partition. You will need to modify the coordinates of other regions assigned to the FIM accordingly to prevent overlap.
-
Recompile your FIM and create the PR relocatable build tree using the following commands:
cd $OFS_ROOTDIR ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x <YOUR_WORK_DIRECTORY>\n
-
Analyze the resource utilization report per partition produced after recompiling the design.
-
Make further modifications to the PR regions until the results are satisfactory. Make sure timing constraints are met.
Refer to the following documentation for more information on how to optimize the floor plan of your Partial Reconfiguration design:
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3: Floorplan the Design
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe sub-system IP and PF/VF MUX can be modified either using the OFSS flow or the IP Presets flow. The OFSS flow supports a subset of all available PCIe Sub-system settings, while the IP Preset flow can make any available PCIe Sub-system settings change. With PCIe-SS modifcations related to the PFs and VFs, the PF/VF MUX logic is automatically configured based on the PCIe-SS configuration when using OFSS. The sections below describe each flow.
- PCIe Configuration Using OFSS
- PCIe Sub-System configuration Using IP Presets
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS Agilex\u00ae 7 SoC Attach FIM for the f2000x can support up to 8 PFs and 2000 VFs distributed accross all PFs on both the Host and the SoC.
For reference FIM configurations, you must have at least 1 PF on the Host, and at least 1 PF with 1 VF on the SoC. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: Host PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 0 Max # of VFs 2000 distributed across all PFs Table: SoC PF/VF Limitations
Parameter Value Min # of PFs 1 Max # of PFs 8 Min # of VFs 1 (on PF0) Max # of VFs 2000 distributed across all PFs New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe configuration registers contains the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00001771 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00001771"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4431-walkthrough-modify-the-pcie-sub-system-and-pfvf-mux-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS","text":"Perform the following steps to modify the PF/VF MUX configuration.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Decide which PCIe PF/VFs require modification. If you are modifying host side PF/VF configuration, you must edit file pcie_host.ofss file found in $OFS_ROOTDIR/tools/pfvf_config_tool. If you want to modify SoC-side PF/VF configuration, edit the pcie_soc.ofss file found in the same location. The the following code shows the default Host OFSS file:
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nbar0_address_width = 21\n[pf1]\n
This default configuration is made up of two physical functions (PF), and neither of them has any virtual functions (VF).
-
In this example, we will modify the Host PCIe configuration. Create a new Host PCIe OFSS file from the existing pcie_host.ofss file.
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_pfvf_mod.ofss\n
-
Modify the new pcie_pfvf_mod.ofss OFSS file with the new PF/VF configuration. An example modification to the OFSS file is shown below. In this example we have changed the configuration to: 6 PFs in total, 4 VFs in PF0, 1 VF in PF2, and 2 VFs on PF3. You can add up to 8 PFs and could conceivably add up to the number of VFs supported by the PCIe IP. Note that more PFs/VFs will use more FPGA resources, which may cause fitter challenges.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nbar0_address_width = 21\nnum_vfs = 4\n[pf1]\n[pf2]\nnum_vfs = 1\n[pf3]\nnum_vfs = 2\n[pf4]\n[pf5]\n
-
Edit the top level OFSS file to use the new PCIe OFSS file pcie_host_pfvf_mod.ofss. In this example, we will edit the f2000x top level OFSS file $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host_pfvf_mod.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n
-
Compile the FIM.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_pfvf_mod\n
-
Copy the resulting .bin user 1 image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the .bin image to the f2000x FPGA. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
From the Host, verify the number of VFs on the PFs. In this example, we defined 4 VFs on PF0 in Step 5.
sudo lspci -vvv -s b1:00.0 | grep VF\n
Example output:
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 6, stride: 1, Device ID: bccf
-
Verify communication with the newly added PFs. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
In the following steps, we will verify the newly added PF5.
-
Initialize the driver on PF5
sudo opae.io init -d b1:00.5\n
-
Read the GUID for the PF5 CSR stub.
sudo opae.io -d b1:00.5 -r 0 peek 0x8\nsudo opae.io -d b1:00.5 -r 0 peek 0x10\n
Example output:
0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n
Note: The PCIe B:D.F associated with your board may be different. Use the fpgainfo fme command to see the PCIe B:D:F for your board.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#444-pcie-sub-system-configuration-using-ip-presets","title":"4.4.4 PCIe Sub-System configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings as a starting point.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#4441-walkthrough-modify-pcie-sub-system-and-pfvf-mux-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets","text":"Perform the following steps to use an IP preset file to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID on PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
[OPTIONAL] Run the setup stage of the build script using your desired OFSS configration to create a working directory for the target board. In this example we will target the f2000x.
./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
Open the host PCIe-SS using Quartus Parameter Editor. If you performed Step 3, open the PCIe-SS IP from the work directory; otherwise, open the PCIe-SS IP from the source files.
```bash qsys-edit $OFS_ROOTDIR/work_f2000x/ipss/pcie/qip/pcie_ss.ip
-
Modify the settings as desired. In this example we will change the Revision ID to 0x2. In the IP Parameter Editor, scroll down and expand the PCIe Interfaces Ports Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab and make this change.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, f2000x-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 6.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = f2000x-rev2\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss file to call new OFSS file created in Step 10.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n
-
Compile the FIM.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_f2000x_pcie_mod/syn/syn_top/output_files/ofs_top.sof image to your deployment environmenment for JTAG programming, or copy a bin file (e.g. ofs_top_page1_unsigned_user1.bin) for RSU programming.
Note: OPAE FPGA management commands require recognition of the FPGA PCIe Device ID for control. If there is a problem between OPAE management recognition of FPGA PCIe values, then control of the card will be lost. For this reason, you are strongly encouraged to program the FPGA via JTAG to load the test FPGA image. If there is a problem with the SOF image working with your host software that is updated for the new PCIe settings, then you can load a known good SOF file to recover. Once you sure that both the software and FPGA work properly, you can load the FPGA into FPGA flash using the OPAE command fpgasupdate.
-
Program the image to the f2000x FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step JTAG programming instructions, or the Program the FPGA via RSU Section for step-by-step RSU programming instructions.
-
Use lspci to verify that the PCIe changes have been implemented.
lspci -nvmms b1:00.0\n
Example output:
Slot: b1:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 1771\nPhySlot: 1\nRev: 02\nNUMANode: 1\n
Note: Some changes to software may be required to work with certain new PCIe settings. These changes are described in Software Reference Manual: Open FPGA Stack
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#451-walkthrough-create-a-minimal-fim","title":"4.5.1 Walkthrough: Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM. A minimal FIM is one that has the host exercisers and ethernet subsystem removed. This frees up resources that can be used as desired.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions. To create this minimal FIM, perform the following steps:
-
Edit the Host PCIe OFSS file to use the minimal number of PFs (1).
-
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nbar0_address_width = 21\n
-
Edit the SoC PCIe OFSS file to use the minimal number of PFs (1) and VFs (1).
-
$OFS_ROOTDIR/tools/ofss_config/pcie/soc_host.ofss
[ip]\ntype = pcie\n\n[settings]\noutput_name = soc_pcie_ss\n\n[pf0]\nnum_vfs = 1\nbar0_address_width = 21\nvf_bar0_address_width = 21
-
Run the build script with exercisers and ethernet subsystem (HSSI) removed.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_f2000x_minimal_fim\n
-
The build will complete with reduced resources as compared to the base version. You may review the floorplan in Quartus Chip Planner and modify the Logic Lock regions to allocate more resources to the PR region if desired. Refer to the How to Resize the Partial Reconfiguration Region section for information regarding modifications to the floorplan.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#46-migrate-to-a-different-agilex-device-number","title":"4.6 Migrate to a Different Agilex Device Number","text":"The following instructions enable you to change the Agilex 7 FPGA device part number of the f2000x, for example, to migrate to a device with a larger density. Be aware that this release works with Agilex\u00ae 7 FPGAs that have P-tile for PCIe and E-tile for Ethernet.
The default device for the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL is AGFC023R25A2E2VR0
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"This walkthrough describes how to change the device to a larger density with the same package. In this example, we will change the device from part AGFC023R25A2E2VR0 to part AGFA027R25A2E2VR0.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the design repository. See the Clone the OFS Git Repo section.
-
Set the environment variables as described in the Setting Up Required Environment Variables section.
-
Navigate to the OFS Root Directory
cd $OFS_ROOTDIR\n
-
Use the following command to change the device part number throughout the OFS Root directory heirarchy, replacing <DEFAULT_OPN> and <NEW_OPN> with the part numbers specific to your update:
grep -rli '<DEFAULT_OPN>' * | xargs -i@ sed -i 's/<DEFAULT_OPN>/<NEW_OPN>/g' @\n
For example, use the following command to change from part AGFC023R25A2E2VR0 to part AGFA027R25A2E2VR0:
grep -rli 'AGFC023R25A2E2VR0'* | xargs -i@ sed -i 's/AGFC023R25A2E2VR0/AGFA027R25A2E2VR0/g' @\n
This changes all occurrences of the default device (AGFC023R25A2E2VR0) in the $OFS_ROOTDIR directory to the new device number (AGFA027R25A2E2VR0).
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/f2000x_base.ofss file to use the new part number; in this example AGFA027R25A2E2VR0. This is only necessary if you are using the OFSS flow.
[ip]\ntype = ofs\n\n[settings]\nplatform = f2000x\nfim = base_x16\nfamily = agilex\npart = AGFA027R25A2E2VR0\ndevice_id = 6100\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/syn_top/ofs_top.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFA027R25A2E2VR0\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/syn_top/ofs_pr_afu.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFA027R25A2E2VR0\n
-
Modify the DEVICE field in te $OFS_ROOTDIR/ipss/pmci/pmci_ss.qsf file.
set_global_assignment -name DEVICE AGFA027R25A2E2VR0\n
-
Compile the flat (non-PR) design to verify the compilation is successful with the new part. The flat design is compiled without any Logic Lock constraints.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat <YOUR_WORK_DIRECTORY>\n
-
To enable the PR region, use Quartus Chip Planner to analyze the compiled flat design and adjust the Logic Lock constraints defined in $OFS_ROOTDIR/syn/setup/pr_assignments.tcl for the new device layout. Refer to the How to Resize the Partial Reconfiguration Region section for instructions. Re-compile the design with the out-of-tree PR region enabled.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x <YOUR_WORK_DIRECTORY>\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM. This section provides an example walkthrough for modifiying the Memory-SS.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#471-walkthrough-modify-the-memory-sub-system-using-ip-presets-with-ofss","title":"4.7.1 Walkthrough: Modify the Memory Sub-System Using IP Presets With OFSS","text":"In this example we will modify the Memory Subsystem to enable ECC on all of the existing memory interfaces. You may make different modifications to meet your own design requirements. Perform the following steps to make this change.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Memory Subsystem IP file in Platform Designer to perform the required edits.
qsys-edit $OFS_ROOTDIR/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
The Memory Subsystem IP will open in IP Parameter Editor. Click Dive Into Packaged Subsystem.
-
The Platform Designer mem_ss view opens. All of the EMIFs are shown in the Filter window.
-
Click each EMIF 0 through 3 and perform the following actions.
-
In the Parameters window, click the Memory tab and change the DQ width to 40.
-
In the Parameters window, click the Controller tab. Scroll down and check the box for Enable Error Detection and Correction Logic with ECC.
-
Once Step 6 has been done for each EMIF 0-3, click File -> Save. Close the Platform Designer window.
-
In the IP Parameter Editor Presets window, click New to create an IP Presets file.
-
In the New Preset window, set the Name for the preset. In this case we will name it f2000x-ecc.
-
Click the ... button to select the location for the Preset file.
-
In the Save As window, change the save location to $OFS_ROOTDIR/ipss/mem/qip/presets and change the File Name to f2000x-ecc.qprs. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close the IP Parameter Editor. You do not need to generate or save the IP.
-
Edit the $OFS_ROOTDIR/syn/setup/emif_loc.tcl file to assign the pins required for ECC enabled interfaces.
-
Uncomment the DQS4 (ECC) pin assignments for all memory interfaces
#---------------------------------------------------------\n# EMIF CH0\n#---------------------------------------------------------\n...\n# # CH0 DQS4 (ECC)\nset_location_assignment PIN_A39 -to ddr4_mem[0].dq[32]\nset_location_assignment PIN_J35 -to ddr4_mem[0].dq[33]\nset_location_assignment PIN_C38 -to ddr4_mem[0].dq[34]\nset_location_assignment PIN_G34 -to ddr4_mem[0].dq[35]\nset_location_assignment PIN_G38 -to ddr4_mem[0].dq[36]\nset_location_assignment PIN_C34 -to ddr4_mem[0].dq[37]\nset_location_assignment PIN_J39 -to ddr4_mem[0].dq[38]\nset_location_assignment PIN_A35 -to ddr4_mem[0].dq[39]\nset_location_assignment PIN_C36 -to ddr4_mem[0].dqs[4]\nset_location_assignment PIN_A37 -to ddr4_mem[0].dqs_n[4]\nset_location_assignment PIN_G36 -to ddr4_mem[0].dbi_n[4]\n#---------------------------------------------------------\n# EMIF CH1\n#---------------------------------------------------------\n...\n# # CH1 DQS4 (ECC)\nset_location_assignment PIN_N7 -to ddr4_mem[1].dq[32]\nset_location_assignment PIN_L12 -to ddr4_mem[1].dq[33]\nset_location_assignment PIN_L6 -to ddr4_mem[1].dq[34]\nset_location_assignment PIN_U14 -to ddr4_mem[1].dq[35]\nset_location_assignment PIN_U7 -to ddr4_mem[1].dq[36]\nset_location_assignment PIN_W12 -to ddr4_mem[1].dq[37]\nset_location_assignment PIN_W6 -to ddr4_mem[1].dq[38]\nset_location_assignment PIN_N14 -to ddr4_mem[1].dq[39]\nset_location_assignment PIN_L9 -to ddr4_mem[1].dqs[4]\nset_location_assignment PIN_N10 -to ddr4_mem[1].dqs_n[4]\nset_location_assignment PIN_W9 -to ddr4_mem[1].dbi_n[4]\n#---------------------------------------------------------\n# EMIF CH2\n#---------------------------------------------------------\n...\n# CH2 DQS4 (ECC)\nset_location_assignment PIN_GC37 -to ddr4_mem[2].dq[32]\nset_location_assignment PIN_GC41 -to ddr4_mem[2].dq[33]\nset_location_assignment PIN_FY37 -to ddr4_mem[2].dq[34]\nset_location_assignment PIN_GE40 -to ddr4_mem[2].dq[35]\nset_location_assignment PIN_FV36 -to ddr4_mem[2].dq[36]\nset_location_assignment PIN_FY41 -to ddr4_mem[2].dq[37]\nset_location_assignment PIN_GE36 -to ddr4_mem[2].dq[38]\nset_location_assignment PIN_FV40 -to ddr4_mem[2].dq[39]\nset_location_assignment PIN_GE38 -to ddr4_mem[2].dqs[4]\nset_location_assignment PIN_GC39 -to ddr4_mem[2].dqs_n[4]\nset_location_assignment PIN_FV38 -to ddr4_mem[2].dbi_n[4]\n#---------------------------------------------------------\n# EMIF CH3\n#---------------------------------------------------------\n...\n# # CH3 DQS4 (ECC)\nset_location_assignment PIN_FP46 -to ddr4_mem[3].dq[32]\nset_location_assignment PIN_FT43 -to ddr4_mem[3].dq[33]\nset_location_assignment PIN_FH47 -to ddr4_mem[3].dq[34]\nset_location_assignment PIN_FP42 -to ddr4_mem[3].dq[35]\nset_location_assignment PIN_FT47 -to ddr4_mem[3].dq[36]\nset_location_assignment PIN_FH43 -to ddr4_mem[3].dq[37]\nset_location_assignment PIN_FK46 -to ddr4_mem[3].dq[38]\nset_location_assignment PIN_FK42 -to ddr4_mem[3].dq[39]\nset_location_assignment PIN_FP44 -to ddr4_mem[3].dqs[4]\nset_location_assignment PIN_FT45 -to ddr4_mem[3].dqs_n[4]\nset_location_assignment PIN_FK44 -to ddr4_mem[3].dbi_n[4]\n
-
Change the pin assignment for ddr4_mem[1].dbi_n[4] from PIN_G12 to PIN_W9
set_location_assignment PIN_W9 -to ddr4_mem[1].dbi_n[4]\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/memory/memory.ofss file to use the new presets file generated previously.
[ip]\ntype = memory\n\n[settings]\noutput_name = mem_ss_fm\npreset = f2000x-mem-ecc\n
-
Compile the design with the f2000x OFSS file which calls the Memory OFSS file edited in the previous step.
cd $OFS_ROOTDIR\nofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x <YOUR_WORK_DIRECTORY>\n
-
You may need to adjust the floorplan of the design in order to meet timing after a design change such as this. Refer to the How to Resize the Partial Reconfiguration Region section for information regarding modifications to the floorplan.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System. There are three flows you may use to make modifications.
- Modify the Ethernet Sub-System with OFSS supported changes only. These modifications are supported natively by the build script, and are made at run-time of the build script. This flow is useful for users who only need to leverage natively supported HSSI OFSS settings.
- Modify the Ethernet Sub-System with OFSS supported changes, then make additional custom modifications not covered by OFSS. These modifications will be captured in a presets file which can be used in future compiles. This flow is useful for users who whish to leverage pre-made HSSI OFSS settings, but make additional modifications not natively supported by HSSI OFSS.
- Modify the Ethernet Sub-System without HSSI OFSS. These modification will be made directly in the source files.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#481-walkthrough-modify-the-ethernet-sub-system-channels-with-pre-made-hssi-ofss","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS","text":"This walkthrough describes how to use OFSS to configure the Ethernet-SS. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This walkthrough is useful for users who only need to leverage the pre-made, natively supported HSSI OFSS settings.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the $OFS_ROTDIR/tools/ofss_config/f2000x.ofss file to use the desired Ethernet-SS OFSS configuration. The pre-provided OFSS configurations are as follows:
-
To select the 2x4x25GbE configuration, include the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x25.ofss\n
-
To select the 2x4x10GbE configuration, include the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x10.ofss\n
-
To select the 2x1x100GbE configuration, include the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_2x100.ofss\n
-
Compile the FIM using the f2000x OFSS file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
The resulting FIM will contain the Ethernet-SS configuration specified in Step 3. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#482-walkthrough-add-channels-to-the-ethernet-sub-system-channels-with-custom-hssi-ofss","title":"4.8.2 Walkthrough: Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS","text":"This walkthrough describes how to create an use a custom OFSS file to add channels to the Ethernet-SS and compile a design with a 3x4x25GbE Ethernet-SS configuration. This walkthrough is useful for users who wish to leverage the natively supported HSSI OFSS settings.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a new HSSI OFSS file $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_12x25.ofss with the following contents. In this example we are using 12 channels.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 12\ndata_rate = 25GbE\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss file to use the new HSSI OFSS file generated in Step 3.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_12x25.ofss\n
-
Identify the which channels will be added. You may use the E-Tile Channel Placement Tool to aid in your design. In this example we will add the 4 new 25GbE channels to Channels 8-11.
-
Based on your channel selection, identify which pins will be used. Refer to the [Pin-Out Files for Altera FPGAs] determine the required pins for your device. In this example we are targeting the AGFC023R25A2E2VR0 device. Set the pin assignments in the $OFS_ROOTDIR/syn/setup/eth_loc.tcl file.
set_location_assignment PIN_DL8 -to hssi_if[8].rx_p\nset_location_assignment PIN_DN13 -to hssi_if[9].rx_p\nset_location_assignment PIN_DY8 -to hssi_if[10].rx_p\nset_location_assignment PIN_EB13 -to hssi_if[11].rx_p\n\nset_location_assignment PIN_DL1 -to hssi_if[8].tx_p\nset_location_assignment PIN_DN4 -to hssi_if[9].tx_p\nset_location_assignment PIN_DY1 -to hssi_if[10].tx_p\nset_location_assignment PIN_EB4 -to hssi_if[11].tx_p\n
-
Change the number of QSFP ports from 2 to 3 in the $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv file.
localparam NUM_QSFP_PORTS_USED = 3; // Number of QSFP cages on board used by target hssi configuration\n
-
Edit $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/hssi_wrapper.sv so that the QSFP LED signals use NUM_QSFP_PORTS_USED defined in the previous step.
// Speed and activity LEDS\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_green, // Link up in Nx25G or 2x56G or 1x100G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_yellow, // Link up in Nx10G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_green, // Link up and activity seen\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_red // LOS, TX Fault etc\n
-
Compile the design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x_12x25g\n
-
You may need to adjust the floorplan in order to compile with a PR region that meets timing. Refer to the How to Resize the Partial Reconfiguration Region section for information regarding modifications to the floorplan.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#483-walkthrough-modify-the-ethernet-sub-system-with-pre-made-hssi-ofss-plus-additional-modifications","title":"4.8.3 Walkthrough: Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications","text":"This walkthrough describes how to use OFSS to first modify the Ethernet-SS, then make additional modifications on top. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This flow is useful for users who whish to leverage pre-made OFSS settings, but make additional modifications not natively supported by OFSS.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the $OFS_ROTDIR/tools/ofss_config/f2000x.ofss file to use the desired Ethernet-SS OFSS configuration starting point. Examples for using pre-provided HSSI OFSS files are given below.
-
To select 2x4x25GbE configuration, add the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x25.ofss\n
-
To select 2x4x10GbE configuration, add the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_8x10.ofss\n
-
To select 2x1x100GbE configuration, add the following line
\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_2x100.ofss\n
-
Run the setup stage of the build script with the OFSS file to create a work directory which contains the Ethernet-SS IP configuration specified in Step 3.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss tools/ofss_config/f2000x.ofss f2000x work_f2000x\n
-
Open the Ethernet-SS IP in Quartus Parameter Editor. The IP settings will match te configuration of the OFSS file defined in Step 3. Make any additional modifications in the Parameter Editor as desired.
qsys-edit $OFS_ROOTDIR/work_f2000x/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the New... button in the Presets pane of IP Parameter Editor.
-
In the New Preset window, create a unique Name. In this example the name is f2000x-hssi-presets.
-
Click the ... button to select where to save the preset file. Give it a name, and save it to $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets. Create the presets directory if necessary.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close out of all Quartus GUIs. You do not need to save or compile the IP.
-
Create a new HSSI OFSS file in the $OFS_ROOTDIR/tools/ofss_config/hssi directory named hssi_preset_f2000x.ofss with the following contents. Note that the num_channels and data_rate settings will be overwritten by the contents of the preset file. The preset setting must match the name you selected in Step 7.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 8\ndata_rate = 25GbE\npreset = f2000x-hssi-presets\n
-
Edit the $OFS_ROOTDIR/tools/ofss_config/f2000x.ofss file to use the new HSSI OFSS file created in Step 10.
[include]\n\"$OFS_ROOTDIR\"/tools/ofss_config/f2000x_base.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_host.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/pcie/pcie_soc.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/iopll/iopll.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/memory/memory.ofss\n\"$OFS_ROOTDIR\"/tools/ofss_config/hssi/hssi_preset_f2000x.ofss\n
-
Compile the design using the f2000x OFSS file. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x_hssi_preset\n
-
The resulting FIM will contain the Ethernet-SS configuration specified by the presets file. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#484-walkthrough-modify-the-ethernet-sub-system-without-hssi-ofss","title":"4.8.4 Walkthrough: Modify the Ethernet Sub-System Without HSSI OFSS","text":"This walkthrough describes how to modify the Ethernet-SS wihout using OFSS. This flow will edit the Ethernet-SS IP source directly. This walkthrough is useful for users who wish to make all Ethernet-SS modifications manually, without leveraging HSSI OFSS.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS Agilex\u00ae 7 SoC Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Ethernet-SS IP in Quartus Parameter Editor. Make your modifications in the Parameter Editor.
qsys-edit $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the Generate HDL. Save the design if prompted.
-
Compile the design.
-
If you are not using any other OFSS files in your compilation flow, use the following command to compile. It is recommended to compile a flat design before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh f2000x:flat work_f2000x\n
- If you are using OFSS files for other IP in the design, ensure that the top level OFSS file (e.g.
$OFS_ROOTDIR/tools/ofss_config/f2000x.ofss) does not specify an HSSI OFSS file. Then use the following command to compile. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/f2000x.ofss f2000x:flat work_f2000x\n
-
The resulting FIM will contain the Ethernet-SS configuration contained in the hssi_ss.ip source file.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the f2000x can be done by Remote System Update (RSU) using OPAE commands, or by programming a SOF image to the FPGA via JTAG using Quartus Programer.
Programming via RSU will program the flash device on the board for non-volatile image updates. Programming via JTAG will configure the FPGA for volatile image updates.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"The f2000x has a 10 pin JTAG header on the top side of the board. This JTAG header provides access to either the Agilex 7 FPGA or Cyclone\u00ae 10 BMC device. A JTAG connection with the FPGA Download Cable II can be used to configure the FPGA and to access the Signal Tap Instance. This walkthrough describes how to connect the FPGA Download Cable II and target the Agilex 7 device.
Pre-requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 23.4 tools installed, specifically the
jtagconfig tool.
- This walkthrough requires an Intel FPGA Download Cable II.
Steps:
-
Locate SW2 and SW3 on the f2000x board as shown in the following figure.
-
Set the switches described in the following table:
Switch Position SW2 ON SW3.3 ON -
Connect the FPGA Download Cable to the JTAG header of the f2000x as shown in the figure below.
-
Depending on your server, install the card in a slot that allows room for the JTAG cable. The figure below shows the f2000x installed in a Supermicro server slot.
-
There are two JTAG modes that exist. Short-chain mode is when only the Cyclone 10 device is on the JTAG chain. Long-chain mode is when both the Cyclone 10 and the Agilex\u00ae 7 FPGA are on the JTAG chain. Check which JTAG mode the f2000x board is in by running the following command.
$QUARTUS_ROOTDIR/bin/jtagconfig\n
-
Example output when in short-chain mode (only Cyclone 10 detected):
1) USB-BlasterII [3-4]\n020F60DD 10CL080(Y|Z)/EP3C80/EP4CE75\n
-
Example output when in long-chain mode (both Cyclone 10 and Agilex\u00ae 7 FPGA):
1) USB-BlasterII [3-4]\n020F60DD 10CL080(Y|Z)/EP3C80/EP4CE75\n234150DD AGFC023R25A(.|AE|R0)\n
If the Agilex\u00ae 7 FPGA does not appear on the chain, ensure that the switches described in Step 1 have been set correctly and power cycle the board. Also ensure that the JTAG Longchain bit is set to 0 in BMC Register 0x378. The BMC registers are accessed through SPI control registers at addresses 0x8040C and 0x80400 in the PMCI. Use the following commands to clear the JTAG Longchain bit in BMC register 0x378.
Note: These commands must be executed as root user from the SoC.
Note: You may find the PCIe BDF of your card by running fpgainfo fme.
opae.io init -d <BDF>\nopae.io -d <BDF> -r 0 poke 0x8040c 0x000000000\nopae.io -d <BDF> -r 0 poke 0x80400 0x37800000002\nopae.io release -d <BDF>\n
For example, for a board with PCIe BDF 15:00.0: opae.io init -d 15:00.0\nopae.io -d 15:00.0 -r 0 poke 0x8040c 0x000000000\nopae.io -d 15:00.0 -r 0 poke 0x80400 0x37800000002\nopae.io release -d 15:00.0\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"Every successful run of build_top.sh script creates the file $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/syn_top/output_files/ofs_top.sof which can be used with the FPGA Download Cable II to load the image into the FPGA using the f2000x JTAG access connector.
This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the f2000x. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
Temporarily disable the PCIe AER feature and remove the PCIe port for the board you are going to update. This is required because when you program the FPGA using JTAG, the f2000x PCIe link goes down for a moment causing a server surprise link down event. To prevent this server event, temporarily disable PCIe AER and remove the PCIe port using the following commands:
Note: enter the following commands as root.
-
Find the PCIe BDF and Device ID of your board from the SoC. You may use the OPAE command fpaginfo fme on the SoC to display this information. Run this command on the SoC.
fpgainfo fme\n
Example output:
Intel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.2.3\nBoard Management Controller Build version: 1.2.3\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50103024BF5B5B1\nBitstream Version : 5.0.1\nPr Interface Id : e7926956-9b1b-5ea1-b02c-307f1cb33446\nBoot Page : user1\nUser1 Image Info : b02c307f1cb33446e79269569b1b5ea1\nUser2 Image Info : b02c307f1cb33446e79269569b1b5ea1\nFactory Image Info : b02c307f1cb33446e79269569b1b5ea1\n
In this case, the PCIe BDF for the board on the SoC is 15:00.0, and the Device ID is 0xBCCE.
-
From the SoC, use the pci_device OPAE command to \"unplug\" the PCIe port for your board using the PCIe BDF found in Step 3.a. Run this command on the SoC.
pci_device <B:D.F> unplug\n
For example:
pci_device 15:00.0 unplug\n
-
Find the PCIe BDF of your board from the Host. Use lspci and grep for the device ID found in Step 3.a to get the PCIe BDF. Run this command on the Host.
For example:
lspci | grep bcce\n
Example output:
31:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n31:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
In this case, the board has PCIe BDF 31:00.0 from the Host.
-
From the Host, use the pci_device OPAE command to \"unplug\" the PCIe port for your board using the PCIe BDF found in Step 3.c. Run this command on the Host.
pci_device <B:D.F> unplug\n
For example:
pci_device 31:00.0 unplug\n
-
Launch \"Quartus Prime Programmer\" software from the device which the FPGA Programmer is connected.
$QUARTUS_ROOTDIR/bin/quartus_pgmw\n
Click on Hardware Setup, select USB-BlasterII in the Current Selected Hardware list, and ensure the JTAG Hardware Frequency is set to 16Mhz (The default is 24MHz).
Alternatively, use the following command from the command line to change the clock frequency:
jtagconfig \u2013setparam \u201cUSB-BlasterII\u201d JtagClock 16M\n
-
Click Auto Detect and make sure the Agilex\u00ae 7 FPGA is shown in the JTAG chain. Select the Cyclone 10 and Agilex\u00ae 7 FPGA if prompted.
-
Right-click on the cell in the File column for the Agilex\u00ae 7 FPGA and click on Change file
-
Select the .sof file that you wish to configure the Agilex\u00ae 7 FPGA with.
-
Tick the checkbox below \"Program/Configure\" column and click on Start to program this .sof file.
-
After successful programming, you can close the \"Quartus Prime Programmer\" software. You can answer 'No' if a dialog pops up asking to save the 'Chain.cdf' file. This completes the Agilex\u00ae 7 FPGA .sof programming.
-
Re-plug the PCIe ports on both the SoC and Host.
Note: enter the following commands as root.
-
From the SoC, use the pci_device OPAE command to \"plug\" the PCIe port for your board using the PCIe BDF found in Step 3.a. Run this command on the SoC.
pci_device <B:D.F> plug\n
For example:
pci_device 15:00.0 plug\n
-
From the Host, use the pci_device OPAE command to \"plug\" the PCIe port for your board using the PCIe BDF found in Step 3.c. Run this command on the Host.
pci_device <B:D.F> plug\n
For example:
pci_device 31:00.0 plug\n
-
Verify the f2000x is present by checking expected bitstream ID using fpaginfo fme on the SoC:
fpgainfo fme\n
Example output:
Intel IPU Platform f2000x-PL\nBoard Management Controller NIOS FW version: 1.2.4\nBoard Management Controller Build version: 1.2.4\n//****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : cf00eed4-a82b-5f07-be25-0528baec3711\nBoot Page : user1\nUser1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nUser2 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nFactory Image Info : None\n
Note: The Image Info fields will not change, because these represent the images stored in flash. In this example, we are programming the Agilex\u00ae 7 FPGA directly, thus only the Bitstream ID should change.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#53-remote-system-update","title":"5.3 Remote System Update","text":"The OPAE fpgasupdate tool can be used to update the Cyclone 10 Board Management Controller (BMC) image and firmware (FW), root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate tool only accepts images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then you must also sign the image using the correct keys.
The RSU flow requires that an OPAE enabled design is already loaded on the FPGA. All OPAE commands are run from the SoC, and the new image will be updated from the SoC.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#531-walkthrough-program-the-fpga-via-rsu","title":"5.3.1 Walkthrough: Program the FPGA via RSU","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL with a BIN image via RSU.
Pre-Requisites:
- This walkthrough requires an OFS Agilex\u00ae 7 SoC Attach deployment environment. Refer to the Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL and Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs for instructions on setting up a deployment environment.
- This walkthrough requires a
BIN image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a BIN image. The image used for programming must be formatted with PACsign before programming. This is done automatically by the build script.
Steps:
-
Start in your deployment environment.
-
Use the fpgainfo fme command to obtain the PCIe s:b:d.f associated with your board. In this example, the PCIe s:b:d.f is 0000:15:00.0.
fpgainfo fme\n
Example output:
Intel IPU Platform f2000x\nBoard Management Controller NIOS FW version: 1.2.4 Board Management Controller Build version: 1.2.4 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : fb25ff1d-c31a-55d8-81d8-cbcedcfcea17\nBoot Page : user1\nUser1 Image Info : a566ceacaed810d43c60b0b8a7145591\nUser2 Image Info : a566ceacaed810d43c60b0b8a7145591\nFactory Image Info : a566ceacaed810d43c60b0b8a7145591\n
-
Use the OPAE fpgasupdate command to program a signed image to flash. The flash slot which will be programmed is determined by the signed header of the image.
sudo fpgasupdate <IMAGE> <PCIe B:D.F>\n
-
Example: update User Image 1 in flash
sudo fpgasupdate ofs_top_page1_unsigned_user1.bin 15:00.0\n
-
Example: update User Image 2 in flash
sudo fpgasupdate ofs_top_page2_unsigned_user2.bin 15:00.0\n
-
Example: update Factory Image in flash
sudo fpgasupdate ofs_top_page0_unsigned_factory.bin 15:00.0\n
Note: The label \"unsigned\" in the images produced by the build script mean that the image has been signed with a null header. These images can only be programmed to devices which have not had any keys provisioned.
-
Use the OPAE rsu command to reconfigure the FPGA with the new image. You may select which image to configure from (User 1, User 2, Factory).
sudo rsu fpga --page <PAGE> <PCIe B:D.F>\n
-
Example: configure FPGA with User 1 Image
sudo rsu fpga --page user1 15:00.0\n
-
Example: configure FPGA with User 2 Image
sudo rsu fpga --page user2 15:00.0\n
-
Example: configure FPGA with Factory Image
sudo rsu fpga --page factory 15:00.0\n
-
Run the fpgainfo fme command again to verify the User1 Image Info has been updated.
Example Output:
Intel IPU Platform f2000x\nBoard Management Controller NIOS FW version: 1.2.4 Board Management Controller Build version: 1.2.4 //****** FME ******//\nObject Id : 0xF000000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010302A97C08A3\nBitstream Version : 5.0.1\nPr Interface Id : fb25ff1d-c31a-55d8-81d8-cbcedcfcea17\nBoot Page : user1\nUser1 Image Info : 81d8cbcedcfcea17fb25ff1dc31a55d8\nUser2 Image Info : a566ceacaed810d43c60b0b8a7145591\nFactory Image Info : None\n
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#6-single-event-upset-reporting","title":"6 Single Event Upset Reporting","text":"A Single Event Upset (SEU) is the change in state of a storage element inside a device or system. They are caused by ionizing radiation strikes that discharge the charge in storage elements, such as configuration memory cells, user memory and registers.
Error Detection CRC (EDCRC) circuitry in the Card BMC is used to detect SEU errors. The CRC function is enabled in Quartus Prime Pro to enable CRC status to be reported to the FM61 via the dedicated CRC_ERROR pin.
With the EDCRC there is no method to determine the severity of an SEU error i.e. there is not way to distinguish between non-critical and catastrophic errors. Hence once the SEU error is detected, the Host system must initiate the Card BMC reset procedure.
SEU errors can be read from either the Card BMC SEU Status Register or the PMCI Subsystem SEU Error Indication Register. The processes to read these registers are described in greater detail in the BMC User Guide. Contact your Altera representative for access to the BMC User Guide.
Additionally, refer to the Agilex 7 SEU Mitigation User Guide for more information on SEU detection and mitigation.
"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#appendix","title":"Appendix","text":""},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#appendix-a-fim-fpga-resource-usage","title":"Appendix A: FIM FPGA Resource Usage","text":"The provided design includes both required board management and control functions as well as optional interface exerciser logic that both creates transactions and validates operations. These exerciser modules include:
- HE_MEM - this module creates external memory transactions to the DDR4 memory and then verifies the responses.
- HE_MEM-TG -The memory traffic generator (TG) AFU provides a way for users to characterize local memory channel bandwidth with a variety of traffic configuration features including request burst size, read/write interleave count, address offset, address strobe, and data pattern.
- HE_HSSI - this module creates ethernet transactions to the HSSI Subsystem and then verifies the responses.
The FIM uses a small portion of the available FPGA resources. The table below shows resource usage for a base FIM built with 2 channels of external memory, a small AFU instantiated that has host CSR read/write, external memory test and Ethernet test functionality.
Note: The host exerciser modules allow you to evaluate the FIM in hardware and are removed when you begin development.
The AGFC023R25A2E2VR0 FPGA has the following resources available for base FIM design :
Resource needed / total on device (%) Logic utilization (ALMs) 229,622 / 782,400 ( 29 % ) M20K blocks 1,241 / 10,464 (12 %) Pins 518 / 742 ( 70 % ) IOPLLs 10 / 15 ( 67 % ) The resource usage for the FIM base:
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 229,646.10 29.35 1241 11.86 soc_afu 87,364.80 11.17 273 2.61 soc_pcie_wrapper 37,160.80 4.75 195 1.86 pcie_wrapper 36,233.40 4.63 187 1.79 host_afu 26,462.20 3.38 140 1.34 hssi_wrapper 20,066.30 2.56 173 1.65 pmci_wrapper 8,449.90 1.08 186 1.78 mem_ss_top 7,907.10 1.01 60 0.57 auto_fab_0 2,708.90 0.35 13 0.12 soc_bpf 1,210.20 0.15 0 0.00 qsfp_1 635.50 0.08 4 0.04 qsfp_0 628.70 0.08 4 0.04 fme_top 628.60 0.08 6 0.06 host_soc_rst_bridge 151.40 0.02 0 0.00 rst_ctrl 16.80 0.00 0 0.00 soc_rst_ctrl 16.50 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00 The following example without the he_lb,he_hssi,he_mem,he_mem_tg:
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 162,010.20 20.71 992 9.48 pcie_wrapper 36,771.70 4.70 195 1.86 soc_afu_top 34,851.30 4.45 85 0.81 pcie_wrapper 33,358.90 4.26 175 1.67 hssi_wrapper 20,109.90 2.57 173 1.65 afu_top 14,084.20 1.80 91 0.87 pmci_wrapper 8,447.90 1.08 186 1.78 mem_ss_top 8,379.70 1.07 60 0.57 alt_sld_fab_0 2,725.10 0.35 13 0.12 bpf_top 1,213.00 0.16 0 0.00 fme_top 638.30 0.08 6 0.06 qsfp_top 626.70 0.08 4 0.04 qsfp_top 619.20 0.08 4 0.04 axi_lite_rst_bridge 147.40 0.02 0 0.00 rst_ctrl 17.40 0.00 0 0.00 rst_ctrl 15.90 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00 The following example without the Ethernet Subsystem (no_hssi):
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 189,827.00 24.26 980 9.37 soc_afu_top 67,751.40 8.66 197 1.88 pcie_wrapper 36,909.30 4.72 195 1.86 pcie_wrapper 36,077.70 4.61 187 1.79 afu_top 26,549.40 3.39 140 1.34 pmci_wrapper 8,688.10 1.11 186 1.78 mem_ss_top 8,079.00 1.03 60 0.57 alt_sld_fab_0 1,751.90 0.22 9 0.09 bpf_top 1,186.00 0.15 0 0.00 dummy_csr 664.70 0.08 0 0.00 dummy_csr 662.80 0.08 0 0.00 dummy_csr 661.20 0.08 0 0.00 fme_top 649.40 0.08 6 0.06 axi_lite_rst_bridge 161.70 0.02 0 0.00 rst_ctrl 16.30 0.00 0 0.00 rst_ctrl 16.00 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00 The following example without the Ethernet Subsystem (no_hssi) + no host exercisers (he_lb, he_hssi, he_mem, he_mem_tg):
Entity Name ALMs needed ALM Utilization % M20Ks M20K Utilization % top 139,105.70 17.78 807 7.71 pcie_wrapper 36,518.80 4.67 195 1.86 pcie_wrapper 33,234.50 4.25 175 1.67 soc_afu_top 32,700.00 4.18 85 0.81 afu_top 14,178.20 1.81 91 0.87 pmci_wrapper 8,693.20 1.11 186 1.78 mem_ss_top 7,999.00 1.02 60 0.57 alt_sld_fab_0 1,758.40 0.22 9 0.09 bpf_top 1,183.50 0.15 0 0.00 dummy_csr 667.20 0.09 0 0.00 dummy_csr 666.30 0.09 0 0.00 dummy_csr 663.10 0.08 0 0.00 fme_top 652.80 0.08 6 0.06 axi_lite_rst_bridge 153.80 0.02 0 0.00 rst_ctrl 18.20 0.00 0 0.00 rst_ctrl 16.50 0.00 0 0.00 sys_pll 0.50 0.00 0 0.00"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/dev_guides/fim_dev/ug_dev_fim_ofs/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/doc_modules/Glossary/","title":"Glossary","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/doc_modules/Notices_%26_Disclaimers/","title":"Notices & Disclaimers","text":""},{"location":"hw/f2000x/doc_modules/Notices_%26_Disclaimers/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for the safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/","title":"Shell Technical Reference Manual for Agilex 7 SoC Attach: Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1-overview","title":"1 Overview","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#11-about-this-document","title":"1.1 About this Document","text":"This document describes the hardware architecture for the SoC Attach reference FIM of the Open FPGA Stack (OFS) targeting the Agilex 7 FPGA. After reviewing this document you should understand the features and functions of the components that comprise the FPGA Interface Manager (FIM), also known as the \"shell.\"
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#12-glossary","title":"1.2 Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#13-introduction-to-open-fpga-stack","title":"1.3 Introduction to Open FPGA Stack","text":"The Open FPGA Stack (OFS) is a modular infrastructure of hardware platform components, open source unstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS Provides a framework of FPGA synthesizable code, simulation environment and synthesis/simulation scripts.
The key components of OFS include:
- Target development platforms such as Intel-branded Acceleration Development Platforms (ADPs) and third-party platforms.
- Board Management Controller RTL and firmware that supports telemetry monitoring and capability for remote configuration updates.
- Source accessible, modular FPGA Interface manager (FIM) RTL with a UVM infrastructure unit tests that can be leveraged for your own custom FIM design. The FIM can be thought of as the FPGA shell that provides the I/O ring and timing closed management components for the FPGA.
- Basic building blocks for interconnect and PF/VF translation and arbitration; Platform Interface Manager (PIM) which provides Avalon\u00ae and Arm\u00ae AMBA\u00ae 4 AXI4 bus compliant interfaces.
- AFU examples both in the git repository and workload examples provided by 3rd party vendors.
- Unit level simulation test suite
- System level simulation through a unified verification methodology (UVM)
- OPAE software development kit (APIs, up-streamed Linux drivers and software tools)
- Support for other frameworks to be built on top of the OPAE such as DPDK
These components are available in a two GitHub locations:
- OFS hardware GitHub site
- OPAE software GitHub site
The OFS hardware repository supports hardware development and simulation. Repositories for OFS high-level design support and board management controller RTL and firmware source code are also provided. These repositories can be found in the Opensource Technology GitHub location, which requires entitlement access. To request access, please contact your local Altera sales representative.
Table 1-2 OFS Hardware Repositories
Repository Contains OFS f2000x FIM Github Branch Contains FIM or shell RTL, automated compilation scripts, and unit tests and UVM framework. The OPAE software GitHub site is fully opensource and contains resources for both software and workload developers.
Table 1-3 OFS Software Repositories
OPAE Git Repository Folder Contains OPAE SDK Repo Contains the files for building and installing OPAE SDK from source. Linux DFL Contains OFS Linux drivers that are being upstreamed to the Linux kernel. ofs-platform-afu-bbb Contains the files and scripts to build the platform interface manager. opae-sim Contains the files for an AFU developer to build the Accelerator Functional Unit Simulation Environment (ASE) for workload development. Providing the hardware and software source code and supporting test frameworks in a GitHub repository allows you to customize your designs with the latest versions easily.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14-ofs-features","title":"1.4 OFS Features","text":"The OFS architecture within the FPGA comprises two partitions:
- FPGA Interface Manager (FIM)
- Accelerator Functional Unit (AFU) - AFU SoC Dynamic Region and Static Region - AFU Host Static Region
The FIM or shell provides platform management functionality, clocks, resets, and interface access to the host and peripheral features of the acceleration platform.
The FIM architecture along with the supporting OPAE software supports features such as partial reconfiguration and virtualization.
The FIM provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 datapath interface. The FIM resides in the static region of the FPGA.
The AFU partition is provided for custom acceleration workloads and may contain both static and partial reconfiguration regions.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#141-fpga-interface-manager-fim","title":"1.4.1 FPGA Interface Manager (FIM)","text":"The updated OFS architecture for Agilex\u00ae 7 FPGA devices improves upon the modularity, configurability and scalability of the first release of the OFS architecture while maintaining compatibility with the original design. The primary components of the FPGA Interface Manager or shell of this reference design are:
- P-tile PCIe Subsystem
- E-Tile Ethernet Subsystem
- Memory Subsystem
- Reset Controller
- FPGA Management Engine (FME)
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The AFU Region provides design space for custom workloads and contains both static and partial reconfiguration regions. Partial reconfiguration allows you to update your specific logic blocks or entire workload while the rest of your static design is still in operation.
Note that as discussed previously, the BMC RTL and firmware, the OFS OPAE software stack and support for building your own customer board support package are also provided in separate OFS repositories.
Figure 1-1 OFS for OFS FIM Platform Block Diagram
The table provides an overview of the OFS features targeting the Agilex\u00ae 7 FPGA. This reference FIM (shell) is a starting point for your custom FPGA design. With this initial starting point, you can add or subtract interfaces or ports to different Agilex 7 devices.
Table 1-4 OFS FIM for Agilex\u00ae 7 FPGA Features
Key Feature Description PCIe P-tile PCIe* Gen4x16 to the HostP-tile PCIe* Gen4x16 to the SoC (IceLake-D) Virtualization Host: 2 physical functions SoC: 1 physical function and 3 Virtual functions Memory Four Fabric DDR4 banks, x40 (optional ECC, be configured as x32 and ECC x8 ), 1200 MHz, 4GB Ethernet Eight Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels of 25G Ethernet interfacing to an E-tile Ethernet Subsystem. Configuration and Board Manageability * FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) * Platform Controller Management Interface (PMCI) Module contained within the Agilex 7 FPGA that interfaces through Avalon-Streaming x8 QuadSPI and SPI to a Board Management Controller Partial Reconfiguration Partial Reconfiguration region supported in hardware and software Software Support * Linux DFL drivers targeting OFS FIMs * OPAE Software Development Kit * OPAE Tools"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#subsystem-interfaces","title":"Subsystem Interfaces","text":"The PCIe, Memory and Ethernet interfaces in this design use a new flexible subsystem design that provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 interface. To access these FPGA IP Subsystem documents, please go here and search for the following ID numbers: * 690604: PCIe Subsystem IP User Guide (Note: you must login to myIntel and request entitled access) * 686148: Memory Subsystem IP User Guide (Note: you must login to myIntel and request entitled access) * 773413: [Ethernet Subsystem Intel FPGA IP] (public document)
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FIM contains only one FME, regardless of the number of host interfaces to the FIM. The FME provides management features for the platform and controls reset and loading of the AFU into the partial reconfiguration region of the FPGA.
Any feature, such as a memory interface or global error control that you want to control through FME, must expose its capability to host software drivers. New features are exposed to the FME by adding a device feature header (DFH) register at the beginning of the feature's control status register (CSR) space. The FME CSR maps to physical function 0 (PF0) Base address register 0 (BAR0) so that software can access it through a single PCIe link. For more information about DFHs, refer to the FPGA Device Feature List Framework Overview
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#streaming-datapath","title":"Streaming Datapath","text":"The FIM implements an Arm\u00ae AMBA\u00ae 4 AXI4-Stream bus protocol for data transfer in the FIM. Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels send data packets to and from the host channel IP without data abstraction. Memory-mapped I/O (MMIO) CSR accesses are routed to the ST2MM module, which converts the Arm\u00ae AMBA\u00ae 4 AXI4-Stream to an Arm\u00ae AMBA\u00ae 4 AXI4 memory-mapped protocol.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#virtualization","title":"Virtualization","text":"This design supports virtualization by making use of the virtualization functionality in the PCIe Hard IP and mapping packets to the appropriate physical or virtual function through a PF/VF multiplexer.
This reference FIM example supports 2 PFs for the host and 1PF, 3VFs for the SoC; however, you may extend your configuration to whatever the PCIe Hard IP can support or your application requires.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#142-afu","title":"1.4.2 AFU","text":"An AFU is an acceleration workload that interfaces with the FIM. The AFU boundary in this design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR-specific modules and logic required for partial reconfiguration. Only one partial reconfiguration region is supported in this design for SoC AFU.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways: * Your AFU (workload) for Host (Static Area) and SoC resides in the partial reconfiguration area. * Your AFU is part of the static region and is compiled from a flat design.
In this design, the AFU region is comprised of: * AFU Interface handler to verify transactions coming from AFU region. * PF/VF Mux to route transactions to and from corresponding AFU components: * Host: * ST2MM module. * PCIe loopback host exerciser (HE-LPBK) .
* SoC: * ST2MM module. * Ethernet Subsystem (previous HSSSI) host exerciser (HE-HSSI). * Memory Host Exerciser (HE-MEM). * Traffic Generator to memory (HE-MEM-TG). * Port Gasket (PRG).
- Arm\u00ae AMBA\u00ae 4 AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and Ethernet Interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
For this design the PF/VF Mux provides the following mappings (found in /src/afu_top/mux/top_cfg_pkg.sv and /src/afu_top/mux/soc_top_cfg_pkg.sv):
Table 1-5 PF/VF Mapping
SoC (IceLake-D)
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 2MB AFU peripherals Board peripherals BAR 0 256KB 768KB PF0 VF0 HE-MEM BAR 0 2MB VF1 HE-HSSI BAR 0 2MB VF2 HE-MEM TG BAR 0 2MB Xeon Host
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 1MB AFU peripherals Board peripherals 256KB 768KB PF1 HE-LPBK BAR 0 16KB Figure 1-2 AFU Diagram
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#143-platform-interface-manager","title":"1.4.3 Platform Interface Manager","text":"The PIM provides a way to abstract the Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface to the AFU by providing a library of shims that convert the host channel native packet into other protocols such as Arm\u00ae AMBA\u00ae 4 AXI4 memory-mapped, Avalon\u00ae streaming (Avalon-ST) or Avalon\u00ae memory-mapped (Avalon-MM).
The FPGA or AFU developer implements these interface abstractions in the AFU regions (afu_top and soc_afu_top) of the design.
For more information, refer to the AFU Developer's Guide.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#144-platform-feature-discovery","title":"1.4.4 Platform Feature Discovery","text":"This reference design comes with specific FPGA drivers that are upstreamed to linux-dfl. These drivers abstract the hardware and operating system specific details of the platform to the host.
The FIM implements a device feature header (DFH) at the beginning of each host-discoverable feature in a linked list format that allows an FPGA platform driver running on the host to discover FME, port, and AFU features.
You must implement a 64-bit DFH Device Feature Header register at the beginning (first 8B aligned address) of the feature CSR space for a new feature to be discovered or enumerated by a driver.
During host discovery, the driver traverses the DFH of the first feature from the first address on PF0 BAR0. Based on the information in the DFH, a driver can determine the CSR address range of the feature and other associated details. The end of the DFH contains a \"next DFH offset\" field that points the driver to the DFH of the next feature.
The software must continue traversing the linked list until it sees the EOL (End-Of-List) bit set to 1 in the \"next DFH offset\" field it inspects. A 1 indicates this is the last feature in the feature set. The figure below gives a simple illustration of the feature discovery by traversing the DFH registers. This model is similar to how PCIe enumeration occurs.
Figure 1-3 Device Feature Header Linked List Traversal
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#145-ofs-reference-design","title":"1.4.5 OFS Reference Design","text":"OFS provides FIM designs you can use as a starting point for your own custom design. These designs target a specific programmable acceleration card or development kit and exercise key FPGA device interfaces.
The Agilex\u00ae 7 code line for OFS targets the IPU Platform F2000X-PL. FIM designs are released to for evaluation and use.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#146-fim-simulation","title":"1.4.6 FIM Simulation","text":"OFS provides unit tests and a UVM environment for the FIM and a framework for new feature verification. UVM provides a modular, reusable, and scalable testbench structure by providing an API framework that can be deployed across multiple projects.
The FIM testbench is UVM compliant and integrates third-party verification IPs from Synopsys that require a license.
Verification components include:
- FIM monitor to detect correct design behavior
- FIM assertions for signal level integrity testing
- Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity
- FIM coverage to collect functional data
The verification infrastructure can be found here for evaluation and use. Please refer to the UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach for more information.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#2-ofs-high-level-architecture","title":"2 OFS High Level Architecture","text":"OFS provides distinct data paths that simplify the design and integration process for adding or removing interface modules:
- High Bandwidth data path for AFU-attached high-performance peripherals (Ethernet Subsystem, Memory, workload).
- Low Bandwidth data path for OFS management and slow peripheral components (JTAG, I2C, SMBus).
- AFU Peripheral Fabric (APF) to Board Peripheral Fabric (BPF) path to communicate with interface control and status registers (CSRs) and board components.
- Peer-to-peer datapath between AFU components.
- Peer-to-peer datapath between BPF components.
Depending on your design goals, you can present peripherals to software as:
- OFS managed peripherals with a device feature header that is part of a device feature list.
- Native driver managed peripherals that are exposed through an independent physical function or virtual function.
Figure 2-1 OFS Datapath Structure
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#3-pcie-subsystem","title":"3 PCIe Subsystem","text":"The FIM's PCIe Subsystem is a hierarchical design that targets the P-tile PCIe* hard IP and is configured to support Gen4 speeds and Arm\u00ae AMBA\u00ae 4 AXI4-Stream Data Mover functional mode. The IP supports SR-IOV and is configured to provide 2 PFs for the host and 1PF, 3VFs for the SoC. Native PCIe TLP packets are sent through the PCIe usingArm\u00ae AMBA\u00ae 4 AXI4 Stream Protocol. Before they reach the AFU, the packets go through an adapter in the subsystem that converts any headers to a data mover format. Tag allocation and management for packets sent from the application to the host are done by the PF/VF Mux which is part of the AFU region.
Figure 3-1 OFS FIM RX-TX Datapath
Some key features of the PCIe interface are:
Feature OFS for Agilex 7 FPGA SoC Attach Subsystem Configuration Mode Host: PCIe Gen4x16SoC: PCIe Gen4x16 Tile P-Tile Port Mode Native Endpoint SR-IOV Host: 2 PFs, No VFsSoC: 1 PFs, 3 VFs MSI-X Support Yes Functional Mode Data Mover Profile Basic TLP Bypass No Header Packing Scheme Simple Data Width 512-bit (64-byte) Arm\u00ae AMBA\u00ae 4 AXI4-ST Clock Frequency 500 MHz Tags Supported 128 Reordering Enabled with buffer 64 KB Maximum Payload Size 512 Bytes Memory Requests Supported 1CL, 2CL, 4CL MMIO transaction Size 4B, 8B Control Shadow Interface Enabled Completion Timeout Interface Enabled The PCIe PF, VF and Base Address Register (BAR) configuration can be modified in the PCIe Subsystem Platform Designer GUI interface. The current implementation for the OFS FIM for IPU Platform F2000X-PL is as follows:
Table 3-1 Function and BAR Table for OFS for IPU Platform F2000X-PL
SoC (IceLake-D)
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 2MB AFU peripherals Board peripherals BAR 0 256KB 768KB PF0 VF0 HE-MEM BAR 0 2MB VF1 HE-HSSI BAR 0 2MB VF2 HE-MEM TG BAR 0 2MB Xeon Host
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 1MB AFU peripherals Board peripherals 256KB 768KB PF1 HE-LPBK BAR 0 16KB"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#31-pcie-subsystem-header-format","title":"3.1 PCIe Subsystem Header Format","text":"The first 32 bytes of the TLP from the PCIe subsystem denotes the PCIe header. There are two types of header format supported \u2013 Power User Mode header and Data Mover mode header. The tuser_vendor[0] bit on the Arm\u00ae AMBA\u00ae 4 AXI4-Stream channel indicates the header format of the TLP; tuser_vendor[0] =0 indicates Power User Mode header and tuser_vendor[0] =1 indicates Data Mover Mode header.
The OFS FIM for Agilex 7 FPGA implements the Data Mover Functional Mode. With this implementation, the application has the flexibility to use either mode for PCIe transaction, as shown in the following table. For more detailed information about the PCIe Subsystem, see the PCIe Subsystem FPGA User Guide.
Table 3-2 PCIe Subsystem Header Format Support for OFS for Agilex 7 FPGA
Direction Type Power User Data Mover Host to Endpoint MWr, MRd Yes No CPL/CPLd Yes Yes Msg Yes No Endpoint to Host MWr, MRd Yes Yes Intr Yes (MWr) Yes CPL/CPLd Yes Yes Msg Yes Yes"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#311-power-user-header-format","title":"3.1.1 Power User Header Format","text":"The Power User Format provides user complete control over PCIe Hard IP. The user can implement functionality of interest with finer control over PCIe Transaction Layer Packet (TLP), credit handling and various mode provided by HIP.
The lower 16 bytes of the Power User Format are standard PCIe header as defined by PCIe specification, and the upper 16 bytes are specific to the PCIe Subsystem FPGA IP.
Table 3-3 Power User Header Format
The mapping of the PCIe defined header to the lower 16 bytes of the Arm\u00ae AMBA\u00ae 4 AXI4-Stream data channel is shown in the figure below. Each double word (DW) or 4 bytes in the PCIe header is mapped from big-endian to little-endian on Arm\u00ae AMBA\u00ae 4 AXI4-S data channel.
Figure 3-2 Power User Header Mapping to Arm\u00ae AMBA\u00ae 4 AXI4 Channel
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#312-data-mover-header-format","title":"3.1.2 Data Mover Header Format","text":"The data mover mode allows high bandwidth data transfers to and from Host memory. It hides the complexity of handling PCIe TLPs. This format provides a simple interface for reading and writing to Host Memory. The data mover checks link partner credits before transmitting packets. It also provides MSI-X interrupt generation capability.
In Data Mover Mode, the lower 16 bytes are data mover specific and do not follow a PCIe standard header.
Table 3-4 Data Mover Header Format
The mapping of the data mover header to the lower 16 bytes of the Arm\u00ae AMBA\u00ae 4 AXI4-S data channel is shown below. Each byte in the data mover header is mapped directly to the Arm\u00ae AMBA\u00ae 4 AXI4-S data channel.
Figure 3-3 Data Mover Header Mapping to Arm\u00ae AMBA\u00ae 4 AXI4 Channel
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#32-pcie-subsystem-interface-module","title":"3.2 PCIe Subsystem Interface Module","text":"The PCIe Subsystem Interface module (/ipss/pcie/rtl/pcie_ss_if.sv), provides the supporting interface between software and the PCIe subsystem.
The interface module provides the following:
- Device Feature Header Registers
- Control and Status Registers
- Indirect access to PCIe subsystem CSR registers through a CSR mailbox in the PCIe Subsystem Interface.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#33-data-mover-request-cycles","title":"3.3 Data Mover Request Cycles","text":"For Host read request cycles using the OFS FIM for Agilex 7 FPGA: * All requests in the RX direction will be MMIO. * Requester ID from the request does get sent to the AFU. It is the AFU's responsibility to send back a completion to the host with the correct completer ID. * Prefix is not supported. * Memory Mapped (MM) Mode is not supported. * Slot Number is 0. * Base address is not sent to the AFU. * Local Address field is not used.
For AFU/application request cycles using the OFS FIM for Agilex 7 FPGA: * All requests in the TX direction will be Memory Read/Write. * The tag must be generated by the AFU/application. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0 (non-0 only for switch) * VF Active, VF number and PF number are obtained from Data Mover Header Packet.
Figure 3-4 Data Mover Request Cycles
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#34-data-mover-completion-cycles","title":"3.4 Data Mover Completion Cycles","text":"For Host completion cycles using the OFS FIM for Agilex 7 FPGA: * All completions in the RX direction will be Data Completions. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0. * Data packet responses (for Memory Read requests from AFU) from the PCIe SS may come out of order when the size is >64B.
For AFU/application completion cycles using the OFS FIM for Agilex 7 FPGA: * All requests in the TX direction will be Memory Read/Write. * Requester ID is generated within the FIM. * That tag must be generated by the AFU/application. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0. * VF Active, VF Number and PF number are obtained from the Data Mover Header Packet.
Figure 3-5 Data Mover Completion Cycles
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#4-platform-interface-manager","title":"4 Platform Interface Manager","text":"The FIM interfaces to an application in the AFU region through Arm\u00ae AMBA\u00ae 4 AXI4-Stream channels. This format allows the AFU to access the host channel's raw interface without any translation.
As a FIM developer, you have the option to provide the raw data format associated with the host interface channel to the workload or AFU developer or you can provide an intermediate protocol using Platform Interface Manager Components or your own custom interface.
If you expose the raw Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface of the FIM, workload developers also have the option to convert to a desired protocol using the PIM resources as well.
Refer to the AFU Developer Guide and the FPGA Interface Manager Developer Guide for more information on using the PIM in your design.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#5-interconnect-fabric","title":"5 Interconnect Fabric","text":"There are three types of interconnect fabric in the OFS FIM design: * Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux/demux fabric * AFU Peripheral Fabric (APF) * Board Peripheral Fabric (BPF)
Figure 5-1 Interconnect Fabric Diagram
TLP packets sent from upstream PCIe Subsystem on Arm\u00ae AMBA\u00ae 4 AXI4-Stream channel are demultiplexed in the Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux/demux fabric and routed to the respective PF/VF function based on the PF/VF information in the TLP header, such as vf_active or the PF/VF number. In the opposite direction, TLP packets from downstream PF/VF function are muxed in the fabric and sent to PCIe subsystem over Arm\u00ae AMBA\u00ae 4 AXI4-Stream channel.
All host MMIO requests targeting PF0 BAR0 are routed to the ST2MM module. The ST2MM converts MMIO TLP packets into Arm\u00ae AMBA\u00ae 4 AXI4-Lite memory requests and places the requests onto AFU Peripheral Fabric (APF). AFU peripherals, such as OFS managed AFU features and ST2MM, and Board Peripheral Fabric (BPF) are interconnected by APF. The BPF is the interconnect fabric one hierarchy below APF which connects all the board peripherals. Both APF and BPF allow multiple Arm\u00ae AMBA\u00ae 4 AXI4-Lite master and slave interconnect topology.
If you are modifying the APF or BPF connections, you must use Platform Designer to generate the fabrics directly. Please refer to the FPGA Interface Manager Developer Guide for directions on what files must be modified and how to generate the interconnect.
For modifying the PF/VF mux you must update parameters in these files: * src/includes/top_cfg_pkg.sv * src/common/pf_vf_mux.sv Then make the corresponding update to AFU top level instantiation and connections: * src/afu_top/soc_afu_top.sv(SoC_AFU) and src/afu_top/afu_top.sv (HOST_AFU)
For details on these modifications, please refer to the FIM Interface Manager Developer Guide.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#51-afu-peripheral-fabric-apf","title":"5.1 AFU Peripheral Fabric (APF)","text":"The AFU Peripheral Fabric (APF) is a 64-bit Arm\u00ae AMBA\u00ae 4 AXI4-lite compliant interconnect fabric that connects AFU peripheral modules to board peripheral modules through the Board Peripheral Fabric (BPF).
The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by the APF is listed below. All components are mapped to PF0 BAR0 and implement Arm\u00ae AMBA\u00ae 4 AXI4-lite slave interface. Note that none of the features in the APF mapping are designed to act as a master.
Table 5-1 APF Address Mapping
SoC PF0 BAR 0
Address Size (Byte) Feature 0x00000\u20130xFFFFF 1024K Board Peripherals (See BPF address mapping) AFU Peripherals 0x100000 \u2013 0x10FFFF 64K ST2MM 0x130000 \u2013 0x13FFFF 64K PR Gasket: 4K= PR Gasket DFH, control and status 4K= Port DFH 4K=User Clock 52K=Remote STP 0x140000 \u2013 0x14FFFF 64K AFU Error Reporting (Protocol checker) Host PF0 BAR 0
Address Size (Byte) Feature 0x00000\u20130xFFFFF 1024K Board Peripherals (See BPF address mapping) AFU Peripherals 0x100000 \u2013 0x10FFFF 64K ST2MM 0x140000 \u2013 0x14FFFF 64K AFU Error Reporting (Protocol checker)"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#52-board-peripheral-fabric-bpf","title":"5.2 Board Peripheral Fabric (BPF)","text":"The Board Peripheral Fabric is the 64-bit Arm\u00ae AMBA\u00ae 4 AXI4-Lite compliant interconnect fabric that connects board peripheral modules to APF. The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by BPF is listed below. All components are mapped to PF0 BAR0 and implement Arm\u00ae AMBA\u00ae 4 AXI4-lite slave interface. The Master column indicates if a component also implements Arm\u00ae AMBA\u00ae 4 AXI4-lite master interface which can send requests to the BPF.
Table 5-2 BPF Address Mapping
SoC PF0 BAR 0
Address Size (Byte) Feature Master 0x00000 \u2013 0x0FFFF 64K FME (FME, Error, etc) No 0x10000 \u2013 0x10FFF 4K SoC PCIe Interface No 0x11000 \u2013 0x11FFF 4K Reserved - 0x12000 \u2013 0x12FFF 4K QSFP Controller 0 No 0x13000 \u2013 0x13FFF 4K QSFP Controller 1 No 0x14000 \u2013 0x14FFF 4K Ethernet Subsystem - 0x15000 \u2013 0x15FFF 4K Memory Subsystem 0x80000 \u2013 0xFFFFF 512K PMCI Controller Yes Host PF0 BAR 0
Address Size (Byte) Feature Master 0x00000 \u2013 0x00FFF 4K Host PCIe Interface No 0x80000 \u2013 0xFFFFF 512K PMCI Controller Yes"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#53-arm-amba-4-axi4-stream-pfvf-muxdemux","title":"5.3 Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux routes the PCIe TLP packets from the PCIe subsystem Arm\u00ae AMBA\u00ae 4 AXI4-Stream RX channel to downstream PF/VF based on the pf_num and vf_num information in the PCIe TLP header.
The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux arbitrates PCIe TLP packets from downstream PF/VF to the PCIe SS Arm\u00ae AMBA\u00ae 4 AXI4-S TX channel. The PF/VF Mux/Demux is an M X N switch that allows any M port to target any N port, and any N port to target any M port, where M is the number of host/upstream ports, and N is the numbers functions/downstream ports.
The fpga top package file, /src/afu_top/mux/top_cfg_pkg.sv and soc_top_cfg_pkg.sv and, contains the PF/VF parameters and mappings.
Figure 5-2 PF/VF Mux
The PF/VF mux integration is part of afu_top (src/afu_top/afu_top.sv and /soc_afu_top). There are two independent TX PF/VF MUX trees, labeled \"A\" and \"B\".
Both an A and a B port are passed to each AFU component with a unique PF/VF. You can design your AFU components to send all requests to the primary A port or partition requests across both A and B ports. A typical high-performance AFU sends read requests to the B port and everything else to the A port, giving the arbiter freedom to keep both the host TX and RX channels busy.
In the reference FIM provided for Agilex 7 OFS, the A and B TX trees have been multiplexed down to a single channel for A and another for B. The A/B multiplexer merges them into a single TX stream that will be passed to the tag remapper.
The tag remapper provides unique tags as required by the PCIe specification. Tags are not provided by the PCIe Subsystem FPGA IP. When creating your own AFU you can leverage this module to generate unique tags.
Note that the primary PF/VF Mux A supports RX and TX ports. For the secondary PF/VF Mux B only TX ports are supported and the RX input to the Mux is tied off.
The default mapping is shown below:
Table 5-3 PF/VF Mapping
SoC (IceLake-D)
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 2MB AFU peripherals Board peripherals BAR 0 256KB 768KB PF0 VF0 HE-MEM BAR 0 2MB VF1 HE-HSSI BAR 0 2MB VF2 HE-MEM TG BAR 0 2MB Xeon Host
PF VF Feature BAR Bar Size PF0 OFS managed peripherals BAR 0 1MB AFU peripherals Board peripherals 256KB 768KB PF1 HE-LPBK BAR 0 16KB For information on how to modify the PF/VF mapping for your own design, refer to the OFS FIM Developer User Guide.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#54-afu-interface-handler","title":"5.4 AFU Interface Handler","text":"The AFU Interface Handler resides inline between the PCIe Arm\u00ae AMBA\u00ae 4 AXI4-Stream Adapter and the Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Demux/Mux logic. Its main function is to provide: * Unique PCIe tags \u2013 Each PCIe transaction shares the 512 tags across all VFs in the AFU region * AFU error logging for all VFs in the AFU region
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#541-unified-tag-remapping","title":"5.4.1 Unified Tag Remapping","text":"When a FPGA function sends out a read cycle, it allocates a unique tag which is subsequently used to identify the read completion. The tag is considered busy; it cannot be assigned to another read cycle until read completion. While a tag may be unique within a unit, two different units could unknowingly send out two read cycles of the same tag. The PCIe subsystem requires unique tags for all read cycles irrespective of their origins. Therefore, a mechanism is needed to uniquify tag globally across different units.
OFS contains a tag remapper (tag_remap.sv) that intercepts the read cycle, finds a globally unique tag, and replaces the original tag value. It also restores the original tag value when returning completion to the read requester. tag_remap is placed between the Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface of the PCIE subsystem and the PF/VF Mux/Demux.
The logic is described as follows:
- A sub-module (
ofs_fim_tag_pool) maintains a pool of available tags. - TX read requests are held until a tag is available from the pool by setting tvalid=0 to the host, and tready=0 to the PF/VF Mux/Demux.
- When a TX read is dispatched, the tag is marked busy in the pool.
- The original tag is stored in tag_reg, so it can be recovered when returning a completion to the unit/function.
- Because completion to a read request can split into multiple smaller transfer sizes, responses are monitored and the final completion is detected using PCIe TLP rules.
- Tags are released in the pool only when all requested data are transferred.
- When the completion returns, the original tag is restored from
tag_reg.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#542-afu-error-handling","title":"5.4.2 AFU Error Handling","text":"In this OFS design, the AFU Interface Handler handles error logging for all VFs in the AFU. Errors handled are as follows
Table 5-4 AFU Error Descriptions
Checker Field Description AFU protocol checker (PCIe TLP) Malformed TLP AFU PCIe TLP contains unsupported format type MaxPayloadError AFU memory write payload size exceeds max_payload_length limit MaxReadReqSizeError AFU memory read payload size exceeds max_read_request_size limit MaxTagError AFU memory read request tag value exceeds the maximum supported tag count UnalignedAddrErr The address field in AFU memory write/read request TLP is not DW-aligned. UnexpMMIOResp AFU is sending a MMIO read response with no matching MMIO read request. MMIOTimedOutAFU is not responding to a MMIO read request within the pre-defined response timeout period. MMIODataPayloadOverrunThe number of data payload sent by AFU for a MMIO response (cplD) is more than the data length specified in the response. MMIOInsufficientDataThe number of data payload sent by AFU for a MMIO response (cplD) is less than the data length specified in the response. TxMWrDataPayloadOverrun The number of data payload sent by AFU for a memory write request is more than the data length specified in the request. TxMWrInsufficientData The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU Protocol Checker (Arm\u00ae AMBA\u00ae 4 AXI4-Stream)TxValidViolationThree checkers are implemented in the FIM to catch errors and protocol violations."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#55-tlp-to-arm-amba-4-axi4-lite-memory-mapped-bridge-st2mm","title":"5.5 TLP to Arm\u00ae AMBA\u00ae 4 AXI4-Lite Memory Mapped Bridge (ST2MM)","text":"ST2MM implements the following key features: * Host MMIO bridge * Maps MMIO TLP packets received from the PCIe Subsystem over streaming interface to Arm\u00ae AMBA\u00ae 4 AXI4-Lite memory-mapped request. The memory-mapped request is sent to AFU or Board peripherals over APF and BPF. * Maps Arm\u00ae AMBA\u00ae 4 AXI4-lite MM response received from AFU or Board peripherals to TLP packets and send the packets over ST streaming channel to host HIA subsystem. * Sends MMIO response of all 0\u2019s for MMIO read to unused BAR region. * Interrupt * Sends interrupt packets to the PCIe subsystem when interrupt requests are received from the peripherals. Interrupts can be requested by a peripheral through a memory write to interrupt CSR registers in the ST2MM.
Figure 5-3 ST2MM Module
ST2MM implements both Arm\u00ae AMBA\u00ae 4 AXI4-lite master and slave interfaces that are connected to the designated slave and master port on APF. Host memory requests are sent on the ST2MM master interface to AFP where the requests are routed to the targeted peripherals.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#6-mmio-regions","title":"6 MMIO Regions","text":"The FIM and AFU expose their functionalities to the host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). An MMIO region is an address space within a base address register (BAR) region to which features are memory mapped.
For example, when a feature is mapped to an MMIO region, the CSR registers of that feature are located within the address range of that region. There can be multiple MMIO regions within a BAR region.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#61-feature-region","title":"6.1 Feature Region","text":"A group of related CSRs can be categorized as a feature region. For example, a DMA engine has queue management function and quality of service (QoS) function; these are two different features of the DMA engine. A feature region is contained within a single PCIe BAR and cannot span across two BAR region boundaries.
A Device Feature Header (DFH) register marks the start of the feature region and sub-feature region, and you must place it at the first address of the region. Each DFH starts at 4KB boundary. A DFH register contains information that OPAE software requires to enumerate the feature. It also has an offset field that points to the next DFH in a feature list. OPAE software traverses the linked list of DFHs in each BAR region to discover all the features implemented on the platform.
The EOL field in a DFH marks the end of a DFH list and is only set in the DFH of the last feature in the feature list. The feature type field in the DFH is used to differentiate between the different types of feature region. Basic building blocks (BBB) and private features are always a child of an AFU or FPGA Interface Unit (FIU) and must be contained within an AFU or FIU, respectively.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#611-device-feature-header-dfh-structure","title":"6.1.1 Device Feature Header (DFH) Structure","text":"All DFHs must follow the following structure to be compatible with OPAE software. Note that only features you want to be exposed to the OPAE software must have a DFH.
Table 6-1: DFH Register Structure
Bitfield Name Range Access Description FeatureType 63:60 RO 4\u2019b0000 \u2013 Reserved 4\u2019b0001 \u2013 AFU4\u2019b0010 \u2013 BBB4\u2019b0011 \u2013 Private Feature4'b0100 \u2013 FIM Reserved 59:41 Rsvd Reserved EOL 40 RO End of DFH List1'b0=No other feature header beyond this one1'b1=This is the last feature header NextDFHByteOffset 39:16 RO Next DFH byte offsetNext DFH Address= Current DFH address + Next DFH byte offset. You can also use this value as an indication of the maximum size of the MMIO region occupied by this feature. FeatureRev 15:12 RO For AFU Feature type= AFU major version number that is user defined.All other feature types= Feature revision number FeatureID 11:0 RO For AFU feature type= CoreFIM version numberFor BBB feature type= Intel defined ID for BBBFor private feature type= User-defined ID to identify within an AFU For FIU type=ID for FIU unit (ex. 0 for FME, 1 for Port) You must increment a feature revision number if a feature changes. This change requires a corresponding change in the software to detect the new version and report mismatches between the hardware and software revision number.
When indicating \"FeatureType\" in your DFH Structure, use the following guidance: * FIM= Any static region feature you want discovered through software. FIM features must implement mandatory FIM registers, including a FIM feature ID. * AFU= User Application region; may be PR region. AFU feature must implement mandatory AFU register including an AFU ID. * Private Feature = Linked list of features within the FIM or AFU which are enumerated and managed by separate software and do not require an ID. * Basic Building Blocks (BBB) = Special features within the AFU or FIM that are meant to be reusable basic building blocks. They are enumerated and called by separate software services.
Table 6-2: Mandatory FIM and AFU Registers
The following table details the registers required for a FIM feature or AFU to be discovered by OPAE software. The ID_H and ID_L fields allow you to create a unique feature or AFU identifier that remains unchanged while the FeatureRev in the DFH register increments.
Byte Address offset with respect to DFH Register Name 0x0000 DFH with corresponding FeatureType field implemented 0x0008 ID_L: Lower 64 bits of unique ID number (GUID) 0x0010 ID_H: Upper 64 bits of unique ID number (GUID) 0x0018 Next AFU Table 6-3: Next AFU Register Definition
The following table details the \"Next AFU Register Definition.\" This register is used if you have multiple AFUs that can be used with your FIM.
Bit Attribute Description 63:24 Reserved Reserved 23:0 RO Next AFU DFH Byte Offset Next AFU DFH address =current address + offset; where a value of 0 implies this is the last AFU in the list. Example: AFU0 at address 0x0: Next AFU offset= 0x100 AFU1@ address 0x100: Next AFU offset = 0x100 AFU2 at address 0x200: Next AFU offset = 0x0 (End of AFU List) 6.2 Control and Status Registers
All the Control and Status Registers (CSRs) in the FIM are 64-bit registers with the following MMIO write, and MMIO read support.
Table 6-4: CSR MMIO Read and Write Support
Request Memory Attribute Payload size Memory Ordering MMIO Write UC 4B or 8B Strongly ordered MMIO Read UC 4B or 8B Strongly ordered The FIM does not reorder the MMIO requests or responses. For MMIO writes, there is no reordering of requests in FIM, and UC ordering rules are followed. Similarly, for MMIO reads, there is no re-ordering of requests or responses in the FIM. An AFU may opt to re-order the MMIO read responses but the FIM does not enforce read response ordering.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#621-software-access-to-registers","title":"6.2.1 Software Access to Registers","text":" - Software accesses 64-bit registers as aligned quadwords. For example, to modify a field (bit or byte) in a 64-bit register, the entire quadword is read, the appropriate field(s) are modified, and the entire quadword is written back.
- When updating registers through multiple accesses (whether in software or due to hardware disassembly), certain registers may have specific requirements on how the accesses must be ordered for proper behavior. These are documented as part of the respective register descriptions.
- For compatibility with future extensions or enhancements, software must assign the last read value to all \u201cReserved and Preserved\u201d (RsvdP) fields when written. In other words, any updates to a register must be read so that the appropriate merge between the RsvdP and updated fields occurs. Also, software must assign a value of zero for \u201cReserved and Zero\u201d (RsvdZ) fields when written.
- PCIe locked operations to FPGA hardware registers are not supported. Software must not issue locked operations to access FPGA hardware registers.
In the following two cases, the FIM terminates MMIO Read requests by sending a completion with the data (CplD) specified below: * MMIO Timeout: This occurs when the AFU does not respond within a set timeout. The timeout value is currently configured to 512 pclks (clk_2x). In this case, the FIM returns all 1s.
- Illegal MMIO Accesses: This occurs when the read is accessing undefined registers in the FIM or if an AFU access violation. An example of an access violation is when a PF attempts to access the AFU when it is set to VF mode, or when a VF attempts to access the AFU when it is set to PF mode. In this case, the FME will returns all 0s.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#62-register-attribute-definition","title":"6.2 Register Attribute Definition","text":"Table 6-5: OFS Register Attribute Definitions
Attribute Expansion Description RW Read/Write This bit can be read or written by software. RO Read Only The bit is set by hardware only. Software can only read this bit. Writes do not have any effect. RW1C Read/ Write 1 to Clear Software can read or clear this bit. The software must write 1 to clear this bit. Writing zero to RW1C bit has no effect. Note that a multi-bit RW1C field may exist. In this case, all bits in the field are cleared if a 1 is written to any of the bits. RW1S Read/ Write 1 to Set Software can read this bit. Writing a 1 to the bit sets it to 1. Writing a 0 has no effect. It is not possible for software to set this bit to 0. The 1 to 0 transition can only be performed by HW. RW1CS Read/Write 1 to Clear Sticky Software can read and clear this bit. Writing a 1 to a bit clears it, while writing a 0 to a bit has no effect. This bit is only reinitialized to its default value by a power-on reset. RWD Read/Write Sticky across Hard Reset The bit can be read or written by SW. This bit is sticky or unchanged by any reset type, including Hard Reset. The bit gets cleared only with power on. *S Sticky across Soft Reset The bit will be sticky or unchanged by soft reset. These bits are only re-initialized to their default value by a power-on reset. *D Sticky across Hard Reset The bit is sticky or unchanged by or unchanged by any reset type, including hard reset. The bit gets cleared only with power on. Rsvd Reserved Reserved for future definitions. Currently don\u2019t care bits. RsvdP Reserved and Protected Reserved for future RW implementations. The software must preserve the value of this bit by read modify write. RsvdZ Reserved and Zero Reserved for future RW1C implementations. The software must write zero to this bit."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#623-csr-offset-in-bars","title":"6.2.3 CSR Offset in BARs","text":"The table below captures the FIM and AFU features in the supported BAR regions. The highlighted offset indicates the first DFH in the DFH list of a BAR region where device driver starts the DFH traversal.
Table 6-6: PF0 BAR0 Features SoC AFU
Offset Feature CSR set 0x0000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 Ethernet Interface 0x15000 Memory Interface 0x80000 PMCI 0x100000 St2MM 0x130000 PR Control & Status (Port Gasket) 0x131000 Port CSRs (Port Gasket) 0x132000 User Clock (Port Gasket) 0x133000 Remote SignalTap (Port Gasket) 0x140000 AFU Errors (Protocol Checker) Table 6-7: PF0 BAR0 Features Host AFU
Offset Feature CSR set 0x0000 PCIe Interface"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#7-clocks","title":"7 Clocks","text":"The following table provides external clock sources which correspond to pins on the FPGA that drive blocks of internal logic or are used as a reference clock for internal PLLs. The names in the table are used in the top.sv or are the logical clock names used by their respective functional blocks.
Table 7-1: External Clock Source
Clock Frequency Description SYS_REFCLK 100MHz Reference clock to system IOPLL (sys_pll) which provides FIM system clocks. PCIE_REFCLK0 100MHz PCIe Refclk 0 PCIE_REFCLK1 100MHz PCIe Refclk 1 SOC_PCIE_REFCLK0 100MHz PCIe Refclk 0 on the SOC side SOC_PCIE_REFCLK1 100MHz PCIe Refclk 1 on the SOC side qsfp_ref_clk 156.25MHz Ethernet Refclk ddr4[0].ref_clk 150MHz Refclk for EMIF channel 0 ddr4[1].ref_clk 150MHz Refclk for EMIF channel 1 ddr4[2].ref_clk 150MHz Refclk for EMIF channel 2 ddr4[3].ref_clk 150MHz Refclk for EMIF channel 3 sdm_config_clk 125MHz SDM Config Clock altera_reserved_tck 10MHz Default JTAG Clock Table 7-2: Internal Clocks
Clock Frequency Description clk_sys 400MHz System clock. Primary clock for PCIe datapath. clk_100m 100MHz 100MHz clock. For RemoteSTP and user clock, or any logic that requires a 100MHz clock. clk_csr 100MHz Arm\u00ae AMBA\u00ae 4 AXI4-lite interconnect fabric and CSR clock. Driven by clk_100m. clk_2x 400MHz 2x clock to AFU in PR slot. Driven by clk_sys. clk_1x 171.43MHz 1x clock to AFU in PR slot. uclk_usr 312.50 MHz Configurable \u201cH\u201d clk for AFU. Default 312.50 MHz uclk_usr_div2 156.25 MHz Configurable \u201cL\u201d clk for AFU. Default 156.25 MHz"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#8-reset","title":"8 Reset","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#81-reset-signals","title":"8.1 Reset Signals","text":"The FIM system reset signals are driven according to a their respective requirements derived from their use cases. The behavioral code defining the reset operation is located in the file rst_ctrl.sv. The FIM System Reset Drivers table below provides a list of the input and feedback signals that compose the various resets.
Table 8-1: FIM System Reset Drivers
Reset Description nPERST pin Active low PCIe reset pin from the PCIe card edge that may be set by the host machine for cold or warm resets. nCONFIG pin Active low input to the FPGA that causes the FPGA to lose its configuration data, enter a reset state, and tri-state all I/O pins. Host software must reload the FPGA FIM after nCONFIG is activated. ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. pcie_reset_status Active high reset status from PCIe hard IP. When driven high, this signal indicates that the PCIe IP core is not ready for usermode. locked Active high SYS IOPLL locked signal pcie_cold_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Cold Reset sequence has been completed. pcie_warm_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Warm Reset sequence has been completed. Upon power-on, the reset module in the FIM holds the FIM in reset until all the reset conditions are de-activated:
nPERST signal is asserted.- The
INIT_DONE pin is driven high to indicate core configuration is complete. - The SYS IOPLL is locked.
- The reset status from PCIe hard IP is de-asserted indicating the IP is ready for transfer.
The reset module places the FIM back into reset if any of these conditions becomes active again. The only way to invoke a system reset to the FIM after power-up is to deassert the nPERST pin either by performing a warm reboot or through PCIe driver intervention. There are soft reset signals set aside to allow software to reset the Port, AFU and partial reconfiguration IP.
The following table lists the reset outputs from the rst_ctrl.sv block.
\u200b Table 8-2: FIM System Reset Outputs
Reset Description rst_n_sys pin System general reset synchronous to clk_sys. This is a warm reset of the FPGA logic. Sticky bits in the FME and other CSR registers will not be reset by this signal. rst_n_100m pin System general reset synchronous to clk_100m. ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. pwr_good_n Hardware reset conditioned by ninit_done and the pll_locked signal. The signal is generally set when power has been cycled or a physical hardware fault has occurred, requiring a reset of the FPGA logic. This signal is synchronous to clk_sys. pcie_cold_rst_ack_n Hardware reset conditioned by the pcie_reset_status which is a summary reset driven by the PCIe Hard IP logic tile on the FPGA die. This signal is synchronous to clk_sys. pcie_warm_rst_ack_n Soft reset conditioned by nPERST when the pcie_reset_status is not asserted, meaning a warm reset is desired. Cold reset sticky bits in the PCIe subsystem will not be reset by this signal."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#9-interrupts","title":"9 Interrupts","text":"The OFS platform supports interrupts through MSI-X feature implemented in the PCIe SS. The MSI-X Interrupt feature handles FME and AFU interrupts.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#91-msi-x","title":"9.1 MSI-X","text":"FME interrupts are primarily used to notify the host of error events happened in the FIM. When any of the bits in the FME error status registers is set, an interrupt request is sent to the MSI-X module. There are FME error status registers for various FPGA features such as RAS Error, Temperature monitoring etc.\u202fIf any of those error registers log an error, we trigger an FME interrupt.
An AFU sends an interrupt to the MSI-X module in the PCIE SS on the AXI interrupt request channel.
The MSI-X table entries and PBA vectors are implemented in the PCIE SS. The PCIE SS supports upto 4096 vectors in Static MSI-X mode
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#92-msi-x-entry-table","title":"9.2 MSI-X Entry Table","text":"Each entry in the MSI-X table stores the message address, message data, and vector control for one interrupt vector. The address, data, and vector control information are populated by software (usually done by the PCIe SRIOV driver) after reading the PCIe endpoint configuration space.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#93-msi-x-pba-table","title":"9.3 MSI-X PBA Table","text":"The PBA table contains a corresponding bit for each MSI-X interrupt vector. This PBA bit is set if that interrupt is triggered but is masked. When the interrupt is unmasked, the PBA bit is unset, and an interrupt is sent to the Host. When the Application generates an interrupt from an IRQ source, it sets the corresponding PBA bit.
When the interrupt is triggered by an IRQ Source, the IRQ Processor checks the masking of that interrupt to determine if the PBA bit should be set. If unmasked, the IRQ Processor looks up MSI-X address and data for the interrupt vector and generates the interrupt request over the PCIe EP.
The FIM implements the pending bit array as per the MSI-X specification. When interrupts are enabled the FIM immediately generates an interrupt in response to a suitable event. When interrupt vectors are masked, the pending bit array entry for the corresponding interrupt vector is set without issuing an interrupt. When interrupt vectors are re-enabled, any pending interrupt entries are issued to the PCIe EP
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#94-interrupts-supported","title":"9.4 Interrupts Supported","text":"Table 9-1: OFS for Agilex 7 FPGA Interrupts Supported
PF/VF Vector Assignment PF0 0-5 Reserved 6 FME Error Interrupt PF0.VF00-3User AFU Interrupt"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#95-msi-x-table-locations","title":"9.5 MSI-X Table Locations","text":"The MSI-X vector tables are at BAR4, address 0x2000. The MSI-X PBA tables are at BAR4, address 0x3000.
Table 9-2: PF0 BAR4 Features
Offset Feature CSR set 0x02000 MSI-X Vector Tables 0x03000 MSI-X PBA Tables"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#10-external-memory-interface-emif","title":"10 External Memory Interface (EMIF)","text":"There are 4 EMIF channels (4 DDR4 banks) on the f2000x platform which is targeted OFS FIM for Agilex 7 FPGA. The HE-MEM exerciser module in AFU. ECC is not implemented in this design. Both memory subsystem and HE-MEM implement Arm\u00ae AMBA\u00ae 4 AXI4-MM interface.
**Table 10-1 Memory Subsystem Configuration on the IPU Platform F2000X-PL platform **
EMIF Channel # Width ECC Size (GB) Speed (MT/s) DDR4 Device FPGA Bank 0 x32 x8 4 2400 Three 1Gx16 3D 1 x32 x8 4 2400 Three 1Gx16 3A 2 x32 x8 4 2400 Three 1Gx16 2F 3 x32 x8 4 2400 Three 1Gx16 2E Figure 10-1: EMIF Interfaces
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#101-emif-csr","title":"10.1 EMIF CSR","text":"TheMIF is implemented as an external FME feature in OFS EA and the CSR for the EMIF feature is memory mapped to the FME BAR region. The following table captures the EMIF CSR registers.
Table 10-2: EMIF CSR Registers
EMIF_DFH 0x000 0x300000001000100F EMIF Management DFH FIELD NAME RANGE ACCESS DEFAULT DESCRIPTION FeatureType [63:60] RO 0x3 Feature Type = Private Feature Reserved40 [59:40] RsvdZ 0x0 Reserved NextDfhByteOffset [39:16] RO 0x1000 Next DFH Byte offset FeatureRev [15:12] RO 0x1 Feature Revision FeatureID [11:0] RO 0x00F Feature Id \u200b
EMIF_STAT 0x008 0x0000000000000000 EMIF Calibration Status Register FIELD NAME RANGE ACCESS DEFAULT DESCRIPTION Reserved [63:8] RsvdZ 0x0 Reserved CalFaliure [7:4] RO 0x0 EMIF PHY Calibration Failure (1 bit per interface) CalSuccess [3:0] RO 0x0 EMIF PHY Calibration Successful (1 bit per interface) EMIF_CAPABILI TY 0x010 0x000000000000000F EMIF Capabliity Register FIELD NAME RANGE ACCESS DEFAULT DESCRIPTION Reserved [63:4] RsvdZ 0x0 Reserved EMIFCap [3:0] RO 0xF Attached Memory Capability (1 bit per interface)"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#11-ethernet-interface-subsystem","title":"11 Ethernet Interface Subsystem","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#111-ethernet-interface-overview","title":"11.1 Ethernet Interface Overview","text":"The Ethernet Subsystem provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack. This reference FIM implements an E-tile Ethernet Subsystem IP in a 2x4x25GbE configuration and provides a Linux driver that can be leveraged for customization. Each group of 4x25GbE routes to a QSFP.
For more information about how to reconfigure the Ethernet Subsystem please refer to the [Ethernet Subsystem Intel FPGA IP].
Table 11-1: Ethernet Configurations for example OFS FIMs for Agilex 7 FPGAs
Configuration/Mode Base Fim IP file name hssi_ss_8x25g Number of ports enabled 8 Enabled ports Port 0-7 Port{x} profile 25GbE Port{x} subprofile MAC+PCS Port{x} RSFEC True Port{x} PTP False Enable AN/LT False (for simulation efficiency) Enable NPDME True Enable JTAG to Avalon master bridge False Enable Tx packet classifier NA PTP accuracy mode NA Enable iCAL and pCAL recipe at power True TX/RX maximum frame size 1518 Enforce maximum frame size False Link fault generation mode Bidirectional Stop TX traffic when link partner sends PAUSE Yes Bytes to remove from RX frames Remove CRC bytes Forward RX PAUSE requests False Use source address insertion False Enable TX/RX VLAN detection True Enable asynchronous adapter clocks False Enable preamble Passthrough False Enable asynchronous adapter clocks False Enable strict preamble check False Enable strict SFD check False Average IPG 12 Enable adaptation load soft IP True Additional IPG removed as per AM period 0 PMA adaptation select NRZ_28Gbps_VSR To determine which Transceiver Subsystem port maps to the QSFP A and B lanes, please refer to the following table:
Table 11-2: Transceiver Subsystem Port Mapping
Port Number base 2x4x25G 0 QSFP-A Lane-0 1 QSFP-A Lane-1 2 QSFP-A Lane-2 3 QSFP-A Lane-3 4 QSFP-B Lane-0 5 QSFP-B Lane-1 6 QSFP-B Lane-2 7 QSFP-B Lane-3 8 NA 9 NA 10 NA 11 NA 12 NA 13 NA 14 NA 15 NA Figure 11-1: Transceiver Subsystem Block Diagram
A host exerciser, named he-hssi, is provided in the pr_slot of the AFU partition. The Transceiver susbystem interface to the AFU has an Arm\u00ae AMBA\u00ae 4 AXI4-Stream data and sideband interface.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#112-ofs-ethernet-subsystem-interface-module","title":"11.2 OFS Ethernet Subsystem Interface Module","text":"A wrapper around the Ethernet Subsystem integrates the following features:
- DFH registers
- Control & status registers
- Indirect access to Transceiver SS CSR registers via CSR mailbox in the Ethernet Subsystem interface
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1121-ethernet-subsystem-control-and-status-register-csr-map","title":"11.2.1 Ethernet Subsystem Control and Status Register (CSR) Map","text":"The Ethernet Subsystem connects to the APF which is memory mapped to PF0 BAR0. The Ethernet Subsystem CSR space in the FIM consists of two parts:
- Ethernet Subsystem CSRs assigned from offset 0x000 to 0x7FF
- Additional CSRs are added to FIM at offset 0x800 to 0xFFF
The PCIe subsystem uses Arm\u00ae AMBA\u00ae 4 AXI4 Memory Mapped accesses to read and write Ethernet Subsystem Control and Status Registers in the FIM. The Ethernet Subsystem CSR Map structure is designed to scale according to IP capabilities.
The Ethernet Subsystem CSR Map can be found ipss/hssi/HSSI_SS_CSR.xls.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#114-ethernet-subsytem-software","title":"11.4 Ethernet Subsytem Software","text":"There are two pieces of software related to running the Ethernet Subsystem: The Linux* dfl network driver and the user space OPAE Tools.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1141-ethernet-subsystem-linux-driver","title":"11.4.1 Ethernet Subsystem Linux Driver","text":"The Ethernet subystem is exposed as a feature in the PCIe PF BAR0 region. It has a Device Feature Header (DFH) specifying the interface.
The primary functionality of the driver is to interact with the ethernet MAC and PHY through an indirect register access mailbox implemented by the HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers described above. To aid in RTL bringup, the driver provides a debugfs interface directly to the indirect register access mailbox. For each HSSI interface in the system there would be a directory with the following form containing two files, regaddr and regval: /sys/kernel/debug/dfl-fme.X.Y
To read a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then read the value as string out of regval file. To write a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then write the value as a C hex string to regval file.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1142-ethernet-subsystem-opae-user-space-tool","title":"11.4.2 Ethernet Subsystem OPAE User Space Tool","text":"User space OPAE Tools that are part of OPAE SDK provide support for the Ethernet Subsystem. You can use the --help option to print more information about each of these commands:
- hssi: Provides a means of interacting with the 10G and 100G HSSI AFUs through the host exerciser.
- hssiloopback: Enables and disables Ethernet loopback.
- hssistats: Provides MAC statistics.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#12-partial-reconfiguration","title":"12 Partial Reconfiguration","text":"Partial Reconfiguration (PR) is an Altera FPGA technology that allows users to reconfigure parts of the FPGA device dynamically, while the remainder of the device continues to operate. In a non-partial reconfiguration flow, any change to the design requires full reprogramming of the entire configuration RAM (CRAM) arrays in the device. With partial reconfiguration, you can dynamically reprogram one or more CRAM frames. A partial reconfiguration design has a static region, and a PR regions, which can be modified to implement new logic. The portion of the CRAM on the chip to be reconfigured is contained within a PR region. For the PR flow, your design must be partitioned into static region and reconfigurable region. The static region is the area of your FPGA that is not reconfigured without reprogramming the entire FPGA. An area of the chip that you plan to partially reconfigure is a PR region.
The Port Gasket contains all the PR specific modules and logic, such as PR slot reset/freeze control, user clock, remote STP etc. For this reference example only one PR slot is supported.
The Figure below shows the fundamental concepts required to support PR in OFS platform. There are 4 OFS management DFH \u2013 PR, Port, User Clock and Remote STP in Port Gasket that is exposed to OFS software. These platform capabilities are generally used in conjunction to Partial Reconfiguration. The Figure below also demonstrates the concepts of adding a new interface to the PR region.
Figure 12-1 Partial Reconfiguration Gasket
The isolation logic is provided on the output signals of the PR region to ensure they don\u2019t glitch and affect the interfacing logic in the Static Region (SR). The isolation logic is controlled by the PR Freeze logic during PR operation.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#121-user-clock-support","title":"12.1 User Clock Support","text":"The reference platform provides two user configurable clock (uclk_usr, uclk_usr_div2) for use by the AFU. These clocks are generated by a reconfigurable IOPLL. The control and status of these clocks is expose through two pairs of command and status registers (USR_CLK_FREQ_CMD0 / USR_CLK_FREQ_STS0 & USR_CLK_FREQ_CMD1 / USR_CLK_FREQ_STS1). The first pair controls the fPLL reconfiguration functionality. The second pair controls a clock frequency measurement block.
The following are the default values of the userclk.
uclk_usr: 312.5 MHz
uclk_usr_div2: 156.2 MHz
Table 12-1 usr_clk_* Acceptable Programming Range
Fabric Frequency Range uclk_usr (H) uclk_usr_div2 (L) Details 0-400 MHz 0-800 MHz 0-400 MHz Clocks set on 2x relationship for L<400 MHz 400-800 MHz 400-800 MHz 400-800 MHz Clks are equal for L>400 MHz PLL Reconfiguration
The blue bit stream hardware exposes the low level IOPLL reconfiguration interfaces directly to software control. Through the USR_CLK_FREQ_CMD0 register software can select the reference clock, assert the PLL power down pin and issue reads and writes on the IOPLL Avalon-mm reconfiguration bus. Through the USR_CLK_FREQ_STS0 register software can query the IOPLL active reference clock, locked status and readdata returned from the IOPLL AVMM interface for read requests.
Clock Frequency Counter
The user clocks, generated by the reconfigurable IOPLL, are connected to a frequency measurement circuit. Software selects which of the two clocks to measure by setting the clock select bit in the USER_CLK_FREQ_CMD1 register. After 10ms software can read the frequency, in 10 KHz units, from the USER_CLK_FREQ_STS1 register. Reading this register before 10ms has passed will return undefined data but otherwise has no effect.
Configuring User Clocks
To program the user clock to the desired frequency, user will set the clock-frequency-low and clock-frequency-high fields in the PR AFU GBS .json file to the desired frequency value. During PR, SW will try to provide the closest possible frequency to the value specified in the .json file.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#13-host-exercisers","title":"13 Host Exercisers","text":"The Host Exerciser (HE) modules are responsible for generating traffic with the intent of exercising the specific interface they are designed to connect to. They are intended to test the interface in simulation and hardware, enable measurement of bandwidth and other performance KPIs and, in some cases, provide an example of data movement between interfaces (PCIe to DDR for e.g.) for adoption into a customer application.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#131-he-lb-and-he-mem-host-exerciser","title":"13.1 HE-LB and HE-MEM Host Exerciser","text":"The Host Exerciser Loopback module is a traffic generator that can be attached both to host memory, over PCIe (HE-LB), and to local memory, such as board-attached DDR (HE-MEM). The Host Exerciser (HE) is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth as well as demonstrating data path correctness. The FIM picks up the HE-LB module behind the PIM (Platform Interface Manager). The PIM converts the Arm\u00ae AMBA\u00ae 4 AXI4 with TLP out of the PCIe SS to standard Arm\u00ae AMBA\u00ae 4 AXI4 (MM for completions and Lite for CSR) interfaces. The figure below shows how the HE-LB is instantiated in the FIM.
Figure 13-1 HE-LB Dataflow Diagram
The exerciser has two primary modes: loopback, in which read data is fed back as writes, and read/write mode, in which reads and writes are generated independently. Furthermore, the host_exerciser software provided with OPAE that drives the RTL has two major modes: \"lpbk\" and \"mem\". These modes only select the UUID of the HE AFU, with lpbk selecting a version configured with no local memory (56e203e9-864f-49a7-b94b-12284c31e02b) and mem seeking a version with local memory (8568ab4e-6ba5-4616-bb65-2a578330a8eb). The behavior of each engine with and without local memory is described below.
Figure 13-2 HE Engine Heirarchy
flowchart TB\n classDef gr fill:green,color:white;\n classDef ol fill:olive,color:white;\n classDef rd fill:maroon,color:white;\n\n he_lb(\"he_lb_top\"):::gr --- he_lb_main\n he_mem(\"he_mem_top\"):::gr --- he_lb_main\n he_lb_main:::gr ---- he_lb_engines\n he_lb_engines:::gr ---- mode_rdwr:::rd\n he_lb_engines:::gr ---- mode_lpbk:::rd\n he_lb_engines:::gr ---- mode_atomic:::rd\n he_lb_main --- he_lb_csr:::ol
For more details of HE-LB and HE-MEM IP, please refer to ofs-fim-common/src/common/he_lb/README.md
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#132-memory-traffic-generator-he-mem-tg","title":"13.2 Memory Traffic Generator (HE-MEM-TG)","text":"The memory traffic generator (TG) AFU provides a way for users to characterize local memory channel bandwidth with a variety of traffic configuration features including request burst size, read/write interleave count, address offset, address strobe, and data pattern.
The AFU consists of a series of configurable Arm\u00ae AMBA\u00ae 4 AXI4-MM traffic generators connected to the available local memory channels not attached to HE-MEM. It has a separate CSR block for AFU feature information and high-level status, including a TG_CAPABILITY register that describes the available traffic generators with a 1 bit active high mask and a TG_STAT register that provides the 4-bit, one-hot status of each TG in adjacent fields. The status is decoded left to right as follows: pass, fail, timeout, active. For additional detail, refer to the MEM_TG CSR table ofs-fim-common/src/common/mem_tg/MEM_TG_CSR.xls
Each traffic generator is configured through a separate Avalon-MM interface at incremental offsets of 0x1000 from the AFU DFH. For details on the TG2 configuration space, refer to the MEM_TG_CSR.xls. The default configuration for each TG performs a single-word write followed by a read at address 0. Triggering the start of the test on a TG will initiate a counter to measure the duration of the test which is recorded in the AFU CSR block and used to report memory channel bandwidth.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#132-hssi-host-exerciser-he-hssi","title":"13.2 HSSI Host Exerciser (HE-HSSI)","text":"HE-HSSI is an Ethernet AFU that handles client side ethernet traffic. The reference HE-HSSI has following features:
- HE-HSSI provides an E-tile compatible interface with the Transceiver Subsystem.
- Includes a traffic generator and checker or monitor
- Provides pause signals to the Transceiver Subsystem for XON and XOFF generation
- Generates traffic or incoming traffic that can be looped back into transmit path by enabling loopback mode, which will bypass traffic generator
- At the HE-HSSI interface boundary the Ethernet data interface is Arm\u00ae AMBA\u00ae 4 AXI4-Stream with 64-bit data at eth_clk clock
- The Traffic generator and checker modules have a 64-bit data interface at eth_clk clock.
- The traffic generator supports the following modes:
- Fixed length or Random Length
- Incremental pattern or Random pattern
- Traffic checker does a 32-bit crc check in 10/25G. In the 100G configuration, there is no data integrity check, only a packet counter.
- The CSR of this AFU is accessible through Arm\u00ae AMBA\u00ae 4 AXI4-Stream PCIe TLP interface
- The PCIe TLP to CSR Interface Conversion module converts PCIe TLPs into simple CSR interface
- The CSR space of the traffic generator and checker modules are accessed in an indirect way using mailbox registers
- If used for more than one channel, each channel has a separate traffic generator and traffic checker with separate CSR space.
- Reads and Writes to individual traffic controller CSR spaces can be done by selecting that particular channel using channel select register.
The HE-HSSI Ethernet block diagram is below.
Figure 13-3: HE-HSSI Block Diagram Block Diagram
The diagram below shows the path followed depending on if you enable loopback mode in HE-HSSI or not. In loopback mode, traffic is looped back into the transmit path which will bypass traffic. Alternatively, the traffic generates traffic.
Figure 13-4: HE-HSSI Operation Mode Traffic Patterns
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1322-he-hssi-csr-map","title":"13.2.2 HE-HSSI CSR Map","text":"The reference HSSI AFU contains the following registers and a similar arrangement of register space can be implemented for other use case-specific HSSI AFUs.
- AFU DFH Register: Device feature header for the AFU (AFU_DFH)
- AFU ID Registers: 128-bit UUID for the AFU which occupies two 64-bit registers (AFU_ID_L, AFU_ID_H)
- Mailbox Registers: Command and Data register for traffic controller register access. It follows the standard access method defined for OFS. Access method and implementation is same as Reconfiguration Interface defined for the HSSI FIM. (TRAFFIC_CTRL_CMD, TRAFFIC_CTRL_DATA)
- Channel Select Register: Channel select register for traffic controller mailbox access. It is used in cases where more than one channel is in the AFU, else it defaults to zero, meaning channel-0 is selected. (TRAFFIC_CTRL_PORT_SEL)
- Scratchpad Register: Scratchpad register for CSR access checking. (AFU_SCRATCHPAD)
The CSR excel for HE-HSSI module can be found at ofs-fim-common/src/common/he_hssi/HE_HSSI_CSR.xls.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#133-he-null-overview","title":"13.3 HE-Null Overview","text":"This module is a simple stub that is used to replace various HE and other blocks in the FIM whenever they are bypassed using the qsf compiler directive such as null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. To find out more about these compiler directives, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
Table 13-1 HE-Null DFH
REGISTER NAME ADDRESS ACCESS DEFAULT DESCRIPTION DFH 0x0000 RO 0x0000_0000_1000_0000 DFH register GUID_L 0x0008 RO 0xaa31f54a3e403501 Lower 64b of GUID GUID_H 0x0010 RO 0x3e7b60a0df2d4850 Upper 64b of GUID SCRATCHPAD 0x0018 RW 0x0 Scratchpad Figure 13-5: HE-Null Block Diagram
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14-reliability-accessibility-serviceability-ras-and-error-handling","title":"14 Reliability, Accessibility, Serviceability (RAS) and Error Handling","text":" - Downstream AFU checker: Identifies AFU violations. For example, these checker flags violations of the interface specification.
- Upstream software or PCIe link checker: Identifies invalid traffic from PCIe that violates FIM or PCIe specifications. For example, this checker flags an application sending traffic if it violates the FIM specification or creates a PCIe link issue by causing completion timeout or malformed TLP.
- FIM - Checks for bugs in the FIM fabric.
Errors reported by the checker are logged in either the FME error registers port error registers, or both, as shown in the table below. For more details on each of the registers, please refer to FME_CSR.xls
Table 14-1: Error Registers
MMIO Region Area Register Description FME CoreFIM FME_ERROR FME Error Status Register 0. Registers parity errors, underflow or overflow errors and access mismatches. FME CoreFIM FME_ERROR0_MASK FME Error Mask Register 0. Write a 0 to mask errors in the FME Error Status Register 0. FME External PCIE0_ERROR PCIe0 Error Status Register. FME External PCIE0_ERROR_MASK PCIe0 Error Mask Register 0. Write a 0 to mask errors in the PCIe0 Error Status Register 0. FME CoreFIM FME_FIRST_ERROR First FME Error Register. FME CoreFIM FME_NEXT_ERROR FME Next Error Register. FME CoreFIM RAS_NOFAT_ERR_STAT Reliability/Accessibility/Serviceability (RAS) Non-Fatal Error Status Register. FME CoreFIM RAS_NOFAT_ERR_MASK RAS Non-Fatal Error Mask Register. Write 0 to mask error fields in RAS_NOFAT_ERR_STAT Register. FME CoreFIM RAS_CATFAT_ERR_STAT RAS Catastrophic and Fatal Errors Status Register. FME CoreFIM RAS_CATFAT_ERR_MASK RAS Catastrophic and Fatal Errors Mask Register. Write 0 to mask error fields in the RAS_CATFAT_ERR_STAT Register. FME CoreFIM RAS_ERROR_INJ RAS error Injection Register. PORT CoreFIM PORT_ERROR Port Error Status Register. PORT CoreFIM PORT_FIRST_ERROR Port First Error Register . PORT CoreFIM PORT_MALFORMED_REQ0 Port Malformed Request Register 0. Provides malformed request header LSBs. PORT CoreFIM PORT_MALFORMED_REQ1 Port Malformed Request Register 1. Provides malformed request header MSBs."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#141-fme-errors","title":"14.1 FME Errors","text":""},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1411-fme_error0","title":"14.1.1 FME_ERROR0","text":"The FME_ERROR0 register flags CoreFIM FME errors in the Global Error (GLBL_ERROR) private feature. The error bits in this register are sticky bits. You can only clear these bits through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in FME_ERROR0_MASK register masks the error.
Table 14-2: FME Error Types
Error Type Description Fabric errors FIFO underflow/overflow condition in CoreFIM. These errors only occur if you have introduced bugs into the FIM or during very rare single event upset (SEU) or SEU-like events. Invalid port access A port can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the Port. If it finds a PF is trying to access a port that is mapped to a VF or vice-versa, an error will be reported. Invalid AFU access An AFU can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the AFU associated with the Port. If it finds a PF is trying to access an AFU that is mapped to a VF or vice-versa, an error is reported and a fake response is sent back to the requester to avoid a completion timeout on the host."},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1412-pcie0_error","title":"14.1.2 PCIE0_ERROR","text":"The PCIe Avalon-ST to Arm\u00ae AMBA\u00ae 4 AXI4-Stream bridge monitors the PCIe link for errors and logs any such errors in the PCIE0_ERROR register (in PCIE0 feature region) and PCIE0_ERROR register in the GLBL_ERR private feature. The error bits in the PCIE0_ERROR register are sticky bits that you can only clear through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in PCIE0_ERROR0_MASK masks the error.
If you have other external FME features, you can add similar _ERROR registers to this space. Please refer to the following spreadsheet: https://github.com/OFS/ofs-f2000x-pl/ipss/pcie/rtl/PCIE_CSR.xls or the SystemVerilog file: https://github.com/OFS/ofs-f2000x-pl/ipss/pcie/rtl/pcie_csr.sv for more details on this register.
NOTE
The PCIE0_ERROR register is located in both the Global Error external feature memory space and a separate PCIe external feature memory space. OPAE software supports the PCIe external feature memory space beginning at offset 0x40000 for OFS EA and going forward. PCIe registers beginning at 0x4000 in the Global Error external feature memory space are there for backward compatibility to the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1413-fme_first_error-fme_next_error","title":"14.1.3 FME_FIRST_ERROR, FME_NEXT_ERROR","text":"The FME_FIRST_ERROR register flags which of the FME error reporting registers, such as FME_ERROR0, PCIE0_ERROR0, has reported the first error occurrence. The error fields of the first error register are then continuously logged into the FME_FIRST_ERROR register until a system reset or software clears all the errors in that first error register. Likewise, the FME_NEXT_ERROR indicates which of the FME error reporting registers (except the first error register) has reported the next occurrence of error after the first error register. The error fields of the next error register are continuously logged into the FME_NEXT_ERROR register until a system reset or software clears all the errors in the second error register.
Please refer to fme_csr.sv for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1414-reliability-accessibility-serviceability-ras-error-status","title":"14.1.4 Reliability, Accessibility, Serviceability (RAS) Error Status","text":"The RAS feature in CoreFIM labels errors as non-fatal, fatal or catastrophic based on their impact to the system. * A non-fatal error usually originates from software or an AFU. With a non-fatal error, the user application may still be able to recover from the error by performing a soft reset on the AFU, fixing the user application software if necessary, and clearing the error. On the other hand, a fatal or catastrophic error is non-recoverable and requires the platform to be reset. * Non-fatal errors are logged in the RAS_NOFAT_ERR_STAT register and fatal or catastrophic errors are logged in the RAS_CATFAT_ERR_STAT register.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14141-non-fatal-errors","title":"14.1.4.1 Non-Fatal Errors","text":"The RAS_NOFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It logs the high-level status of non-fatal errors in the hardware. Unlike the error bits in the PCIE0_ERROR and FME_ERROR0 registers which are RW1C (software can write a 1 to clear the error), the error bits in this register are read-only and can only be cleared by system reset. Software has an option to mask the error using RAS_NOFAT_ERR_MASK.
Please refer to FME_CSR.xls for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14142-catastrophic-fatal-errors","title":"14.1.4.2 Catastrophic & Fatal Errors","text":"The RAS_CATFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It captures the high-level status of errors that can only be recovered with a system reset. Therefore, the error bits in the RAS_CATFAT_ERR_STAT register are read-only and can only be cleared by system reset or masked through RAS_CATFAT_ERR_MASK. Please refer to FME_CSR.xls for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1415-ras-error-injection","title":"14.1.5 RAS Error Injection","text":"For software testing purposes, you can inject non-fatal, fatal, and catastrophic errors into the platform through the RAS_ERROR_INJ register. These errors are reported in the RAS_CATFAT_ERR_STAT and RAS_NOFAT_ERR_STAT registers. Please refer to FME_CSR.xls for individual register field descriptions or the SystemVerilog file: fme_csr.sv
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1416-fme-error-interrupts","title":"14.1.6 FME Error Interrupts","text":"In an event of an FME error, the MSI-X module in the FIM generates an interrupt so the host can decide on the next course of action. The FIM does not stall upstream and downstream traffic after the FME error. However, a port error triggered by invalid request from a user AFU stalls all the traffic going from AFU to PCIe. The interrupt capability is discoverable by querying the NumbSuppInterrupt field of the PORT_CAPABILITY register in the Port private feature. The MSI-X vector number is recorded in the InterruptVectorNumber field of the GLBL_ERROR_CAPABILITY register of the Global Error external feature.
An FME error interrupt is generated in response to the FME_ERROR0, PCIE0_ERROR0, RAS_NOFAT_ERR_STAT, or RAS_CATFAT_ERR_STAT registers recording a new, unmasked, error per the rules defined by CoreFIM interrupt feature.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14161-msi-x-masking-pending-bit-array-pba-clearing","title":"14.1.6.1 MSI-X Masking & Pending Bit Array (PBA) Clearing","text":"If the MSI-X vector corresponding to the FME error interrupt is masked, events are recorded in the PBA array. Clearing the FME error status registers clears the corresponding MSI-X PBA entries. If only some events are cleared, the normal interrupt triggering rules apply and a new pending interrupt is registered.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1417-fme-error-handling","title":"14.1.7 FME Error Handling","text":"When the host receives an FME error interrupt, it must take the recommended actions described below to bring the system back to its normal operation.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14171-catastrophicfatal-error","title":"14.1.7.1 Catastrophic/Fatal Error","text":"A system reset is mandatory for any catastrophic or fatal error.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#14172-non-fatal-error","title":"14.1.7.2 Non-Fatal Error","text":"When software receives a non-fatal error interrupt which does not require a system reset, it can take the following steps to clear the error after software handles the error: 1. Set the *_ERROR_MASK register to all 1\u2019s to mask all errors 2. Clear the *_FIRST_ERROR register 3. Clear the *_ERROR register 4. Set *_ERROR_MASK register to all 0\u2019s to enable all errors
- Result: The *_ERROR & *_FIRST_ERROR registers begin capturing new errors.
NOTE
A system reset can only clear RAS Error status registers.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#143-mmio-requests","title":"14.3 MMIO Requests","text":"The FIM is designed to gracefully handle MMIO request scenarios.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1431-unsupported-functions-and-bar","title":"14.3.1 Unsupported Functions and BAR","text":"The PCIe hard IP in the FIM guarantees that only TLP packets for the functions and BARs supported by the FIM (as configured in PCIe HIP IP instantiation) are sent to the FIM.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1432-mmio-request-decoding","title":"14.3.2 MMIO Request Decoding","text":"The packet router and memory decoder in the FIM ensure that only legal MMIO requests are forwarded to the targeted MMIO region. Full address and BAR decoding is done both in the packet router and the memory decoder to ensure the requests are forwarded to the designated CSR region as defined in the MMIO Regions chapter. Any unsolicited/illegal MMIO request is dropped, and an error is reported back to host through the FME error register.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1433-unused-fmeport-csr-regions","title":"14.3.3 Unused FME/Port CSR Regions","text":"All the CSR slaves in FIM which are mapped to the FME or Port CSR regions must always respond to MMIO read requests targeting its associated CSR region. A CSR slave must return all 0s for MMIO reads to its unused CSR region such as a reserved space or a region where no CSR register is implemented for the address.
The FIM ensures MMIO reads to FME or Port CSR regions that are not mapped to any CSR slave always gets a response of all 0s. The memory decoder module and fake responder module in the FIM provide this guaranteed response.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1434-unsupported-mmio-request","title":"14.3.4 Unsupported MMIO Request","text":"Any MMIO request targeting FME or Port CSR regions with a length or address alignment that are not supported by the FIM is dropped, and an error is logged in PCIE0_ERROR register. The MMIO checker module in the FIM guarantees this response. When an unsupported MMIO read request to the FIM CSR region is detected, the FIM sends back a CPL (completion without data) with error status (UR) back to host.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1435-afu-access-violation","title":"14.3.5 AFU Access Violation","text":"AFU access violations refer to the scenarios where a PF is attempting to access the MMIO region of an AFU bound to a VF (virtualization enabled), or when a VF is trying to access the MMIO region of an AFU bound to a PF (virtualization disabled). When such a violation is detected, the FIM drops the request and logs the error in the FME_ERROR0 register. If the request is an MMIO read request, the FIM returns a fake response to the host.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#1436-afu-mmio-response-timeout","title":"14.3.6 AFU MMIO Response Timeout","text":"An AFU MMIO Response timeout functions in the same manner described in the MMIO Response Timeout section.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#15-ofs-design-hierarchy","title":"15 OFS Design Hierarchy","text":"Files for design, build and unit test simulation are found at https://github.com/OFS/ofs-f2000x-pl, release branch ofs-2024.1-1.
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#151-design-guidance","title":"15.1 Design Guidance","text":"The OFS FIM is designed with configurability and scalability in mind. At a high level, these are the necessary steps for a user to customize the design. Please refer to the [FPGA Interface Manager Developer's Guide]
Table 15-2 Features
Step Description Comments 1 Re-configure PCIe HIP for additional VFs (if necessary) * PF0 is mandatory for running OPAE software* Only modification of the VFs (added or subtracted) by the user is recommended. * Default configuration supports 1 PF and 3 VFs. 2 Update Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF MUX-DEMUX configuration (if necessary) * The PF/VF MUX-DEMUX is parameterized for flexibility. You can change, add or delete PF or VF functions by updating the top_cfg_pkg.sv file. * You also have the option of keeping the default configuration and tying off the unused VFs if needed. 3 Update top level and AFU level as necessary * If you integrating additional external interfaces, make the edits at the top level (ofs_top.sv) and propagate the interface down to the AFU level (afu_top.sv and soc_afu_top.sv) 4 Add user implemented function(s) in AFU * All of your implemented functions must have the required Arm\u00ae AMBA\u00ae 4 AXI4-Stream interface for both the data path and the MMIO control path to CSRs. * All CSRs in the user-implemented function must have the required DFH layout. * See host exerciser CSRs for reference. 5 Update UVM testbench * The OFS full chip UVM environment is coded specifically for verifying the default configuration containing the host exercisers for the PCIe, memory, and Ethernet. * User must update the UVM testbench to reflect new RTL behavior for any customization changes. See The UVM Simulation User Guide: OFS for Agilex\u00ae 7 SoC Attach For more information on modifying the FIM, refer to the [FPGA Interface Manager Developer's Guide]
"},{"location":"hw/f2000x/reference_manuals/ofs_fim/mnl_fim_ofs/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/","title":"Platform Evaluation Script: Open FPGA Stack for Agilex 7 SoC Attach FPGAs","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#1-overview","title":"1 Overview","text":""},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the checkout and evaluation of an IPU Platform F2000X-PL development platform using Open FPGA Stack (OFS). After reviewing the document, you will be able to:
- Set-up and modify the script to the your environment
- Compile and simulate an OFS reference design
- Run hardware and software tests to evaluate the complete OFS flow
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#table-1-2-software-version-summary","title":"Table 1-2: Software Version Summary","text":"Component Version Description FPGA Platform Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL Agilex platform you can use for your custom board development OFS FIM Source Code Branch: https://github.com/OFS/ofs-f2000x-pl, Tag: ofs-2024.1-1 OFS Shell RTL for Agilex 7 FPGA (targeting https://github.com/OFS/ofs-f2000x-pl) OFS FIM Common Branch: https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.1-1, Tag: https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.1-1 Common RTL across all OFS-based platforms AFU Examples Branch: examples-afu , Tag:ofs-2024.1-1 Tutorials and simple examples for the Accelerator Functional Unit region (workload region) OPAE SDK Branch: 2.12.0-5, Tag: 2.12.0-5 Open Programmable Acceleration Engine Software Development Kit Kernel Drivers Branch: ofs-2024.1-6.1-2, Tag: ofs-2024.1-6.1-2 OFS specific kernel drivers OPAE Simulation Branch: opae-sim, Tag: 2.12.0-5 Accelerator Simulation Environment for hardware/software co-simulation of your AFU (workload) Quartus Prime Pro Edition Design Software 23.4 Quartus\u00ae Prime Pro Edition Linux Software tool for Altera FPGA Development Operating System Ubuntu 22.04 Operating system on which this script has been tested A download page containing the release and already-compiled FIM binary artifacts that you can use for immediate evaluation on the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL can be found on the OFS ofs-2024.1-1 official release drop on GitHub.
Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#2-introduction-to-ofs-evaluation-script","title":"2 Introduction to OFS Evaluation Script","text":"By following the setup steps and using the OFS evaluation script you can quickly evaluate many features that the OFS framework provides and also leverage this script for your own development.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#21-pre-requisites","title":"2.1 Pre-Requisites","text":"This script uses the following set of software tools which should be installed using the directory structure below. Tool versions can vary.
- Quartus\u00ae Prime Pro Software : The software can be downloaded here. For details on installation of required patches and quartus installation, refer to section 4.2 of the Shell Developer Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
- Synopsys\u00ae VCS Simulator : The simulator can be downloaded in the Synopsys page here
- Siemens\u00ae Questa\u00ae Simulator : The simulator can be downloaded in the Siemens page here
Figure 2-1 Folder Hierarchy for Software Tools
- Download tar(scripts_ofs-2024.1-1_external.tar.gz) from the assets tab of the release page
-
Untar to folder using the following command
tar -xvf scripts_ofs-2024.1-1_external.tar.gz\n````\n\n3. The folder structure containing the clone script (ofs-clone_repositories.sh) and evaluation script (ofs_f2000x_eval.sh) is as shown in the figure 2-2\n\n**Figure 2-2 Directory Structure for the clone script**\n\n``` sh\n## ofs_scripts\n## -> host_chan_mmio.stp\n## -> ofs_f2000x_eval.sh\n## -> README_ofs_f2000x_eval.txt\n## ofs-clone_repositories.sh\n
-
Open the clone script ofs-clone_repositories.sh in any text editor
- Please enter the location of your proxy server to allow access to external internet to build software packages. (lines 6-8)
Note: Failing to add proxy server will prevent cloning of repositories and the user will be unable to build the OFS framework.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
-
Please enter the license file locations (lines 11-13) for the following tool variables
export LM_LICENSE_FILE=<user_license>\n export DW_LICENSE_FILE=<user_license>\n export SNPSLMD_LICENSE_FILE=<user_license>\n
-
Add the name of the directory where you want the platform repositories to be cloned (line 19)
export OFS_RELEASE=ofs-2024.1-1\n
-
After setting the required variables, source the clone script based on which platform you want to test
source ofs-clone_repositories.sh\n
- The script ofs-clone_repositories.sh has different platforms to choose from the menu to clone as shown in the figure 2-3.
Figure 2-3 Directory Structure for OFS Project
-
Once the repositories are cloned, navigate to the directory where you cloned
cd ofs-2024.1-1\n
-
After cloning, the OFS repositories cloned will look as per Figure 2-4.
Figure 2-4 Directory Structure for OFS Project
## ofs-2024.1-1\n## f2000x(OFS platform)\n## -> examples-afu\n## -> ofs-f2000x-pl\n## -> oneapi-asp\n## -> oneAPI-samples\n## -> opae-sim\n## -> opae-sdk\n## -> linux-dfl\n## -> ofs_f2000x_eval.sh\n## -> README_ofs_f2000x_eval.txt\n## -> host_chan_mmio.stp\n
- Once the repositories are cloned, in the platform specific evaluation script, for e.g., in ofs_f2000x_eval.sh, please follow the instructions below which explains which line numbers to change to adapt this script to the user environment.
a) Set-Up Proxy Server (lines 67-69)
Please enter the location of your proxy server to allow access to external internet to build software packages.
export http_proxy=<user_proxy>\nexport https_proxy=<user_proxy>\nexport no_proxy=<user_proxy>\n
b) Tools Location (line 87, 88, 89, 90)
Set Location of Quartus, Synopsys, Questasim and oneAPI Tools
export QUARTUS_TOOLS_LOCATION=/home\nexport SYNOPSYS_TOOLS_LOCATION=/home\nexport QUESTASIM_TOOLS_LOCATION=/home\nexport ONEAPI_TOOLS_LOCATION=/opt\n
In the example above /home is used as the base location of Quartus, Synopsys and Questasim tools, /opt is used for the oneAPI tools
c) PCIe (Bus Number)
The Bus number must be entered by the user after installing the hardware in the chosen server, in the example below \"b1\" is the Bus Number for a single card as defined in the evaluation script.
export OFS_CARD0_BUS_NUMBER=b1\n
The evaluation script uses the bus number as an identifier to interrogate the card. The command below will identify the accelerator card plugged into a server.
lspci | grep acc\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\nb1:00.1 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.2 Processing accelerators: Intel Corporation Device bcce<br>\nb1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\nb1:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
The result identifies the card as being assigned \"b1\" as the bus number so the entry in the script changes to
export OFS_CARD0_BUS_NUMBER=b1\n
The user can also run the following command on the ofs_f2000x_eval.sh script to automatically change the bus number to b1 in the ofs_f2000x_eval.sh script.
grep -rli 'b1' * | xargs -i@ sed -i 'b1' @
if the bus number is 85 for example
85:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)<br>\n85:00.1 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.2 Processing accelerators: Intel Corporation Device bcce<br>\n85:00.3 Processing accelerators: Red Hat, Inc. Virtio network device<br>\n85:00.4 Processing accelerators: Intel Corporation Device bcce<br>\n
the command to change to 85 in the evaluation script would be
grep -rli 'b1' * | xargs -i@ sed -i '85' @
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#ofs-platform-example-n6000-n6001-fseries-dk-iseries-dk-custom_board-choice-line-173","title":"OFS Platform (Example:= n6000, n6001, fseries-dk, iseries-dk, custom_board) choice (line 173)","text":"The script is designed to accommodate many OFS platform choices eg, n6000, n6001, fseries-dk, iseries-dk or a custom_board of your choice. By default the script defaults to n6001 as shown below and is set by a variable on line 173 of the ofs_f2000x_eval.sh script.
export OFS_PLATFORM=n6001\n
but the user can switch platform by changing the OFS_PLATFORM variable, so for example if the user wants to switch to the i-series development kit the command would be
export OFS_PLATFORM=iseries-dk\n
The ofs_f2000x_eval.sh script has now been modified to the server set-up and the user can proceed to build, compile and simulate the OFS stack
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#3-f2000x-evaluation-script","title":"3 f2000x Evaluation Script","text":""},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#31-overview","title":"3.1 Overview","text":"The figure below shows a snapshot of the full evaluation script menu showing all 62 options and each one one of 11 sub-menus which focus on different areas of evaluation. Each of these menu options are described in the next section.
Figure 3-1 ofs_f2000x_eval.sh Evaluation Menu
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#311-ofs-tools-menu","title":"3.1.1 OFS TOOLS MENU","text":"By selecting \"List of Documentation for OFS f2000x Project,\" a list of links to the latest OFS documentation appears. Note that these links will take you to documentation for the most recent release which may not correspond to the release version you are evaluating. To find the documentation specific to your release, ensure you clone the OFS tag that corresponds to your OFS version.
By selecting \"Check Versions of Operating System and Quartus Premier Design Suite\", the tool verifies correct Operating System, Quartus version, kernel parameters, license files and paths to installed software tools.
Menu Option Example Output 1 - List of Documentation for OFS f2000x Project Open FPGA Stack Overview Guides you through the setup and build steps to evaluate the OFS solution https://ofs.github.io 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS) Checking Linux release Linux version 5.15.92-dfl-66b0076c2c-lts (oe-user@oe-host) (x86_64-ese-linux-gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Wed Feb 8 21:21:27 UTC 2023 Checking RedHat release cat: /etc/redhat-release: No such file or directory Checking Ubuntu release cat: /etc/lsb-release: No such file or directory Checking Kernel parameters initrd=\\microcode.cpio LABEL=Boot root=PARTUUID=2beb0add-07bf-4736-bc31-9b60e7b78375 rootfstype=ext4 console=ttyS5,115200 iomem=relaxed intel_iommu=on pci=realloc default_hugepagesz=1G hugepagesz=1G hugepages=4 rootwait console=ttyS0,115200 console=tty0 Checking Licenses LM_LICENSE_FILE is set to port@socket number:port@socket number DW_LICENSE_FILE is set to port@socket number:port@socket number SNPSLMD_LICENSE_FILE is set to port@socket number:port@socket number Checking Tool versions QUARTUS_HOME is set to /home/intelFPGA_pro/23.4/quartus QUARTUS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus IMPORT_IP_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../ip QSYS_ROOTDIR is set to /home/intelFPGA_pro/23.4/quartus/../qsys/bin Checking QPDS Patches Quartus Prime Shell Version 23.4 Build XXX XX/XX/XXXX Patches X.XX SC Pro Edition Copyright (C) XXXX Intel Corporation. All rights reserved."},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#312-ofs-hardware-menu","title":"3.1.2 OFS HARDWARE MENU","text":"Identifies card by PCIe number, checks power, temperature and current firmware configuration.
Menu Option Example Output 3 - Identify Acceleration Development Platform (OFS) f2000x Hardware via PCIe PCIe card detected as 15:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) Host Server is connected to SINGLE card configuration 4 - Identify the Board Management Controller (BMC) Version and check BMC sensors Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** BMC SENSORS ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e 5 - Identify the FPGA Management Engine (FME) Version Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** FME ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e Boot Page : user1 User1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899 User2 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899 Factory Image Info : None 6 - Check Board Power and Temperature Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** POWER ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e ( 1) QSFP (Primary) Supply Rail Voltage : N/A etc ...................... Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** TEMP ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e ( 1) FPGA FABRIC Remote Digital Temperature#3 : 33.00 Celsius etc ...................... 7 - Check Accelerator Port status //****** PORT ******// Object Id : 0xEF00000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 8 - Check MAC and PHY status Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** MAC ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e Number of MACs : 255 mac info is not supported Intel IPU Platform F2000X-PL Board Management Controller NIOS FW version: 1.1.9 Board Management Controller Build version: 1.1.9 //****** PHY ******// Object Id : 0xF000000 PCIe s:b:d.f : 0000:15:00.0 Vendor Id : 0x8086 Device Id : 0xBCCE SubVendor Id : 0x8086 SubDevice Id : 0x17D4 Socket Id : 0x00 Ports Num : 01 Bitstream Id : 0x5010302EB5E4B14 Bitstream Version : 5.0.1 Pr Interface Id : 3e46f208-8b49-5899-9e3d-b8b6a4d25a4e QSFP0 : Connected QSFP1 : Connected //****** HSSI information ******// HSSI version : 1.0 Number of ports : 8 Port0 :25GbE DOWN Port1 :25GbE DOWN Port2 :25GbE DOWN Port3 :25GbE DOWN Port4 :25GbE DOWN Port5 :25GbE DOWN Port6 :25GbE DOWN Port7 :25GbE DOWN"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#313-ofs-pfvf-mux-and-ofss-menu","title":"3.1.3 OFS PF/VF MUX and OFSS MENU","text":"This menu reports the number of PF/VF functions in the reference example and also allows you to reduce the number to 1PF and 1VF to reduce resource utilisation and create larger area for your workload development. This selection is optional and if the user wants to implement the default number of PF's and VF's then option 9, 10 and 11 should not be used. Additionally the code used to make the PF/VF modification can be leveraged to increase or modify the number of PF/VFs in the existing design within the limits that the PCIe Subsystem supports (8PF/2K VFs)
Menu Option Description 9 - Check PF/VF Mux and OFSS Configuration for OFS f2000x Project This menu selection displays the current configuration of all f2000x ofss files located in the following directory $OFS_ROOTDIR/tools/ofss_config Check f2000x base config OFSS set up [ip] type = ofs [settings] platform = f2000x fim = base_x16 family = agilex part = AGFC023R25A2E2VR0 device_id = 6100 Check f2000x hssi_2x100 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GCAUI-4 Check f2000x hssi_2x100_caui2 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 2 data_rate = 100GAUI-2 Check f2000x hssi_8x10 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 10GbE Check f2000x hssi_8x25 OFSS set up [ip] type = hssi [settings] output_name = hssi_ss num_channels = 8 data_rate = 25GbE Check f2000x IOPLL OFSS set up [ip] type = iopll [settings] output_name = sys_pll instance_name = iopll_0 [p_clk] freq = 470 Check f2000x Memory OFSS set up [ip] type = memory [settings] output_name = mem_ss_fm preset = f2000x Check f2000x PCIe Host OFSS set up [ip] type = pcie [settings] output_name = pcie_ss [pf0] bar0_address_width = 21 [pf1] Check f2000x PCIe SoC OFSS set up [ip] type = pcie [settings] output_name = soc_pcie_ss [pf0] num_vfs = 3 bar0_address_width = 21 vf_bar0_address_width = 21 10 - Modify PF/VF Mux and OFSS Configuration for OFS f2000x Project As an example this menu selection modifies the pcie_host.ofss and pcie_soc.ofss file to 1 PF located in the following directory $OFS_ROOTDIR/tools/ofss_config/pcie This option also displays the the modified pcie_host.ofss and pcie_soc.ofss files 11 - Build PF/VF Mux and OFSS Configuration for OFS f2000x Project If option 10 is not used then the default number of PF's and VF's is used to build the FIM, if option 10 is selected then only 1 VF is built to reduce logic utilisation"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#314-ofs-fimpr-build-menu","title":"3.1.4 OFS FIM/PR BUILD MENU","text":"Builds FIM, Partial Reconfiguration Region and Remote Signal Tap
Menu Option Description 12 - Check OFS software versions for OFS f2000x Project OFS_ROOTDIR is set to /home/user_area/ofs-X.X.X/ofs-f2000x-pl OPAE_SDK_REPO_BRANCH is set to release/X.X.X OPAE_SDK_ROOT is set to /home/user_area/ofs-X.X.X/ofs-f2000x-pl/../opae-sdk LD_LIBRARY_PATH is set to /home/user_area/ofs-X.X.X/ofs-f2000x-pl/../opae-sdk/lib64: 13 - Build FIM for f2000x Hardware This option builds the FIM based on the setting for the $OFS_PLATFORM, $FIM_SHELL environment variable. Check this variable in the following file ofs_f2000x_eval.sh 14 - Check FIM Identification of FIM for f2000x Hardware The FIM is identified by the following file fme-ifc-id.txt located at $OFS_ROOTDIR/$FIM_WORKDIR/syn/syn_top/ 15 - Build Partial Reconfiguration Tree for f2000x Hardware This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development 16 - Build Base FIM Identification(ID) into PR Build Tree template This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU workloads 17 - Build Partial Reconfiguration Tree for f2000x Hardware with Remote Signal Tap This option builds the Partial Reconfiguration Tree which is needed for AFU testing/development for the Remote Signal Tap flow 18 - Build Base FIM Identification(ID) into PR Build Tree template with Remote Signal Tap This option copies the contents of the fme-ifc-id.txt into the Partial Reconfiguration Tree for Remote Signal Tap to allow the FIM amd Partial Reconfiguration Tree to match and hence allow subsequent insertion of AFU workloads"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#315-ofs-hardware-programmingdiagnostic-menu","title":"3.1.5 OFS HARDWARE PROGRAMMING/DIAGNOSTIC MENU","text":"The following submenu allows you to: * Program and check flash * Perform a remote system update (RSU) of the FPGA image into the FPGA * Bind virtual functions to VFIO PCIe driver * Run host exerciser (HE) commands such as loopback to test interfaces VFIO PCI driver binding * Read the control and status registers (CSRs) for bound modules that are part of the OFS reference design.
Menu Option Description 19 - Program BMC Image into f2000x Hardware The user must place a new BMC flash file in the following directory $OFS_ROOTDIR/bmc_flash_files. Once the user executes this option a new BMC image will be programmed. A remote system upgrade command is initiated to store the new BMC image 20 - Check Boot Area Flash Image from f2000x Hardware This option checks which location area in FLASH the image will boot from, the default is user1 Boot Page : user1 21 - Program FIM Image into user1 area for f2000x Hardware This option programs the FIM image \"ofs_top_page1_unsigned_user1.bin\" into user1 area in flash 22 - Initiate Remote System Upgrade (RSU) from user1 Flash Image into f2000x Hardware This option initiates a Remote System Upgrade and soft reboots the server and re-scans the PCIe bus for the new image to be loaded 2022-11-10 11:26:24,307 - [[pci_address(0000:b1:00.0), pci_id(0x8086, 0xbcce)]] performing RSU operation 2022-11-10 11:26:24,310 - [[pci_address(0000:b0:02.0), pci_id(0x8086, 0x347a)]] removing device from PCIe bus 2022-11-10 11:26:24,357 - waiting 10 seconds for boot 2022-11-10 11:26:34,368 - rescanning PCIe bus: /sys/devices/pci0000:b0/pci_bus/0000:b0 2022-11-10 11:26:35,965 - RSU operation complete 23 - Check PF/VF Mapping Table, vfio-pci driver binding and accelerator port status This option checks the current vfio-pci driver binding for the PF's and VF's 24 - Unbind vfio-pci driver This option unbinds the vfio-pci driver for the PF's and VF's 25 - Create Virtual Functions (VF) and bind driver to vfio-pci f2000x Hardware This option creates vfio-pci driver binding for the PF's and VF's Once the VF's have been bound to the driver the user can select menu option 23 to check that the new drivers are bound 26 - Verify FME Interrupts with hello_events The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors 27 - Run HE-LB Test This option runs 5 tests 1) checks and generates traffic with the intention of exercising the path from the AFU to the Host at full bandwidth 2) run a loopback throughput test using one cacheline per request 3) run a loopback read test using four cachelines per request 4) run a loopback write test using four cachelines per request 5) run a loopback throughput test using four cachelines per request 28 - Run HE-MEM Test This option runs 2 tests 1) Checking and generating traffic with the intention of exercising the path from FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host 2) run a loopback throughput test using one cacheline per request 29 - Run HE-HSSI Test This option runs 1 test HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic 1) Send traffic through the 10G AFU 30 - Run Traffic Generator AFU Test This option runs 3 tests TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank 1) Run the preconfigured write/read traffic test on channel 0 2) Target channel 1 with a 1MB single-word write only test for 1000 iterations 3) Target channel 2 with 4MB write/read test of max burst length for 10 iterations 31 - Read from CSR (Command and Status Registers) for f2000x Hardware This option reads from the following CSR's HE-LB Command and Status Register Default Definitions HE-MEM Command and Status Register Default Definitions HE-HSSI Command and Status Register Default Definitions"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#316-ofs-hardware-afu-testing-menu","title":"3.1.6 OFS HARDWARE AFU TESTING MENU","text":"This submenu tests partial reconfiguration by building and loading an memory-mapped I/O example AFU/workload, executes software from host, and tests remote signal tap.
Menu Option Description 32 - Build and Compile host_chan_mmio example This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS (Green Bit Stream) ready for hardware programming 33 - Execute host_chan_mmio example This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test 34 - Modify host_chan_mmio example to insert Remote Signal Tap This option inserts a pre-defined host_chan_mmio.stp Signal Tap file into the OFS code to allow a user to debug the host_chan_mmio AFU example 35 - Build and Compile host_chan_mmio example with Remote Signal Tap This option builds the host_chan_mmio example from the following repo $OFS_PLATFORM_AFU_BBB/plat_if_tests/$AFU_TEST_NAME, where AFU_TEST_NAME=host_chan_mmio. This produces a GBS(Green Bit Stream) ready for hardware programming with Remote Signal tap enabled 36 - Execute host_chan_mmio example with Remote Signal Tap This option builds the host code for host_chan_mmio example and programs the GBS file and then executes the test. The user must open the Signal Tap window when running the host code to see the transactions in the Signal tap window"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#317-ofs-hardware-afu-bbb-testing-menu","title":"3.1.7 OFS HARDWARE AFU BBB TESTING MENU","text":"This submenu tests partial reconfiguration using a hello_world example AFU/workload, executes sw from host
Menu Option Description 37 - Build and Compile hello_world example This option builds the hello_ world example from the following repo $FPGA_BBB_CCI_SRC/samples/tutorial/afu_types/01_pim_ifc/$AFU_BBB_TEST_NAME, where AFU_BBB_NAME=hello_world. This produces a GBS(Green Bit Stream) ready for hardware programming 38 - Execute hello_world example This option builds the host code for hello_world example and programs the GBS file and then executes the test"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#319-ofs-unit-test-project-menu","title":"3.1.9 OFS UNIT TEST PROJECT MENU","text":"Builds, compiles and runs standalone simulation block tests. More unit test examples are found at the following location ofs-f2000x-pl/sim/unit_test
Menu Option Result 39 - Generate Simulation files for Unit Test This option builds the simulation file set for running a unit test simulation 40 - Simulate Unit Test dfh_walker and log waveform This option runs the dfh_walker based on the environment variable \"UNIT_TEST_NAME=dfh_walker\" in the evaluation script. A user can change the test being run by modifying this variable"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#3110-ofs-uvm-project-menu","title":"3.1.10 OFS UVM PROJECT MENU","text":"Builds, compiles and runs full chip simulation tests. The user should execute the options sequentially ie 41,42, 43 and 44
Menu Option Description 41 - Check UVM software versions for f2000x Project DESIGNWARE_HOME is set to /home/synopsys/vip_common/vip_Q-2020.03A UVM_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm VCS_HOME is set to /home/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel VERDIR is set to /home/user_area/ofs-X.X.X/fim-f2000x-pl/verification VIPDIR is set to /home/user_area/ofs-X.X.X/fim-f2000x-pl/verification 42 - Compile UVM IP This option compiles the UVM IP 43 - Compile UVM RTL and Testbench This option compiles the UVM RTL and Testbench 44 - Simulate UVM dfh_walking_test and log waveform This option runs the dfh_walking test based on the environment variable \"UVM_TEST_NAME=dfh_walking_test\" in the evaluation script. A user can change the test being run by modifying this variable 45 - Simulate all UVM test cases (Regression Mode) This option runs the f2000x regression mode, cycling through all UVM tests defined in verification/tests/test_pkg.svh file"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#3111-ofs-build-all-project-menu","title":"3.1.11 OFS BUILD ALL PROJECT MENU","text":"Builds the complete OFS flow, good for regression testing and overnight builds
For this menu a user can run a sequence of tests (compilation, build and simulation) and executes them sequentially. After the script is successfully executed, a set of binary files is produced which a you can use to evaluate your hardware. Log files are also produced which checks whether the tests passed.
A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 46 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 41, 42, 43 and 44. These 19 menu options are chosen to build the complete OFS flow covering build, compile and simulation.
Menu Option Result 46 - Build and Simulate Complete f2000x Project Generating Log File with date and timestamp Log file written to /home/guest/ofs-2.3.1/log_files/f2000x_log_2022_11_10-093649/ofs_f2000x_eval.log"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#definition-of-multi-test-set-up","title":"Definition of Multi-Test Set-up","text":"Menu Option 46 above in the evaluation script can be refined to tailor it to the users need and is principally defined by the variable below
MULTI_TEST[A,B]=C
where
A= Total Number of menu options in script B= Can be changed to a number to select the test order C= Menu Option in Script
Example 1 MULTI_TEST[46,0]=2
A= 46 is the total number of options in the script B= 0 indicates that this is the first test to be run in the script C= Menu option in Script ie 2- List of Documentation for OFS f2000x Project
Example 2 MULTI_TEST[46,0]=2 MULTI_TEST[46,1]=9
In the example above two tests are run in order ie 0, and 1 and the following menu options are executed ie 2- List of Documentation for OFS f2000x Project and 9 - Check OFS software versions for OFS f2000x Project
The user can also modify the build time by de-selecting options they do not wish to use, see below for a couple of use-case scenarios.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#default-user-case","title":"Default User Case","text":"A user can run a sequence of tests and execute them sequentially. In the example below when the user selects option 46 from the main menu the script will execute 24 tests ie (main menu options 2, 9, 12, 13, 14, 15, 16, 17, 18, 32, 34, 35, 37, 39, 40, 41, 42, 43 and 44. All other tests with an \"X\" indicates do not run that test
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#user-case-for-ofs-fimpr-build-menu","title":"User Case for OFS FIM/PR BUILD MENU","text":"In the example below when the user selects option 46 from the main menu the script will only run options from the OFS FIM/PR BUILD MENU (7 options, main menu options 12, 13, 14, 15, 16, 17 and 18). All other tests with an \"X\" indicates do not run that test.
"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#4-f2000x-common-test-scenarios","title":"4 f2000x Common Test Scenarios","text":"This section will describe the most common compile build scenarios if a user wanted to evaluate an acceleration card on their server. The Pre-requisite column indcates the menu comamnds that must be run befere executing the test eg To run Test 5 then a user needs to have run option 13, 15 and 16 before running options 23, 24, 25, 32 and 33.
Test Test Scenario Pre-Requisite Menu Option Menu Option Test 1 FIM Build - 13 Test 2 Partial Reconfiguration Build 13 15, 16 Test 3 Program FIM and perform Remote System Upgrade 13 21, 22 Test 4 Bind PF and VF to vfio-pci drivers - 23, 24, 25 Test 5 Build, compile and test AFU on hardware 13, 15, 16 23, 24, 25, 32, 33 Test 6 Build, compile and test AFU Basic Building Blocks on hardware 13, 15, 16 23, 24, 25, 37, 38 Test 7 Build and Simulate Unit Tests - 39, 40 Test 8 Build and Simulate UVM Tests - 42, 43, 44, 45"},{"location":"hw/f2000x/user_guides/ug_eval_ofs/ug_eval_script_ofs_f2000x/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/","title":"OFS Getting Started Guide for Intel Agilex 7 SoC Attach FPGAs","text":"Last updated: November 19, 2024
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#10-about-this-document","title":"1.0 About this Document","text":"The purpose of this document is to help users get started in evaluating the 2024.1-1 version of the SoC Attach release targeting the Intel\u00ae Infrastructure Processing Unit (Intel\u00ae IPU) Platform F2000X-PL. After reviewing this document, a user shall be able to:
- Set up their server environment according to the Best Known Configuration (BKC)
- Build a Yocto image with the OPAE SDK and Linux DFL Drivers included
- Load and verify firmware targeting the SR and PR regions of the board, the BMC, and the ICX-D SoC NVMe and BIOS
- Verify full stack functionality offered by the SoC Attach OFS solution
- Know where to find additional information on other SoC Attach ingredients
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the IPU Platform F2000X-PL. The card is an acceleration development platform (ADP) intended to be used as a starting point for evaluation and development. This document will cover key topics related to initial bring up and development of the IPU Platform F2000X-PL, with links for deeper dives on the topics discussed therein.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#13-references-and-versions","title":"1.3 References and Versions","text":""},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-2-software-and-firmware-component-version-summary-for-soc-attach","title":"Table 2: Software and Firmware Component Version Summary for SoC Attach","text":"Component Version Download link (where applicable) Available FIM Version(s) PR Interface ID: c2dac77b-757c-5e27-a566-aad3ffba2f4e ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell Host Operating System Ubuntu 22.04 LTS Official Release Page Host OPAE SDK 2.12.0-5 https://github.com/OFS/opae-sdk/releases/tag/2.12.0-5 SoC OS meta-intel-ese Reference Distro 1.0-ESE (kirkstone) ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell SoC Kernel Version 6.1.78-dfl ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell SoC OPAE SDK 2.12.0-5 https://github.com/OFS/opae-sdk/releases/tag/2.12.0-5 SoC Linux DFL ofs-2024.1-6.1-2 https://github.com/OFS/linux-dfl/releases/tag/ofs-2024.1-6.1-2 SoC BMC and RTL 1.2.4 ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell SoC BIOS 0ACRH608_REL ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell Not all components shown in Table 2 will have an update available upon release. The OPAE SDK and Linux DFL software stacks are incorporated into a Yocto image and do not need to be downloaded separately. Updates required for the ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell are shown under Table 9 in section 2.0 Updating the IPU Platform F2000X-PL.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-3-related-repositories","title":"Table 3: Related Repositories","text":"Name Location Intel-FPGA-BBB https://github.com/OPAE/intel-fpga-bbb.git OFS-PLATFORM-AFU-BBB https://github.com/OFS/ofs-platform-afu-bbb.git, tag: ofs-2024.1-1 AFU-EXAMPLES https://github.com/OFS/examples-afu.git, tag: ofs-2024.1-1 OPAE-SDK https://github.com/OFS/opae-sdk, tag: 2.12.0-5 LINUX-DFL https://github.com/OFS/linux-dfl, tag: ofs-2024.1-6.1-2 META-OFS https://github.com/OFS/meta-ofs, tag: ofs-2024.1-2"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#14-board-installation-and-server-requirements","title":"1.4 Board Installation and Server Requirements","text":"The F2000X-PL device physical setup procedure and required server settings are detailed in this device's Board Installation Guide: OFS For Agilex\u00ae 7 SoC Attach IPU F2000X-PL. Please review this document and follow proper procedure when installing your device. The rest of this document will assume you have completed basic platform bring-up.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#15-reference-documents","title":"1.5 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.1-1/.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#20-updating-the-ipu-platform-f2000x-pl","title":"2.0 Updating the IPU Platform F2000X-PL","text":"Every IPU Platform F2000X-PL ships with pre-programmed firmware for the FPGA user1, user2, and factory images, the Cyclone 10 BMC RTL and FW, the SoC NVMe, and the SoC BIOS. The combination of FW images in Table 4 compose the official ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell. Upon initial receipt of the board from manufacturing you will need to update two of these regions of flash to conform with the best known configuration for SoC Attach. As shown in Table 9, not all devices require firmware updates. To instruct users in the process of updating on-board IPU Platform F2000X-PL firmware, examples are provided in this guide illustrating the firmware update process for all devices.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-4-ipu-platform-f2000x-pl-fw-components","title":"Table 4: IPU Platform F2000X-PL FW Components","text":"HW Component File Name Version Update Required (Yes/No) Download Location FPGA SR Image1 user1: ofs_top_page1_unsigned_user1.binuser2: ofs_top_page2_unsigned_user2.binfactory: ofs_top_page0_unsigned_factory.bin PR Interface ID: c2dac77b-757c-5e27-a566-aad3ffba2f4e Yes ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell FPGA PR IMAGE2 ofs_pr_afu.green_region_unsigned.gbs N/A Yes Compiled with FIM ICX-D NVMe core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic N/A Yes ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell BMC RTL and FW AC_BMC_RSU_user_retail_1.2.4_unsigned.rsu 1.2.4 No ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell BIOS Version 0ACRH608_REL.BIN 0ACRH608_REL Yes ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell If a component does not have a required update, it will not have an entry on ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell.
1To check the PR Interface ID of the currently programmed FIM and the BMC RTL and FW version, use fpgainfo fme from the SoC. 2Must be programmed if using AFU-enabled exercisers, not required otherwise.
$ fpgainfo fme\nIntel IPU Platform F2000X-PL\n**Board Management Controller NIOS FW version: 1.2.4**\n**Board Management Controller Build version: 1.2.4**\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n**Bitstream Id : 360572752842090923**\nBitstream Version : 5.0.1\n**Pr Interface Id : c2dac77b-757c-5e27-a566-aad3ffba2f4e**\nBoot Page : user1\nUser1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nUser2 Image Info : None\nFactory Image Info : None\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#21-updating-the-f2000x-pl-icx-d-soc-nvme","title":"2.1 Updating the F2000X-PL ICX-D SoC NVMe","text":"The IPU Platform F2000X-PL ships with a Yocto image pre-programmed in NVMe, which is not the same as the SoC Attach OFS image that we will be using. The latter provides only the OPAE SDK and Linux DFL drivers and is fully open source. This section will show how you can use an attached USB drive to load a new image into flash. You will use a serial terminal to install the new image - Minicom and PuTTy terminal emulators have both been tested. minicom is used for demonstration purposes as the serial terminal to access the ICX-D SoC UART connection in this section. Information on compiling your own Yocto image for use with the IPU Platform F2000X-PL is discussed in section 4.0 Compiling a Custom Yocto SoC Image.
Note: Username and password for the default SoC NVMe boot image are \"root\" and \"root@123\".
-
First, make sure to complete the steps in section 1.5.5 Creating a Bootable USB Flash Drive for the SoC, and attach the USB drive either directly into the rear of the IPU Platform F2000X-PL, or into a USB Hub that itself is connected to the board.
-
Ensure your Minicom terminal settings match those shown below. You must direct Minicom to the USB device created in /dev associated with your RS-232 to USB Adapter cable. This cable must be attached to a server that is separate from the one housing your IPU Platform F2000X-PL. Check the server logs in dmesg to figure out which device is associated with your cable: [ 7.637291] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0. In this example the special character file /dev/ttyUSB0 is associated with our cable, and can be connected to using the following command: sudo minicom --color=on -D /dev/ttyUSB0.
-
Change the SoC boot order to boot from USB first. Reboot the server. From your serial Minicom terminal, watch your screen and press 'ESC' key to go into BIOS setup mode. Once BIOS setup comes up as shown below, click the right arrow key six times to move from 'Main' set up menu to 'Boot' setup:
Main setup menu:
Your order of boot devices may differ. You need to move the USB flash up to Boot Option #1 by first using the down arrow key to highlight the USB device then use '+' key to move the USB device to #1 as shown below:
Press 'F4' to save and exit.
-
You will boot into Yocto automatically. Log in with username root and an empty password using Minicom. Take note of the IP address of the board, you can use this to log in without needing the serial cable.
Verify that you have booted from the USB and not the on-board NVMe lsblk -no pkname $(findmnt -n / | awk '{ print $2 }'). You should see a device matching /dev/sd*, and not nvme*n*p*. If you see nvme*n*p*, then review the previous steps.
Record the IP address of the SoC at this time ip -4 addr. This will be used to log in remotely using SSH.
-
Check that 4 partitions created in 1.5.5 Creating a Bootable USB Flash Drive for the SoC are visible to the SoC in /dev/sd*:
$ lsblk -l\nroot@intel-corei7-64:/mnt# lsblk -l\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nsda 8:0 1 117.2G 0 disk\nsda1 8:1 1 38.5M 0 part /boot\nsda2 8:2 1 20G 0 part /\nsda3 8:3 1 44M 0 part [SWAP]\nsda4 8:4 1 97.1G 0 part /mnt\nnvme0n1 259:0 0 59.6G 0 disk\nnvme0n1p1 259:1 0 300M 0 part\nnvme0n1p2 259:2 0 22.1G 0 part\nnvme0n1p3 259:3 0 44M 0 part\n
Mount partition 4, and cd into it.
$ mount /dev/sda4 /mnt`\n$ cd /mnt\n
-
Install the Yocto release image in the SoC NVMe.
$ dd if=core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic of=/dev/nvme0n1 bs=1M status=progress conv=sync\n$ sync\n$ sgdisk -e /dev/nvme0n1\n
The transfer from USB to NVMe will take several minutes.
-
Reboot the SoC and update the SoC BIOS to boot from NVMe. Follow steps 2 and 3 from this section again, and this time move the NVME back to the front of the boot order. The NVMe is named UEFI OS (PCIe SSD) by the BIOS. Press F4 to save and exit.
You can use wget to retrieve a new version of the Yocto release image from meta-ofs once the SoC's network connection is up. Use wget to copy the image to the SoC over the network under /mnt. You may need to delete previous Yocto images to save on space: $ wget --no-check-certificate --user <Git username> --ask-password https://github.com/OFS/meta-ofs/releases/download/ofs-2024.1-2/core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.gz. Uncompress the newly retrieved file: gzip -d core-image-full-cmdline-intel-corei7-64-20240227185330.rootfs.wic.gz. This may take several minutes.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#211-setting-the-time","title":"2.1.1 Setting the Time","text":" -
Use Linux command to set system time using format: date --set=\"STRING\".
date -s \"26 APRIL 2023 18:00:00\"\n
-
Set HWCLOCK to current system time:
hwclock --systohc\n
Verify time is set properly
date\n...\nhwclock --show\n...\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#30-updating-the-ipu-platform-f2000x-pl","title":"3.0 Updating the IPU Platform F2000X-PL","text":""},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#31-preparing-the-ipu-platform-f2000x-pl-soc-for-updates","title":"3.1 Preparing the IPU Platform F2000X-PL SoC for Updates","text":"Updating the IPU Platform F2000X-PL firmware often requires reboots of the SoC or reconfiguration of the FPGA region. If there are processes connected from the host to the SoC that do not expect the downtime to occur, or if the host is not tolerant to a surprise PCie link down, the following instructions can be used to properly orchestrate updates with the host when reboots occur.
Note: Intel IPU Platform F2000X-PL FPGA and BMC updates are initiated by commands issued on the IPU SoC. Issue the following commands from the host to remove any processes that would be impacted by this update. The instructions on properly removing the IPU from PCIe bus require the OPAE SDK to be installed on the host. Refer to section 6.0 Setting up the Host for this process.
-
From a host terminal shell, find PCIe Bus/Device/Function (BDF) address of your Intel IPU Platform F2000X-PL. Run the command lspci | grep bcce to print all boards with a DID that matches bcce.
$ lspci | grep bcce\n31:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\n31:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\n# In this example, 31:00.0 is the proper PCIe BDF of our device\n
-
Shut down all VMs and software applications attached to any PFs/VFs on the host
- Issue the command
sudo pci_device <<PCIe BDF>> unplug on the host to remove the PCIe device from the PCIe bus - Shut down all software applications on the SoC accessing non-management PFs/VFs
- Issue your update command on the SoC, which will cause an SoC reboot and surprise PCIe link down on the host (ex.
reboot, rsu bmc/bmcimg/fpga) - Once you have completed all firmware updates, you may restart application software on the SoC
- Issue command
sudo pci_device <<PCIe BDF>> plug on the host to rescan the PCIe bus and rebind the device to its native driver - Restart software applications on the host
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#32-updating-fim-bmc-and-afu-with-fpgasupdate","title":"3.2 Updating FIM, BMC and AFU with fpgasupdate","text":"The fpgasupdate tool updates the Intel\u00ae C10 10 BMC image and firmware, root entry hash, FPGA Static Region (SR) and user image (PR). The fpgasupdate will only accept images that have been formatted using PACSign. If a root entry hash has been programmed onto the board, then the image will also need to be signed using the correct keys. Please refer to the Security User Guide: Intel Open FPGA Stack for information on creating signed images and on programming and managing the root entry hash. This repository requires special permissions to access - please reach out to your Intel representative to request. The fpgasupdate tool is used to program images into flash memory. The rsu tool is used to configure the FPGA/BMC with an image that is already stored in flash memory, or to switch between user1 and user2 images. All images received from an official Intel release will be \"unsigned\", as described in the Security User Guide: Intel Open FPGA Stack.
Note: 'Unsigned' in this context means the image has passed through PACsign and has had the proper security blocks prepended using a set of 'dummy' keys. FIMs with image signing enabled will require all programmable images to pass through PACsign even if the currently programmed FIM/BMC do not require specific keys to authenticate.
There are two regions of flash you may store FIM images for general usage, and one backup region. These locations are referred to as user1, user2, and factory. The factory region is not programmed by default and can only be updated once keys have been provisioned. The BMC FW and RTL will come pre-programmed with version 1.1.9.
Updating the FIM from the SoC requires the SoC be running a Yocto image that includes the OPAE SDK and Linux DFL drivers. Updating the FIM using fpgasupdate also requires an OFS enabled FIM to be configured on the F2000X-PL, which it will ship with from manufacturing. You need to transfer any update files to the SoC over SSH. The OPAE SDK utility fpgasupdate will be used to update all ofthe board's programmable firmware . This utility will accept files of the form *.rsu, *.bin, and *.gbs, provided the proper security data blocks have been prepended by PACSign. The default configuration the IPU platform ships with will match the below:
$ fpgainfo fme\nIntel IPU Platform F2000X-PL\n**Board Management Controller NIOS FW version: 1.2.4**\n**Board Management Controller Build version: 1.2.4**\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\n**Bitstream Id : 360572752842090923**\nBitstream Version : 5.0.1\n**Pr Interface Id : c2dac77b-757c-5e27-a566-aad3ffba2f4e**\nBoot Page : user1\nUser1 Image Info : 9e3db8b6a4d25a4e3e46f2088b495899\nUser2 Image Info : None\nFactory Image Info : None\n
To load a new update image, you need to pass the IPU's PCIe BDF and the file name to fpgasupdate on the SoC. The below example will update the user1 image in flash:
$ fpgasupdate ofs_top_page1_unsigned_user1.bin 15:00.0\n
After loading an update image, rsu fpga/bmc/bmcimg can be used to reload the firmware and apply the change, as discussed below. An RSU of the BMC will always cause a reload of both the BMC and FPGA images.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#33-loading-images-with-rsu","title":"3.3 Loading images with rsu","text":"RSU performs a Remote System Update operation on an IPU Platform F2000X-PL, given its PCIe address. An rsu operation sends an instruction to the device to trigger a power cycle of the card only if run with bmcimg. rsu will force reconfiguration from flash for either the BMC or FPGA. PCIe Advanced Error Reporting (AER) is temporarily disabled for the card when RSU is in progress
The IPU Platform F2000X-PL contains two regions of flash you may store FIM images. These locations are referred to as user1 and user2. After an image has been programmed to either of these regions in flash using fpgasupdate, you may perform an rsu to reconfigure the Agilex 7 FPGA with the new image stored in flash. This operation will indicate to the BMC which region to configure the FPGA device from after power-on.
If the factory image has been updated, Intel strongly recommends you immediately RSU to the factory image to ensure the image is functional.
You can determine which region of flash was used to configure their FPGA device using the command fpgainfo fme and looking at the row labelled Boot Page.
When loading a new FPGA SR image, use the command rsu fpga. When loading a new BMC image, use the command rsu bmc. When using the RSU command, you may select which image will be configured to the selected device. For example, when performing an RSU for the Intel Agilex 7 FPGA, you may select to configure the user1, user2, or factory image. When performing an RSU for the C10 BMC, you may select to configure the user or factory image. You may also use RSU to reconfigure the SDM on devices that support it. The RSU command sends an instruction to the BMC to reconfigure the selected device from an image stored in flash.
$ rsu fpga --page=user1 15:00.0\n
Useage:
rsu bmc --page=(user|factory) [PCIE_ADDR]\nrsu fpga --page=(user1|user2|factory) [PCIE_ADDR]\nrsu sdm [PCIE_ADDR]\n
You can use RSU to change which page in memory the FPGA will boot from by default.
Synopsis:
rsu fpgadefault --page=(user1|user2|factory) --fallback=<csv> 15:00.0\n
Use to set the default FPGA boot sequence. The --page option determines the primary FPGA boot image. The --fallback option allows a comma-separated list of values to specify fallback images.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#34-updating-the-icx-d-soc-bios","title":"3.4 Updating the ICX-D SoC BIOS","text":"The ICX-D SoC NVMe comes pre-programmed with BIOS v7 (0ACRH007). This version will need to be replaced with 0ACRH608_REL. BIOS update files come in the form 0ACRH\\<\\<version>>.bin, and can be downloaded on ofs-2024.1-1 Release for Agilex 7 SoC Attach Reference Shell. This update process is in-band, and requires you to download and install a BIOS UEFI utility from AMI called \"APTIO V AMI Firmware Update Utility\", available here. This package will install a utility in the UEFI shell called AfuEfix64.efi which will be used to overwrite the ICX-D BIOS.
-
Check your BIOS Version. Reboot the SoC and wait for the BIOS version to be shown. In this example, the BIOS will need to be updated.
-
Download both the ICX-D SoC Update image and APTIO V AMI Firmware Update Utility. Unzip the BIOS update image and locate your BIOS update binary. Unzip Aptio_V_AMI_Firmware_Update_Utility.zip, and then navigate to Aptio_V_AMI_Firmware_Update_Utility\\afu\\afuefi\\64 and unzip AfuEfi64.zip. The file we need from this package is called AfuEfix64.efi.
-
Copy both files over to the SoC into /boot/EFI using the SoC's IP.
$ scp 0ACRH608_REL.BIN root@XX.XX.XX.XX:/boot/EFI\n$ scp AfuEfix64.efi root@XX.XX.XX.XX:/boot/EFI\n
-
Reboot the SoC from a TTY Serial terminal. Watch your screen and press 'ESC' key to go into BIOS setup mode. Select 'Built-in EFI Shell'.
-
At EFI prompt enter the following:
Shell> FS0:\nFS0:> cd EFI\nFS0:\\EFI\\> AfuEfix64.efi 0ACRH608_REL.BIN /p /n /b\n
Press 'E'. When the update has completed type 'reset -w' to reboot.
-
Watch your screen and press 'ESC' key to go into BIOS setup mode. Verify BIOS version matches expectation.
-
Click the right arrow key six times to move from 'Main' set up menu to 'Boot' setup. Select NVMe as the primary boot source.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#40-compiling-a-custom-yocto-soc-image","title":"4.0 Compiling a Custom Yocto SoC Image","text":"Custom Yocto image compilation steps are shown in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#50-verifying-the-icx-d-soc-opae-sdk","title":"5.0 Verifying the ICX-D SoC OPAE SDK","text":"The reference SoC Attach FIM and unaltered FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Full supported functionality of the HEMs is documented in the host_exerciser opae.io GitHub page. SoC Attach supports HEMs run both with and without an AFU image programmed into the board's one supported PR region. This image is available on the offial SoC Attach GitHub Page, and is programmed using fpgasupdate as shown in section 3.2. A few select examples run from the SoC and their expected results will be shown below.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#51-checking-telemetry-with-fpgainfo","title":"5.1 Checking Telemetry with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files.
The command argument is one of the following: errors, power, temp, port, fme, bmc, phy, mac, and security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
An example output for fpgainfo fme is shown below. Your IDs may not match what is shown here:
$ fpgainfo fme\nIntel IPU Platform F2000X-PL\nBoard Management Controller NIOS FW version: 1.1.9\nBoard Management Controller Build version: 1.1.9\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:15:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x17D4\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50103023DFBFC8E\nBitstream Version : 5.0.1\nPr Interface Id : bf74e494-ad12-5509-98ab-4105d27979f3\nBoot Page : user1\nUser1 Image Info : 98ab4105d27979f3bf74e494ad125509\nUser2 Image Info : None\nFactory Image Info : None\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#52-host-exercisers","title":"5.2 Host Exercisers","text":"Of these five tests listed below, the first three do not require an AFU be loaded into the board's PR region. They exercise data paths that pass exclusively through the FIM. The latter three tests exercise data through the AFU data path, and require SoC Attach release AFU Image to be configured using fpgasupdate.
- Run HE-MEM with 2 cachelines per request in
mem mode, exercising the FPGA's connection to DDR. No AFU required.Note: If you see the error message Allocate SRC Buffer, Test mem(1): FAIL, then you may need to manually allocate 2MiB Hugepages using the following: echo 20 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF0 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.1 <username>:<username>\n# Check for HE-MEM Accelerator GUID 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Run desired HE-MEM test(s)\n$ host_exerciser --cls cl_1 --mode lpbk mem\nstarting test run, count of 1\nAPI version: 2\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 9948\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.096 GB/s\n Test mem(1): PASS\n
- Generate traffic with HE-HSSI. No AFU required.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF2 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.2 <username>:<username>\n# Check for HE-HSSI Accelerator GUID 823c334c-98bf-11ea-bb37-0242ac130002\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Show number of configured ports\n$ fpgainfo phy | grep Port\n# Generate traffic for specific port number\n$ hssi --pci-address <PCIe Bus>:00.2 hssi_10g --num-packets 100 --port <port number>\n10G loopback test\nTx/Rx port: 1\nTx port: 1\nRx port: 1\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n...\n
This command will generate a log file in the directory is was run from. Check TxPackets and RxPackets for any loss. Two more supported HSSI commands are hssistats, which provides MAC statistics, and hssimac, which provides maximum TX and RX frame size.
- Test memory traffic generation using MEM-TG. This will exercise and test available memory channels with a configurable memory pattern. Does not require an AFU image.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF2 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.3 <username>:<username>\n# Check for MEM-TG Accelerator GUID 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Example MEM-TG Test\n$ mem_tg --loops 500 -w 1000 -r 0 -b 0x1 --stride 0x1 -m 0 tg_test\n
- HE-LPBK is designed to demo how AFUs move data between host memory and the FPGA. Will check latency, MMIO latency, MMIO bandwidth, and PCIe bandwidth. LPBK workload requires the SoC Attach AFU be loaded into the board's PR slot.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF1 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.1 <username>:<username>\n# Check for LPBK GUID 56e203e9-864f-49a7-b94b-12284c31e02b\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Example loopback test\n$ host_exerciser --mode lpbk lpbk\n
- Exercise He-HSSI subsystem from the AFU. This test will generate and receieve packets from any of the 8 available ports. This HSSI workload requires the SoC Attach AFU be loaded into the board's PR slot.
# Create VF device\n$ pci_device <PCIe Bus>00.0 vf 3\n# Bind VF6 to vfio-pci\n$ opaei.io init -d <PCIe Bus>:00.2 <username>:<username>\n# Check for HSSI GUID 823c334c-98bf-11ea-bb37-0242ac130002\n$ fpgainfo port -B <PCIe Bus>:00.0\n# Geneate traffic for specific port\n$ hssi --pci-address <bus_num>:00.6 hssi_10g --num-packets 100 --port <port_num>\n
This command will generate a log file in the directory is was run from. Check TxPackets and RxPackets for any loss. Two more supported HSSI commands are hssistats, which provides MAC statistics, and hssimac, which provides maximum TX and RX frame size.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#53-additional-opae-sdk-utilities","title":"5.3 Additional OPAE SDK Utilities","text":"This section will discuss OPAE SDK utilities that were not covered by previous sections. These commands are all available on the ICX-D SoC Yocto image by default. A full description and syntax breakdown for each command is located on the official OPAE SDK github.io repo.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#table-5-opae-sdk-utilities","title":"Table 5: OPAE SDK Utilities","text":"Utility Description fpgasupdate 3.2 Updating FIM, BMC and AFU with fpgasupdate rsu Section 3.3 Loading images with rsu host_exerciser Section 5.2 Host Exercisers hssi Section 5.2 Host Exercisers hssistats Section 5.2 Host Exercisers hssimac Section 5.2 Host Exercisers mem_tg Section 5.2 Host Exercisers usrclk userclk tool is used to set high and low clock frequency to an AFU. mmlink Remote signaltap is software tool used for debug RTL (AFU), effectively a signal trace capability that Quartus places into a green bitstream. Remote Signal Tap provides access the RST part of the Port MMIO space, and then runs the remote protocol on top. opaevfio The opaevfio command enables the binding/unbinding of a PCIe device to/from the vfio-pci device driver. See https://kernel.org/doc/Documentation/vfio.txt for a description of vfio-pci. opae.io An interactive Python environment packaged on top of libopaevfio.so, which provides user space access to PCIe devices via the vfio-pci driver. The main feature of opae.io is its built-in Python command interpreter, along with some Python bindings that provide a means to access Configuration and Status Registers (CSRs) that reside on the PCIe device. bitstreaminfo Prints bitstream metadata. fpgaconf Lower level programming utility that is called automatically by fpgasupdate. fpgaconf writes accelerator configuration bitstreams (also referred to as \"green bitstreams\") to an FPGA device recognized by OPAE. In the process, it also checks the green bitstream file for compatibility with the targeted FPGA and its current infrastructure bitstream (the \"blue bistream\"). fpgad Periodically monitors/reports the error status reflected in the device driver's error status sysfs files. Establishes the channel by which events are communicated to the OPAE application. Programs a NULL bitstream in response to AP6 event. fpgad is required to be running before API calls fpgaRegisterEvent and fpgaUnregisterEvent will succeed."},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#60-setting-up-the-host","title":"6.0 Setting up the Host","text":"The steps to set up the host and build and install the OPAE SDK are shown in the Software Installation Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs.
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#61-verifying-the-soc-attach-solution-on-the-host","title":"6.1 Verifying the SoC Attach Solution on the Host","text":"The SoC Attach workload supports testing MMIO HW and Latency and PCIe BW and latency out of box. Execution of the host_exerciser binary on the host requires the OPAE SDK to be installed as shown in section 6.1 Installing the OPAE SDK On the Host. You will also need to have a proper SoC Attach FIM configured on your board as shown in section 3.2 Updating FIM, BMC and AFU with fpgasupdate.
-
Initialize PF attached to HSSI LPBK GUID with vfio-pci driver.
$ sudo opae.io init -d 0000:b1:00.0 <username>:<username>\n$ sudo opae.io init -d 0000:b1:00.1 <username>:<username>\n
-
Run host_exerciser loopback tests (only lpbk is supported). There are more methods of operation than are shown below - read the HE help message for more information.
# Example lpbk tests.\n$ sudo host_exerciser lpbk\n$ sudo host_exerciser --mode lpbk lpbk\n$ sudo host_exerciser --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --cls cl_4 lpbk\n# Number of cachelines per request 4.\n$ sudo host_exerciser --mode lpbk --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode lpbk --cls cl_4 lpbk\n# vNumber of cachelines per request 4.\n$ sudo host_exerciser --mode read --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode read --cls cl_4 lpbk\n# Number of cachelines per request 4.\n$ sudo host_exerciser --mode write --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode write --cls cl_4 lpbk\n# Number of cachelines per request 4.\n$ sudo host_exerciser --mode trput --cls cl_4 lpbk\n$ sudo host_exerciser --perf true --mode trput --cls cl_4 lpbk\n# Enable interleave requests in throughput mode\n$ sudo host_exerciser --mode trput --interleave 2 lpbk\n$ sudo host_exerciser --perf true --mode trput --interleave 2 lpbk\n#with delay option.\n$ sudo host_exerciser --mode read --delay true lpbk\n$ sudo host_exerciser --mode write --delay true lpbk\n# Test all modes of operation\n$ host_exerciser --testall=true lpbk\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#62-fpga-device-access-permissions","title":"6.2 FPGA Device Access Permissions","text":"Access to FPGA accelerators and devices is controlled using file access permissions on the Intel\u00ae FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
sudo chmod a+rw /dev/dfl-port.0\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#63-memlock-limit","title":"6.3 Memlock limit","text":"Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"hw/f2000x/user_guides/ug_qs_ofs_f2000x/ug_qs_ofs_f2000x/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/","title":"UVM Simulation User Guide: Open FPGA Stack for Intel Agilex 7 SoC Attach FPGAs","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#1-overview","title":"1 Overview","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#11-about-this-document","title":"1.1 About this Document","text":"This document serves as a set-up and user guide for the UVM simulation tool using OFS. After reviewing the document, you will be able to:
- Set-up the UVM verification tool suite
- Run pre-existing UVM unit tests and also create new UVM tests for your design
Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#2-introduction-to-uvm","title":"2 Introduction to UVM","text":"The Functional Verification Environment for OFS is UVM (Universal Verification Methodology) compliant and provides configurable setup for verifying various FIM features in simulation.
The purpose of this document is to demonstrate a full chip level and unit level UVM based verification environment for the current base shell FIM architecture as well as providing examples to extend the setup for different OFS variants by reusing the existing architecture and infrastructure for UVM based verification.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#3-universal-testbench-architecture","title":"3 Universal Testbench Architecture","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#31-overview","title":"3.1 Overview","text":"The main idea behind UVM is to develop modular, reusable, and a scalable testbench structure by providing an API framework that can be deployed across multiple projects.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#32-core-verification-concepts","title":"3.2 Core Verification Concepts","text":"The following is the list of verification components that will be used to design a UVM testbench architecture:
\u2022 Sequencer \u2022 Driver \u2022 Monitor \u2022 Scoreboard \u2022 Virtual Sequencer \u2022 Interface \u2022 Verification Environment \u2022 TOP Testbench
Figure 1 provides the general UVM Testbench and the verification components involved in the top-level architecture.
Figure 1 Typical UVM Testbench
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4-ofs-testbench-architecture","title":"4 OFS Testbench Architecture","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#41-overview","title":"4.1 Overview","text":"OFS (Open FPGA Stack) provides a UVM (Universal Verification Methodology) environment for the FIM with a modular, reusable, and scalable testbench structure via an API framework.
The framework consists of a FIM Testbench which is UVM compliant and integrates third party VIPs from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this Testbench. UVM RAL(Register Abstraction Level) is used for CSR (Command and Status Registers) verification.
The qualified verification IPs will help to detect incorrect protocol behavior, help to focus on FIM features and accelerate the verification process.
Verification components include:
\u2022 FIM monitor to detect correct design behavior\n\u2022 FIM assertions for signal level integrity testing\n\u2022 Arm AMBA Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity\n\u2022 FIM coverage to collect functional data\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#42-base-fim-dut","title":"4.2 Base FIM DUT","text":"The hardware architecture of an Intel Agilex 7 FIM is based on the OFS hardware architecture. The following is the list of features and subsystems supported in the base shell.
\u2022 PCIe Subsystem\n\u2022 HSSI Subsystem\n\u2022 Memory Subsystem\n\u2022 HPS Subsystem\n\u2022 FME\n\u2022 AFU with PR support\n\u2022 QSFP Controllers\n\u2022 PMCI Controller, MCTP\n
Figure 2 DUT Base Shell Diagram
Figure 2 shows the high level architecture of an Intel Agilex 7 Base Shell. It has a Gen4x16, 100G Ethernet Datapath in a 2x4x25G configuration. The Intel Agilex 7 Base Shell is a shell that will enable a user to build other shell variants for a custom configuration. For the f2000x board there is one shell variant
base_x16
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43-full-chip-level-verification-archiecture-for-fim","title":"4.3 Full Chip Level Verification Archiecture for FIM","text":"Figure 3 shows a graphical representation a full chip testbench that includes major RTL blocks depicted in a OFS Intel Agilex 7 based UVM environment
Figure 3 OFS FIM Testbench
The major connection is the interface between the Xeon CPU and the FPGA where the PCIe Verification IP is connected to PCIe Subsystem. Therefore, as a full chip simulation environment, PCIe host VIP is the sole VIP/BFM used. PCIe host VIP connects to PCIe device which resides in FPGA in serial mode.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#431-testbench-components","title":"4.3.1 Testbench components","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4311-tb-top","title":"4.3.1.1 TB TOP","text":"TOP is the top level testbench and consists of a FIM DUT instance and top-level UVM Verification Environment instance. It binds RTL inputs with the verification environmnet interfaces to drive stimulus. It also has clock generation and reset driving logic.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4312-fim-verification-environment","title":"4.3.1.2 FIM Verification Environment","text":"This is the top most verification environment class and consists of the protocol specific PCI Express and AXI UVM environment VIP instances, Monitors, Scoreboards, Assertions, Functional coverage Modules and other verification components. It instantiates Virtual sequencers to control stimuli for FIM traffic from different sequencers of the VIPs.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4313-synopsys-vips","title":"4.3.1.3 Synopsys VIPs","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43131-pci-vip-as-host","title":"4.3.1.3.1 PCI VIP as Host","text":"This is Synopsys Licensed PCI Express Gen4 VIP and acts as Root Port. The PCI Express link is connected to the DUT using TX-RX lanes. Agent is an active component and includes a sequencer to generate TLPs, Driver to drive it on a PCI Express link and Monitor to check the protocol correctness.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43132-axi-streaming-vip-monitors","title":"4.3.1.3.2 AXI-Streaming VIP Monitors","text":"This is Synopsys Licensed AXI streaming interface Verification IP used as a Passive Agent to monitor AXI-ST links at various points. Please refer to Figure 3 to see all the AXI-ST monitors connected to different modules.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43133-axi4-memory-mapped-vip-monitors","title":"4.3.1.3.3 AXI4-Memory Mapped VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 memory mapped interface Verification IP used in passive mode to observe memory requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and performing data integrity checks.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43134-axi4-lite-vip-monitors","title":"4.3.1.3.4 AXI4-Lite VIP Monitors","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used in passive mode to observe MMIO requests. For each master-slave pair, the verification environment has a VIP instantiated in passive mode for monitoring protocol violations and perfoming data integrity checks. Please refer to Figure 3 to see all the Arm\u00ae AMBA\u00ae 4 AXI4-Lite monitors connected to different modules.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#43135-axi4-lite-vip-as-pmci-master","title":"4.3.1.3.5 AXI4-Lite VIP as PMCI Master","text":"This is Synopsys Licensed Arm\u00ae AMBA\u00ae 4 AXI4 Lite interface Verification IP used to drive and observe MMIO requests as PMCI master. For each master-slave pair, the verification environment has a VIP instantiated in active mode and includes a sequencer to generate MMIO requests, driver to drive these requests on AXI-4 Lite interface to BPF and a monitor for observing protocol violations and data integrity checks.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4314-axi4-s-scoreboard","title":"4.3.1.4 AXI4-S Scoreboard","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-S scoreboard checks data integrity of source and sink components. It has input transactions from VIP monitors and a TLP to AXI converter for PCIe TLP packets. It makes sure the valid TLPs or AXI transactions are not missed while traversing from Host to AFU and reverse. The scoreboard will be disabled for error testing especially for invalid TLP requests and UR responses.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4315-virtual-sequencer","title":"4.3.1.5 Virtual Sequencer","text":"The virtual sequencer is the top-level sequencer which controls Enumeration, MMIO Requests, downstream and Upstream traffic as well as HSSI and Memory transactions. It makes sure that the transactions are ordered correctly as per the design specification while running a UVM test simulation.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4316-fim-monitor","title":"4.3.1.6 FIM Monitor","text":"The FIM monitor is used for checking the correctness of a specific FIM feature. As an example, a user can add interrupt related checks, error generation and clearing checks, Protocol checker impacts etc. in this component.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4317-fim-assertions","title":"4.3.1.7 FIM Assertions","text":"The assertion component is used for checking the signal level integrity of a specific FIM feature or behavior. As an example, we can add interrupt signal triggering and clear assertions, Error detection to error register bit assertions, protocol checker and clearing logic, FLR behavior assertion etc. in this top-level module. There are alos assertions to make sure the inputs are driven correctly to catch errors in a users simulation.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#4318-fim-functional-coverage","title":"4.3.1.8 FIM Functional Coverage","text":"The FIM functional coverage component will have the functional coverage of each feature like interrupts, CSR's, MMIO requests, Byte align transactions, error reporting etc. to demonstrate a variety of FIM features. This will help us to find holes in the design as well as wide verification coverage to make sure thorough testing of a design. It will also provide a user what the FIM is capable of and how much code coverage they are testing.
This would be an independent plug-n-play component and can be reused with any user developed UVM FIM TB.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#5-uvm-verification-set-up","title":"5 UVM Verification set-up","text":"To run the tutorial steps in this guide requires the following development environment:
Item Version Intel Quartus Prime Pro Intel Quartus Prime Pro 23.4 Simulator Synopsys VCS S-2021.09-SP1 or newer for UVM simulation of top level FIM Simulator (Questasim) Questasim 2023.4 or newer for UVM simulation of top level FIM"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#51-uvm-prerequisite","title":"5.1 UVM Prerequisite","text":"Retrieve OFS repositories.
The OFS FIM source code is included in the OTCShare GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files. The following is a short description of each repository, followed by the git commands for cloning. The instructions section uses the HTTPS git method for cloning repositories. Cloning the repo using the HTTPS method requires a personal access token. Please see this blog post for information about obtaining a personal access token Token authentication requirements for Git operations.
Navigate to location for storage of OFS source, create the top-level source directory and clone OFS repositories.
$ mkdir ofs-2024.1-1\n$ cd ofs-2024.1-1\n$ export OFS_BUILD_ROOT=$PWD\n$ git clone --branch --recurse-submodules https://github.com/ofs-f2000x-pl.git\n\nCloning into 'ofs-f2000x-pl'...'\nUsername for 'https://github.com': <<Enter your git hub username>>\nPassword for 'https://<<Your username>>': <<Enter your personal access token>>\nremote: Enumerating objects: ....\n...\n...\nResolving deltas ..., done.\n\n$ cd ofs-f2000x-pl\n$ git checkout tags/ofs-2024.1-1\n
Verify that the correct tag/branch have been checked out
$ git describe --tags\n\n$ ofs-2024.1-1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#52-license-requirements","title":"5.2 License Requirements","text":"The FIM Testbench is UVM compliant and integrates third party VIP's from Synopsys for PCI Express, Arm\u00ae AMBA\u00ae 4 AXI4-Streaming interface and Arm\u00ae AMBA\u00ae 4 AXI4Arm\u00ae AMBA\u00ae 4 AXI4-Memory Mapped interface for comprehensive verification. The user will need to acquire licenses for these VIPs to use this TB. UVM RAL (Register Abstraction Layer) is used for CSR Verification.
The Qualified Verification IPs will help to detect incorrect protocol behavior easily, help to focus on FIM features and accelerate the verification process.
\u2022 VCS & DVE\n\u2022 SNPS-Assertions\n\u2022 Verdi\n\u2022 VerdiCoverage\n\u2022 VerdiSimDB\n\u2022 VerdiTransactionDebugUltra\n\u2022 VIP-AMBA-AXI-SVT\n\u2022 VIP-AMBA-STREAM-SVT\n\u2022 VIP-PCIE-SVT\n\u2022 VIP-PCIE-TS-SVT\n\u2022 VIP-PCIE-G3-OPT-SVT\n\u2022 VIP-Ethernet-SVT\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#53-software-tools-requirements","title":"5.3 Software Tools Requirements","text":"The following tools are required for successful UVM set-up
- Python 3.6.8
- Synopsys PCIE and AMBA AXI UVM VIP Q-2020.03A License
- VCS R-2020.12-SP2 License Note: Makefile can be modified to use DVE instead of Verdi
- VCS R-2020.12-SP2 License
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#54-creating-a-software-tools-script","title":"5.4 Creating a Software Tools Script","text":"The UVM tool set-up is best done by creating a simple set-up script so all applicable tools are sourced before running the tests.
The following environment variables can be pasted into a script and used prior to running the UVM verification environment
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#license-files","title":"License Files","text":"export LM_LICENSE_FILE=\nexport SNPSLMD_LICENSE_FILE=\n
The license environment variables LM_LICENSE_FILE and SNPSLMD_LICENSE_FILE can point to a server license on your system.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#general-environment-variables","title":"General Environment Variables","text":"export IOFS_BUILD_ROOT=$PWD\nexport OFS_ROOTDIR=<user_path>/ofs-f2000x-pl\nexport WORKDIR=$OFS_ROOTDIR\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#quartus-tools","title":"Quartus Tools","text":"export QUARTUS_HOME=<user_path>/intelFPGA_pro/23.4/quartus\nexport QUARTUS_ROOTDIR=$QUARTUS_HOME\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$PATH\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-verification-tools","title":"Synopsys Verification Tools","text":"export DESIGNWARE_HOME=<user_path>/synopsys/vip_common/vip_Q-2020.03A\nexport PATH=$DESIGNWARE_HOME/bin:$PATH\nexport UVM_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel/etc/uvm\nexport VCS_HOME=<user_path>/synopsys/vcsmx/S-2021.09-SP1/linux64/rhel\nexport PATH=$VCS_HOME/bin:$PATH\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-verification-tools","title":"QuestaSIM Verification Tools","text":"export MTI_HOME=<user_path>/mentor/questasim/2023.4/linux64\nexport PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\nexport QUESTA_HOME=$MTI_HOME\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#6-running-a-uvm-simulation-test-and-analysing-results","title":"6 Running a UVM Simulation Test and Analysing Results","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#61-simulation","title":"6.1 Simulation","text":"The default simulator used in the simulation script is Synopsys VCS-MX. Users can refer to the options and adopt the options for other simulators. The script is a makefile that calls vlogan, vcs and simv for compilation, elaboration and simulation, respectively.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#62-file-structure","title":"6.2 File Structure","text":"After cloning the repo, the verification and ofs-common directories contain all UVM verification related files. The directory structure is shown in Figure 4 below.
Figure 4 UVM Verification Directory File Structure
ofs-f2000x-pl/verification/testbench has a testbench, uvm env, virtual sequencer, RAL etc.
ofs-f2000x-pl/tests contains all uvm tests and sequences.
Users can run the simulation under \"ofs-f2000x-pl/verification/scripts\" directory and the simulation result is outputted to a \"sim\" directory for Synopsys VCS or sim_msim for Questasim.
The simulation result folder is named after the test name with increasing suffix number. If user runs the same test multiple times, the suffix is incremented by 1 each time.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#63-uvm-test-suite","title":"6.3 UVM Test Suite","text":"The UVM environment contains a variety of tests that have been developed to test out the FIM portion of OFS.
The table below has four columns which describe the \"Test Name\", \"DUT Scope\", \"Test Scenario\" and the \"Checking Criteria\".
Tests are located at ofs-f2000x-pl/ofs-common/verification/fpga_family/agilex/tests
Test Name DUT Scope Test Scenario Checking Criteria afu_mmio_flr_pf0_test PF0 FLR Reset Apply FLR Reset for PF0 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF0 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf0_test PF0_VF0_FLR Reset Apply FLR Reset for PF0_VF0 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf1_test PF0_VF1_FLR Reset Apply FLR Reset for PF0_VF1 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf0_vf2_test PF0_VF2_FLR Reset Apply FLR Reset for PF0_VF2 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf1_vf0_test PF1_VF0_FLR Reset Apply FLR Reset for PF1_VF0 and deassert Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf2_test PF2 FLR Reset Apply FLR Reset for PF2 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF2 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf3_test PF3 FLR Reset Apply FLR Reset for PF3 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF3 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_mmio_flr_pf4_test PF4 FLR Reset Apply FLR Reset for PF4 and deassert. Initiate MMIO transactions for all PFs. Make sure all completions are sent/received and no pending transactions are seen. Apply FLR Reset for PF4 and deassert. Initiate mmio access and ensure all PFs CSR access are working fine Initiate mmio access before and after FLR Reset and ensure all PF/VFs CSR access are working fine afu_stress_5bit_tag_test AFU-StressAFU-Stress To check the AFU Stress by sending traffic with 5bit tag from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_8bit_tag_test AFU-Stress To check the AFU Stress by sending traffic with 5bit tag from all PF/VF. i.e Send traffic on HE-LPK/HE-MEM and initiating MMIO access to other PF/VFs Data checking afu_stress_test AFU-Stress 1. Initiate transactions to all the supported PF/VF from PCIE VIP and ensure that traffic is sent to all blocks of the AFU. 2. Ensure that CE/HE-LB/HE-MEM/HSSI/BPF/FME are seeing traffic. 3. Ensure that HE-LB/HE-MEM/CE sends DMWR/DMRD requests to PCIE VIP. 4. Ensure the Mux/DeMux blocks is able to handle the traffic based on the PF's/VF's and proper muxing/demuxing happens. Data checking bar_32b_test PCIe MMIO Path Set the BAR values to 32bit and test mmio access BAR address, Register Base Offset bar_64b_test PCIe MMIO Path Set the BAR values to 64bit and test mmio access BAR address, Register Base Offset dfh_walking_test DFH DFH walking offset checking, eol checking -> tb emif_csr_test EMIF CSR access data checking fme_csr_test FME CSR CSR accesses data checking fme_err_intr_test Interrupt FME Interrupt request using FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_intr_test Interrupt FME interrupt request using RAS ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read fme_multi_err_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_hssi_csr_test HE-HSSI CSR accesses data checking he_hssi_rx_lpbk_25G_10G_test HE-HSSI Sending back to back ethernet data traffic with 25G speed on HSSI RX Port0-7 lanes using Ethernet VIPs Enable the loopback mode in HE-HSSI and compare the pkts recived on HSSI TX Port(DATA CHECKING) he_hssi_tx_lpbk_P0_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port0 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P1_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port1 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P2_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port2 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P3_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port3 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P4_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port4 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P5_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port5 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P6_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port6 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_hssi_tx_lpbk_P7_test HE-HSSI Send back to back traffic with 25G speed on HSSI TX Port7 lanes using Traffic generator registers Check the CRC errors on loopback packets on Traffic Monitor Registers he_lpbk_cont_test HE-LPBK Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking he_lpbk_csr_test HE-LPBK CSR accesses data checking he_lpbk_flr_rst_test HE-LPBK FLR RST is generated in HE_LPBK access data checking, counter checking he_lpbk_long_rst_test HE-LPBK Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_lpbk_long_test HE-LPBK Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_lpbk_multi_user_intr_test HE-LPBK generate user HE_LB interrupt interrupt checking he_lpbk_rd_cont_test HE-LPBK Read only mode/ Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_rd_test HE-LPBK Read only mode. Randomize num_lines, addresses, req_len counter checking he_lpbk_reqlen16_test HE-LPBK To check the behavior of HE_LPK block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_lpbk_reqlen1_test HE-LPBK Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_lpbk_reqlen2_test HE-LPBK Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_lpbk_reqlen4_test HE-LPBK Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_lpbk_reqlen8_test HE-LPBK Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_lpbk_rst_in_middle_test PCIe MMIO Path Set HE_LPK in all the modes randomly and iterate the transactions in loop. At the end of every loop assert the Soft reset in the middle of the transactions Register Base Offset he_lpbk_test HE-LPBK Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_lpbk_thruput_contmode_test HE-LPBK Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking he_lpbk_thruput_test HE-LPBK Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_lpbk_wr_cont_test HE-LPBK Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking he_lpbk_wr_test HE-LPBK Write only mode. Randomize num_lines, addresses, req_len counter checking he_mem_cont_test HE-MEM Continuous mode/LPBK mode, random num_lines, addresses, req_len data checking, counter checking he_mem_csr_test HE-MEM CSR accesses data checking he_mem_lpbk_long_rst_test HE-MEM Multiple iterations of he_lpb_seq with soft reset HE-LB in middle data checking, counter checking he_mem_lpbk_long_test HE-MEM Multiple iterations of he_lpb_seq with STOP HE-LB in middle data checking, counter checking he_mem_lpbk_reqlen16_test HE-MEM To check the behavior of HE_LPK block when req_length 16 and num_lines set to 1024 Cache lines in Loopback mode data checking, counter checking he_mem_lpbk_reqlen1_test HE-MEM Loopback mode. 128 CLs, req_len = 1CL, random addresses data checking, counter checking he_mem_lpbk_reqlen2_test HE-MEM Loopback mode. 128 CLs, req_len = 2CL, random addresses data checking, counter checking he_mem_lpbk_reqlen4_test HE-MEM Loopback mode. 128 CLs, req_len = 4CL, random addresses data checking, counter checking he_mem_lpbk_reqlen8_test HE-MEM Loopback mode. 128 CLs, req_len = 8CL, random addresses data checking, counter checking he_mem_lpbk_test HE-MEM Loopback mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_multi_user_intr_test Interrupt Back to back multiple User interrupt request from HE MEM Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read for multiple back to back request he_mem_rd_cont_test HE-MEM Read only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_rd_test HE-MEM Read only mode. Randomize num_lines, addresses, req_len counter checking he_mem_thruput_contmode_directed_test HE-MEM Set HE_MEM in Thrput Continuous mode and test data checking, counter checking he_mem_thruput_contmode_test HE-MEM Continuous mode, Read/Write mode. 50/50. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_thruput_test HE-MEM Read/Write mode. 50/50. Randomize num_lines, addresses, req_len counter checking he_mem_user_intr_test Interrupt FME interrupt request using RAS ERROR and FME ERROR assertion Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests generated from FME and RAS ERROR bits he_mem_wr_cont_test HE-MEM Write only mode/Continuous mode. Randomize num_lines, addresses, req_len data checking, counter checking he_mem_wr_test HE-MEM Write only mode. Randomize num_lines, addresses, req_len counter checkingt he_random_test All HEs Enable all HEs and randomize modes data checking if in lpbk mode, counter checking helb_rd_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Read only mode Measure the performance helb_rd_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Read only mode Measure the performance helb_rd_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Read only mode Measure the performance helb_thruput_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Thruput mode Measure the performance helb_thruput_4cl_5bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_8bit_tag_test Performance Set HE_LPK in thruput mode and send traffic with req len 4 and num_lines set to 1024. Measure the Read/Write performance Measure the performance helb_thruput_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Thruput mode Measure the performance helb_wr_1cl_test Performance HE-LB; ReqLen = 1CL; 1024 CLs; Write only mode Measure the performance helb_wr_2cl_test Performance HE-LB; ReqLen = 2CL; 1024 CLs; Write only mode Measure the performance helb_wr_4cl_test Performance HE-LB; ReqLen = 4CL; 1024 CLs; Write only mode Measure the performance hssi_ss_test HSSI CSR accesses data checking malformedtlp_pcie_rst_test Protocol Checker generate mllformed TLP protocol error and initiate pcie reset to clear the error Check for error malformedtlp_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MaxTagError_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transactions are completing. mem_tg_csr_test MEM-TG CSR access data checking mem_tg_traffic_gen_test MEM-TG This test checks the MEM_TG traffic generator flow data checking mini_smoke_test All HEs shorter simpler version of random test for turn-in sanity check data checking if in lpbk mode, counter checking mix_intr_test Interrupt Mix interrupt testcase to send multiple FME and User interrupt request simultaneously Test checks for interrupt assertion, deassertion, mask feature, PBA bits and MSIX host memory data integrity through backdoor memory read plus verifying interrupt requests through different sources - FME and HE-MEM modules mmio_pcie_mrrs_128B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_128B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_128B_test PCIe - Max Payload/Max Read Req Size Random length mmio Write Checking valid possible combination of MPS & MRRS mmio_pcie_mrrs_256B_mps_256B_test PCIe - Max Payload/Max Read Req Size Random length mmio Read Checking valid possible combination of MPS & MRRS mmio_stress_nonblocking_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux with non-blocking MMIO reads data checking mmio_stress_test PF/VF Mux/Demux Stressing MMIO on PF/VF Mux/Demux data checking mmio_test PCIe MMIO Path MMIO targeting PF0(ST2MM, FME, PMCI, QSFP, HSSI SS), PF1, PF2,PF3, PF4, PF1.VF1, PF1.VF2 data checking mmio_unimp_test PCIe MMIO Path MMIO acccess to unimplemented addresses data checking MMIODataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. MMIOTimedout_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. pcie_csr_test PCIESS Earlier msix registers were in fme block but now it has moved from fme to pciess. Hence coded a seperate test for msix data checking pcie_pmci_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pcie_pmci_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM (Vendor Defined Message) single packet received from PCIe HIA subsystem to PMCI controller over APF and BPF via AXI4-lite memory write request pmci_csr_test PMCI CSR CSR accesses data checking pmci_fme_csr_test PMCI FME CSR CSR accesses data checking pmci_pcie_mctp_multi_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM multiple packets received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pcie_mctp_vdm_test MCTP Vendor specific messaging capability MCTP PCIe VDM single packet received from PMCI controller over APF and BPF to PCIe HIA subsystem pmci_pciess_csr_test PMCI PCIESS CSR CSR accesses data checking pmci_qsfp_csr_test PMCI QSFP CSR CSR accesses data checking port_gasket_csr_test PORT GASKET CSR accesses data checking qsfp_csr_test QSFP CSR accesses data checking TxMWrDataPayloadOverrun_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. TxMWrInsufficientData_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. uart_intr_test UART Checking Generates UART interrupt Check interrupt UnexpMMIORspErr_test Protocol Checker Checks for different PCIe Protocol error generation (upstram/downstream) and check the clear mechanism of AFU for these detected protocol errors. 1. Apply the error. 2. Wait 5us 1us 3. Set the port reset bit true and then false again (within 1us) (You will not be able to perform a read-modify-write because F's will be returned on reads. Write a 0x5 to set and a 0x4 to clear). 4. Wait 124us 135us (or 7.5us 28us if MMIO_TIMEOUT_CYCLES is 512.) 5. Read bit 31 of the AFU_INTF_ERROR, BlockingTraffic. If it is set it means that we did not wait long enough in step 4. 6. Read the AFU_INTF_ERROR register, Be sure only the expected error(s) are set. 7. Read the AFU_INTF_FIRST_ERROR CSR. Be sure only one error bit is set and it is the expected one. 8. Clear the errors in the AFU_INTF_ERROR CSR by writing one to the bits that are set. 9. Read the AFU_INTF_FIRST_ERROR CSR. Be sure all bits are cleared 1.Check AFU_INTF_ERROR and AFU_INTF_FIRST_ERROR register is getting set with correct error vector. 2.After clearing the error register ,check if normal transcation are completing. vdm_err_vid_test Vendor ID check generate a packet with undefined Vendor-ID from Host to PMCI_SS ID check The next section describes how to compile and build the UVM environment prior to running each UVM test and analyinsg the results in the log files
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#64-ip-compile","title":"6.4 IP Compile","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs","title":"Synopsys VCS","text":"To compile all IPs for the Synopsys VCS simulater:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk cmplib_adp\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd","title":"Questasim (TBD)","text":"To compile all IPs for the Questasim simulater:
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk cmplib_adp\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#65-rtl-test-bench-compile","title":"6.5 RTL & Test Bench Compile","text":"The RTL file list for compilation is located here: verification/scripts/rtl_comb.f
The TB file list for compilation is located here: verification/scripts/ver_list.f
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_1","title":"Synopsys VCS","text":"To compile RTL and Testbench for the Synopsys VCS simulater
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_adp DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_1","title":"Questasim (TBD)","text":"To compile RTL and Testbench for the Questasim simulater
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk build_adp DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#66-ip-and-rtl-test-bench-compile","title":"6.6 IP and RTL & Test Bench Compile","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_2","title":"Synopsys VCS","text":"If the user wants to compile all IPs and RTL Testbench in one command for Synopsys VCS then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk build_all DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_2","title":"Questasim (TBD)","text":"If the user wants to compile all IPs and RTL Testbench in one command for Questasim then follow the procedure below
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk build_all DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_3","title":"Synopsys VCS","text":"To run a simulation for Synopsys VCS:
cd $VERDIR/scripts\n\n gmake -f Makefile_VCS.mk run TESTNAME=ofs_mmio_test DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_3","title":"Questasim (TBD)","text":"To run a simulation for Questasim:
cd $VERDIR/scripts\n\n gmake -f Makefile_MSIM.mk run TESTNAME=ofs_mmio_test DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_4","title":"Synopsys VCS","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Synopsys VCS build and simulation.
gmake -f Makefile_VCS.mk build_adp DUMP=1\n\n gmake -f Makefile_VCS.mk run TESTNAME=<test_case_name> DUMP=1\n
Or
gmake -f Makefile_VCS.mk build_run TESTNAME=<test_case_name> DUMP=1\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_4","title":"Questasim (TBD)","text":"To dump the waveform, \"DUMP=1\" must be added into the command line for Questasim build and simulation.
gmake -f Makefile_MSIM.mk build_adp DUMP=1\n\n gmake -f Makefile_MSIM.mk run TESTNAME=<test_case_name> DUMP=1\n
Or
gmake -f Makefile_MSIM.mk build_run TESTNAME=<test_case_name> DUMP=1\n
There are some optimizations in the Table below for convenience if you want to bypass some commands for both Synopsys VCS and Questasim:
Command (Synopsys VCS) Command (Questasim) Details gmake -f Makefile_VCS.mk build_all DUMP=1 gmake -f Makefile_MSIM.mk build_all DUMP=1 compile IP + compile RTL gmake -f Makefile_VCS.mk build_run TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk build_run TESTNAME= DUMP=1 compile RTL + run test gmake -f Makefile_VCS.mk do_it_all TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk do_it_all TESTNAME= DUMP=1 compile IP, RTL and run test gmake -f Makefile_VCS.mk rundb TESTNAME= DUMP=1 gmake -f Makefile_MSIM.mk rundb TESTNAME= DUMP=1 run test in sim dir + over-writes content"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#67-uvm-regression-test","title":"6.7 UVM Regression Test","text":"cd $VERDIR/scripts\n\npython uvm_regress.py -l -n 8 -p adp -k top_pkg -s vcs -c none\n\nFor Regression in VCS with top/test package, execute the following command \n python uvm_regress.py -l -n 8 -p adp -k top_pkg -s vcs -c none\n\nResults are created in a sim directory ($VERDIR/sim) with individual testcase log dir\n\nFor Regression in MSIM with top/test package, execute the following command \n python uvm_regress.py -l -n 8 -p adp -k top_pkg -s msim -c none\n
Results are created in a sim directory ($VERDIR/sim_msim) with individual testcase log dir
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#68-uvm-waveform-and-transcript-analysis","title":"6.8 UVM Waveform and Transcript Analysis","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#synopsys-vcs_5","title":"Synopsys VCS","text":"Running Synopsys VCS UVM tests will generate a ofs-f2000x-pl/verification/sim directory
\u2022 All build time logs are located at ofs-f2000x-pl/verification/sim\n\n\u2022 Each testcase will have separate directory inside sim ofs-f2000x-pl/verification/sim/<test_case_name>\n
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Synopsys VCS. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 5
Figure 5 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 6
Figure 6 trans.log
The waveform generated is named as \"inter.vpd\". To open the waveform, go to simulation result directory and run
dve -full64 -vpd inter.vpd &\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#questasim-tbd_5","title":"Questasim (TBD)","text":"Running Questasim UVM tests will generate a ofs-f2000x-pl/verification/sim_msim directory
\u2022 All build time logs are at ofs-f2000x-pl/verification/sim_msim\n\n\u2022 Each testcase will have separate directory inside sim ofs-f2000x-pl/verification/sim_msim/<test_case_name>\n
There are two tracker or log files that are available: runsim.log and trans.log.
runsim.log is the simulation log file generated from Questasim. The test sequence prints useful information for debugging purpose, such as the base address for each function or block. For HE-LB and HE-MEM, key information such as SRC_ADDR, DST_ADDR, NUM_LINES, mode, req_len etc is printed out as shown in Figure 7
Figure 7 runsim.log
trans.log is generated from PCIe host VIP. trans.log records all transaction information coming in or going out of the VIP. Users can find traffic direction(DIR), TLP type, Tag, Address or BDF, 3 or 4 dword header of the TLP as shown in Figure 8
Figure 8 trans.log
The waveform generated is named as \"vsim.wlf\". To open the waveform, go to simulation result directory and run
vsim -view vsim.wlf &\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#69-uvm-coverage-analysis","title":"6.9 UVM Coverage Analysis","text":"The following command allows to run a single testcase with coverage enabled
gmake -f Makefile_VCS.mk cmplib_adp && gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1 COV_FUNCTIONAL=1&& gmake -f Makefile_VCS.mk run TESTNAME=<TESTCASE-NAME> DUMP=1 DEBUG=1 COV_FUNCTIONAL=1 &\n
The following command shows how to merge and generate the coverage report
urg -dir <$VERDIR/sim/simv.vdb> <$VERDIR/sim/regression.vdb> -format both -dbname <regression_database_name>\n
This will generate both urgreport directory and .vdb file Multiple regression.vdb from different regressions can be merged with the same command.
e.g \"urg -dir <path1_till_simv.vdb> <path1_till_regression.vdb> <path2_till_regression.vdb> -report <dir> -format both -dbname <dirname>\"\n
The following commands shows how to launch DVE and check the coverage reports
To open DVE of a single regression or testcase, execute:\n\n dve -full64 -cov -covdir simv.vdb regression.vdb &\n\nTo open DVE of a merged regression test, execute:\n\n dve -full64 -cov -covdir <dirname.vdb> &\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#7-csr-verification-using-uvm-ral","title":"7 CSR Verification using UVM RAL","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#71-overview","title":"7.1 Overview","text":"The UVM Register Layer provides a standard base class library that enable users to implement the object-oriented model to access the DUT registers and memories. The UVM Register Layer is also referred to as UVM Register Abstraction Layer (UVM RAL). Design registers can be accessed independently of the physical bus interface. i.e. by calling read/write methods. This is shown in Figure 9 below.
Figure 9 RAL UVM Testbench
The RAL register models for different CSR's mimics the design registers. All RAL files are located here.
ofs-f2000x-pl/verification/testbench/ral\n
The RAL model is generated through the Synopsys RALGEN tool and is used for CSR verification.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#72-ral-integration","title":"7.2 RAL Integration","text":"For UVM RAL model integration to the environment, adapters for each CSR is implemented and integrated into the Testbench Environment. It is used to convert the PCIe bus sequence items into uvm_reg_bus_op and vice versa. The CSR test cases pick up all the registers from the respective CSR blocks and perform a default value, wr/rd check on them.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#73-ral-model-generation","title":"7.3 RAL Model Generation","text":"Steps for RAL model generation
Excel(xls) file containing the registers is required. Make sure there are separate xls files for each CSR and the worksheet name must contain the name \"reg_fields\".
Excel sheet snapshot example below for EMIF_CSR.xls located at /ipss/mem/rtl/adp
\u2022 Navigate to ofs-f2000x-pl/ofs-common/verification/common/scripts/ral\n\u2022 Copy the excel file (xls) to the above area\n\u2022 In the bash terminal run mk_ral.sh <Excel sheet name without extension > <output *.sv file name without ral_ prepended >\n\u2022 The above steps generate two ral *.sv files. File with _cov suffix is a coverage enabled ral model. \n\u2022 Copy *.sv files to ofs-f2000x-pl/verification/testbench/ral\n
\u2022 As an example to generate ral_ac_ce.sv from AC_CE_CSR.xls file the command is\n\n mk_ral.sh AC_CE_CSR ac_ce\n
This generates two ral models (ral_ac_ce.sv and ral_ac_ce_cov.sv)
To add new registers
\u2022 To create new registers, copy existing ones and modify the cells in the xls. Make sure the last line is also a copied blank line\n\u2022 Follow all the steps of RAL model generation\n
To Generate a RAL model when a new xls sheet is created for a new component
\u2022 Copy the relevant xls sheet to ofs-f2000x-pl/ofs-common/verification/common/scripts/ral\n\u2022 Follow all the steps of RAL model generation\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#74-top-level-verification-architecture-for-csr-testing","title":"7.4 Top Level Verification Architecture for CSR testing","text":""},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#741-testbench-components","title":"7.4.1 Testbench components","text":"The testbench components for RAL are defined below
\u2022 ral_reg_iofs_* (uvm_reg) generated by the steps as mentioned in section 5.3\n
The uvm register class is written by extending the uvm_reg. A register represents a set of fields that are accessible as a single entity Each register contains any number of fields, which mirror the values of the corresponding elements in hardware
\u2022 ral_block_iofs_* (uvm_block) generated in the same register file\n
A register model is an instance of a register block, which may contain any number of registers, register files, memories, and other blocks
\u2022 ral_block_ofs (uvm_block) \u2013 Contains all the CSR block instantiations\n\u2022 Reg2vip_*_adapter (uvm_reg_adapter) \u2013 This class defines an interface for converting between uvm_reg_bus_op and a specific bus transaction. For each CSR a respective adapter is present\n
All the components are defined in ofs-f2000x-pl/ofs-common/verification/testbench
Integration of components in testbench
\u2022 The RAL blocks and adapters for each CSR is instantiated in tb_env\n\u2022 The bar range for each CSR is also set in the tb_env\n
Sample Environment Integration snippets
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#8-modifying-uvm-testbench","title":"8 Modifying UVM Testbench","text":"The next sections describe what needs to be considered when modifying the UVM, adding a new interface to the testbench and creating a new UVM test for a customised OFS Accelerator platform.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#81-modifying-uvm-environment-for-new-shell-variant","title":"8.1 Modifying UVM environment for new Shell Variant","text":"OFS f2000x comprises a shell based on PCIe Gen4x16 and is named base_x16
This base_x16 shell is described by an RTL file list, IP File lists and setup scripts to complete the build flow
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#82-modifying-uvm-environment-and-setting-up-compile-and-run-flow","title":"8.2 Modifying UVM environment and setting up Compile and Run flow","text":"All the variants can mostly reuse the existing UVM infrastructure to setup the build and run flow
\u2022 Create directory under $OFS_BUILD_ROOT new variant e.g ofs-n9000\n\u2022 Change directory to $OFS_BUILD_ROOT/ofs-n9000/verification/scripts\n\u2022 modify Makefile it to point to the new RTL, IP and script files.\n
Following these three steps above will enable the build and sim flow to run the existing UVM TB and tests with new IOFS n9000 variant.
Adding a new interface requires signal connections in the testbench. An additional BFM or verification IP is needed to drive the new interface. The main testbench file tb_top.sv is found at the following location
ofs-f2000x-pl/verification/testbench\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#83-adding-a-new-ral-directory","title":"8.3 Adding a new RAL directory","text":"In case the design has many register changes and the user decides to generate all the new RAL models instead of reusing from existing base RAL models, the following steps will help to create and integrate a new RALDIR in the UVM environment.
\u2022 Generate the new RAL files in desired directory. Preferably under the \"ofs-f2000x-pl/verification/common/testbench\" \n\u2022 By default, the Makefile points to base FIM RAL so set the RALDIR path in the Makefile to the new generated RAL file directory\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#84-modifying-tbtest-files-for-new-variant","title":"8.4 Modifying TB/Test files for new Variant","text":"Create a define for the new variant. e.g 'define FIM_NEW. If you are modifying common files then add new logic under this define so other projects will not get affected with variant specific change.
If there are more changes, please create separate \"testbench\" and \"test\" folders under this new variant.
\u2022 Extend variant specific env file from base env file to add variant specific changes.\n\u2022 Create new test/seq lib files in \"tests\" folder.\n\u2022 Create new variant package, add new TB/Tests/Sequence lib files and also import the base package files.\n
If you are adding new files then make sure it's included in Makefile for the build+run flow.
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#85-uvm-pcie-drivers","title":"8.5 UVM PCIe Drivers","text":"The \"svt_pcie_driver_app_transaction_base_sequence\" is part of Synopsys PCIe VIP library. You can find the sequence definition in the following two directories
\u2022 Navigate to \"$DESIGNWARE_HOME/vip/svt/pcie_svt/Q-2020.03/sverilog/src/vcs/svt_pcie_driver_app_transaction_sequence_collection.svp\" file. All the base and PCIe sequences are defined in this file.\n\n\u2022 When the OFS UVM build command is executed, it creates \"vip\" directory under \"$OFS_BUILD_ROOT/ofs-f2000x-pl/verification\". You can also find the same sequence file at \"$IOFS_BUILD_ROOT/ofs-f2000x/verification/vip/pcie_vip/src/sverilog/vcs/svt_pcie_driver_app_transaction_sequence_collection.sv\"\n
"},{"location":"hw/f2000x/user_guides/ug_sim_ofs/ug_sim_ofs/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/ftile_devkit/","title":"F-Series (2xF-tile) Development Kit Collateral for OFS","text":"This folder contains applicable collateral for OFS PCIe Attach reference shell targeting the F-Series (2xF-tile) Development Kit DK-DEV-AGF027F1.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/","title":"Shell Developer Guide for Open FPGA Stack: Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a guide for OFS Agilex PCIe Attach developers targeting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). The following topics are covered in this guide:
- Compiling the OFS Agilex PCIe Attach FIM design
- Simulating the OFS Agilex PCIe Attach design
- Customizing the OFS Agilex PCIe Attach FIM design
- Configuring the FPGA with an OFS Agilex PCIe Attach FIM design
The FIM Development Walkthroughs Table lists all of the walkthroughs provided in this guide. These walkthroughs provide step-by-step instructions for performing different FIM Development tasks.
Table: FIM Development Walkthroughs
Walkthrough Name Category Install Quartus Prime Pro Software Setup Clone FIM Repository Setup Set Development Environment Variables Setup Set Up Development Environment Setup Compile OFS FIM Compilation Manually Generate OFS Out-Of-Tree PR FIM Compilation Change the Compilation Seed Compilation Run Individual Unit Level Simulation Simulation Run Regression Unit Level Simulation Simulation Add a new module to the OFS FIM Customization Modify and run unit tests for a FIM that has a new module Customization Hardware test a FIM that has a new module Customization Debug the FIM with Signal Tap Customization Compile the FIM in preparation for designing your AFU Customization Resize the Partial Reconfiguration Region Customization Modify PCIe Configuration Using OFSS Customization Modify PCIe Configuration Using IP Presets Customization Create a Minimal FIM Customization Migrate to a Different Agilex Device Number Customization Modify the Ethernet Sub-System to 2x4x10GbE Customization Modify the Ethernet Sub-System to 3x4x10GbE Customization Remove the HPS Customization Set up JTAG Configuration Program the FPGA via JTAG Configuration"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#111-knowledge-pre-requisites","title":"1.1.1 Knowledge Pre-Requisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex 7 PCIe Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)).
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Signal Tap Logic Analyzer tool software.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex PCIe Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex PCIe Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new acceleration card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1211-top-level","title":"1.2.1.1 Top Level","text":"Figure: OFS Agilex PCIe Attach fseries-dk FIM Top-Level Diagram
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex PCIe Attach design are listed in the Release Capabilities Table. It describes the capabilities of the fseries-dk hardware as well as the capabilities of the default OFS Agilex PCIe Attach design targeting the fseries-dk.
Table: Release Capabilities
Interface fseries-dk Hardware Capabilities OFS Agilex PCIe Attach Default Design Implementation Host Interface[1] PCIe Gen4x8 PCIe Gen4x16 Network Interface 1 QSFP-56 cage1 QSFPDD-56 Cage 2x4x25GbE External Memory 3xDDR4 DIMMs sockets - 72-bits (1 available for HPS) 2xDDR4 - 2400MHz - 8Gb (1Gbx8) - 64-bits - No ECC1xDDR4 - 2400MHz - 16Gb (2Gbx8) - 40-bits - With ECC - For HPS [1] F-Tile ES version development kit has a form factor of PCIe Gen4x16, however it only supports PCIe Gen4x8. The OFS Agilex PCIe Attach design implements PCIe Gen4x16 with the assumption that it will train down to PCIe Gen4x8.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex PCIe Attach fseries-dk FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem AXI Streaming IP for PCI Express User Guide 790711 Memory Subsystem Memory Subsystem Intel FPGA IP User Guide for Intel Agilex OFS 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workload in the OFS Agilex PCIe Attach fseries-dk FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI Used to exercise and characterize HSSI interfaces. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex PCIe Attach fseries-dk FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the host software and AFU and board peripherals. The APF Address Map Table describes the address mapping of the APF, followed by the BPF Address Map Table which describes the address mapping of the BPF.
Table: APF Address Map
Address Size (Bytes) Feature 0x00000\u20130x3FFFF 256K Board Peripherals (See BPF Address Map table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket:4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x80000 \u2013 0x80FFF 4K AFU Error Reporting Table: BPF Address Mapping
Address Size (Bytes) Feature 0x00000 - 0x0FFFF 64K FME 0x10000 - 0x10FFF 4K PCIe 0x11000 - 0x11FFF 4K Reserved 0x12000 - 0x12FFF 4K QSFP0 0x13000 - 0x13FFF 4K QSFP1 0x14000 - 0x14FFF 4K HSSI 0x15000 - 0x15FFF 4K EMIF"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customizations Table lists the general user flows for OFS Agilex PCIe Attach fseries-dk FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customizations
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify PCIe Configuration Using OFSS Modify PCIe Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 2x4x10GbE Modify the Ethernet Sub-System to 3x4x10GbE Remove the HPS"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA acceleration card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Please see the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up the environment for deployment machines.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 N/A Quartus Prime Software Quartus Prime Pro Version 24.1 for Linux + Patches 0.18, 0.26 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A git 1.8.3.1 or later Section 1.3.1.2 FIM Source Files ofs-2024.2-1 Section 1.3.2.1"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"The source files for the OFS Agilex PCIe Attach FIM are provided in the following repository: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
Some essential directories in the repository are described as follows:
ofs-agx7-pcie-attach\n| syn // Contains files related to synthesis\n| | board // Contains synthesis files for several cards, including the fseries-dk | | | fseries-dk // Contains synthesis files for fseries-dk\n| | | | setup // Contains setup files, including pin constraints and location constraints\n| | | | syn_top // Contains Quartus project files\n| verification // Contains files for UVM testing\n| ipss // Contains files for IP Sub-Systems\n| | qsfp // Contains source files for QSFP Sub-System\n| | hssi // Contains source files for HSSI Sub-System\n| | pmci // Contains source files for PMCI Sub-System (not used in F-Tile FIM)\n| | pcie // Contains source files for PCIe Sub-System\n| | mem // Contains source files for Memory Sub-System\n| sim // Contains simulation files\n| | unit_test // Contains files for all unit tests\n| | | scripts // Contains script to run regression unit tests\n| license // Contains Quartus patch\n| ofs-common // Contains files which are common across OFS platforms\n| | verification // Contains common UVM files\n| | scripts // Contains common scripts\n| | | common\n| | | | syn // Contains common scripts for synthesis, including build script\n| | | | sim // Contains common scripts for simulation\n| | tools // Contains common tools files\n| | | mk_csr_module // Contains common files for CSR modules\n| | | fabric_generation // Contains common files for APF/BPF fabric generation\n| | | ofss_config // Contains common files for OFSS configuration tool\n| | | | ip_params // Contains default IP parameters for certain Sub-Systems when using OFSS\n| | src // Contains common source files, including host exercisers\n| tools //\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| | | hssi // Contains OFSS files for Ethernet-SS configuraiton\n| | | memory // Contains OFSS files for Memory-SS configuration\n| | | pcie // Contains OFSS files for PCIe-SS configuration\n| | | iopll // Contains OFSS files for IOPLL configuration\n| src // Contains source files for Agilex PCIe Attach FIM\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | includes // Contains source file header files\n| | top // Contains top-level source files, including design top module\n| | afu_top // Contains top-level source files for AFU\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 PCIe Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n
-
Check out the correct tag of the repository
cd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.2-1)\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-agx7-pcie-attach\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.FTILE_DK_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.FTILE_DK_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 24.1 for Linux with Agilex FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 24.1 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
git 1.8.3.1 or later.
-
Verify version number
git --version\n
Example output:
git version 1.8.3.1\n
-
Clone the ofs-agx7-pcie-attach repository. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.18, 0.26. All required patches are provided in the Assets of the OFS FIM Release Tag: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
-
Extract and unzip the patch-agx7-2024-1.tar.gz file.
tar -xvzf patch-agx7-2024-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-24.1-0.18-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
- Change the Compilation Seed
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This sections describes the theory behind FIM compilation.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2121-top-level-ofss-file","title":"2.1.2.1 Top Level OFSS File","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2122-base-ofss-file","title":"2.1.2.2 Base OFSS File","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2123-pcie-ofss-file","title":"2.1.2.3 PCIe OFSS File","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2124-iopll-ofss-file","title":"2.1.2.4 IOPLL OFSS File","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2125-memory-ofss-file","title":"2.1.2.5 Memory OFSS File","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is: $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/fseries-dk/syn_top/output_files.
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes the images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Name Description ofs_top.sof The FIM design SRAM Object File; a binary file of the compiled FIM image. ofs_top_hps.sof The build assembly process combines the FPGA ofs_top.sof programming file with u-boot-spl-dtb.hex to produce this file."},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <PATH_OF_RELOCATABLE_PR_TREE> <BOARD_TARGET> <WORK_DIRECTORY>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <PATH_OF_RELOCATABLE_PR_TREE> String Specifies the location of the relocatable PR directory tree to be created. <BOARD_TARGET> fseries-dk Specifies the name of the board target. <WORK_DIRECTORY> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_hps.sof\n\u2514\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#224-he_null-fim","title":"2.2.4 HE_NULL FIM","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex PCIe Attach FIM for fseries-dk:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. The following is used build the default fseries-dk design:
./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk\n
The build command options allow for many modifications to the shell design at build time. The following tool is provided to help you conveniently get the build command for a specific shell configuration:
OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command. Note: If no OFSS file is specified, the build script will use the <target>.ofss file by default.
- Once the build script is complete, the build summary should report that the build is complete and passes timing. For example:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: fseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#226-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM. This can be automatically done for you if you run the build script with the -p option. This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the fseries-dk OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_fseries-dk/pr_build_template fseries-dk work_fseries-dk\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
set_global_assignment -name SEED 1\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#3-fim-simulation","title":"3. FIM Simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config> OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> fseries-dk | n6001 Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional Refer to the Run Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files automatically; refer to the Run Regression Unit Level Simulation section for step-by-step instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#321-walkthrough-run-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Run Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the fseries-dk
./gen_sim_files.sh fseries-dk\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"You may use the regression script regress_run.py to run some or all of the unit tests available with a single command. The regression script is in the following location:
$OFS_ROOTDIR/sim/unit_test/scripts/regress_run.py\n
The usage of the regression script is as follows:
regress_run.py [-h] [-l] [-n <num_procs>] [-k <test_package>] [-s <simulator>] [-g] [--ofss <ip_config>] [-b <board_name>] [-e]\n
The Regression Unit Test Script Options table describes the options for the regress_run.py script.
Table: Regression Unit Test Script Options
Field Options Description -h | --help N/A Show the help message and exit -l | --local N/A Run regression locally -n | --n_procs Integer Maximum number of processes/tests to run in parallel when run locally. This has no effect on farm runs. -k | --pack all | fme | he | hssi | list | mem | pmci Test package to run during regression. The \"list\" option will look for a text file named \"list.txt\" in the \"unit_test\" directory for a text list of tests to run (top directory names). The default test package is all. -s | --sim vcs | vcsmx | msim Specifies the simulator used for the regression tests. The default simulator is vcs -g | --gen_sim_files N/A Generate simulation files. This only needs to be done once per repo update. This is the equivalent of running the gen_sim_files.sh script. -o | --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. -b | --board_name n6000 | n6001 | fseries-dk Specifies the board target -e | --email_list String Specifies email list to send results to multiple recipients The log for each unit test that is run by the regression script is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#331-walkthrough-run-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Run Regression Unit Level Simulation","text":"Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a test list file to only run the unit level simulations that are supported for the fseries-dk FIM.
touch $OFS_ROOTDIR/sim/unit_test/list.txt\n
Copy the following list into the new file. You may remove tests from this list as desired.
./bfm_test/set_params.sh\n./csr_test/set_params.sh\n./dfh_walker/set_params.sh\n./flr/set_params.sh\n./fme_csr_access/set_params.sh\n./fme_csr_directed/set_params.sh\n./he_lb_test/set_params.sh\n./he_mem_lb_test/set_params.sh\n./he_null_test/set_params.sh\n./hssi_csr_test/set_params.sh\n./hssi_kpi_test/set_params.sh\n./hssi_test/set_params.sh\n./indirect_csr/set_params.sh\n./mem_ss_csr_test/set_params.sh\n./mem_ss_rst_test/set_params.sh\n./mem_tg_test/set_params.sh\n./pcie_ats_basic_test/set_params.sh\n./pcie_csr_test/set_params.sh\n./pcie_ss_axis_components/set_params.sh\n./pf_vf_access_test/set_params.sh\n./port_gasket_test/set_params.sh\n./qsfp_test/set_params.sh\n./remote_stp_test/set_params.sh\n./uart_csr/set_params.sh\n
-
Navigate to the unit test scripts directory.
cd $OFS_ROOTDIR/sim/unit_test/scripts\n
-
Run regression test with the your desired options. For example, to simulate with the options to generate simulation files, run locally, use 8 processes, run all tests, use VCS simulator, and target the fseries-dk:
python regress_run.py -g -l -n 8 -k list -s vcs -b fseries-dk\n
Note: You may run all available tests by using -k all instead of creating and using -k list, however not all tests are supported depending on the target board.
-
Once all tests are complete, check that the tests have passed.
Example output:
Passing Unit Tests:24/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n bfm_test:............... PASS -- Time Elapsed:0:01:14.600452\n csr_test:............... PASS -- Time Elapsed:0:01:30.972054\n dfh_walker:............. PASS -- Time Elapsed:0:01:15.179127\n flr:.................... PASS -- Time Elapsed:0:01:14.579890\n fme_csr_access:......... PASS -- Time Elapsed:0:00:48.545993\n fme_csr_directed:....... PASS -- Time Elapsed:0:00:54.702789\n he_lb_test:............. PASS -- Time Elapsed:0:02:11.371956\n he_mem_lb_test:......... PASS -- Time Elapsed:0:41:32.226191\n he_null_test:........... PASS -- Time Elapsed:0:01:11.791063\n hssi_csr_test:.......... PASS -- Time Elapsed:0:44:10.611323\n hssi_kpi_test:.......... PASS -- Time Elapsed:2:28:24.465424\n hssi_test:.............. PASS -- Time Elapsed:2:23:52.603328\n indirect_csr:........... PASS -- Time Elapsed:0:01:02.535460\n mem_ss_csr_test:........ PASS -- Time Elapsed:0:23:37.683859\n mem_ss_rst_test:........ PASS -- Time Elapsed:0:45:19.603426\n mem_tg_test:............ PASS -- Time Elapsed:0:28:59.823955\n pcie_ats_basic_test:.... PASS -- Time Elapsed:0:01:10.104139\n pcie_csr_test:.......... PASS -- Time Elapsed:0:01:10.891950\n pcie_ss_axis_components: PASS -- Time Elapsed:0:02:04.448343\n pf_vf_access_test:...... PASS -- Time Elapsed:0:01:09.465886\n port_gasket_test:....... PASS -- Time Elapsed:0:01:11.912088\n qsfp_test:.............. PASS -- Time Elapsed:0:05:10.887379\n remote_stp_test:........ PASS -- Time Elapsed:0:01:14.684407\n uart_csr:............... PASS -- Time Elapsed:0:01:34.763679\nFailing Unit Tests: 0/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n Number of Unit test results captured: 24\nNumber of Unit test results passing.: 24\nNumber of Unit test results failing.: 0\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4-fim-customization","title":"4. FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Table: FIM Customization Walkthroughs
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify PCIe Configuration Using OFSS Modify PCIe Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 2x4x10GbE Modify the Ethernet Sub-System to 3x4x10GbE Remove the HPS"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a information for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example will add a hello_fim module to the design. The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the Host. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the fseries-dk card. The process for these are described in this section.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can be mastered by the Host. The BPF is an interconnect generated by Platform Designer. The Hello FIM BPF Interface Diagram figure shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
Figure: Hello FIM BPF Interface Diagram
The BPF fabric is defined in $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt.
We will add the Hello FIM module to an un-used address space in the MMIO region. The Hello FIM MMIO Address Layout table below shows the MMIO region for the Host with the Hello FIM module added at base address 0x16000.
Table: Hello FIM MMIO Address Layout
Offset Feature CSR set 0x00000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 HSSI Interface 0x15000 EMIF Interface 0x16000 Hello FIM"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the Hello FIM CSR table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Table: Hello FIM CSR
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Make hello_fim source directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create hello_fim_top.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_top.sv\n
Copy the following code into hello_fim_top.sv:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : September 2023\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h001,\nparameter bit [3:0] FEAT_VER = 4'h1,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create hello_fim_com.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_com.sv\n
Copy the following code to hello_fim_com.sv:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h001,\nparameter bit [3:0] FEAT_VER = 4'h1,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Create hello_fim_design_files.tcl file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_design_files.tcl\n
Copy the following code into hello_fim_design_files.tcl
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf to include Hello FIM module
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top_sources.tcl to include Hello FIM design files
###########################################\n# Design Files\n###########################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_design_files.tcl\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/fabric_design_files.tcl to include BPF Hello FIM Slave IP.
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE $::env(BUILD_ROOT_REL)/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify $OFS_ROOTDIR/src/includes/fabric_width_pkg.sv to add Hello FIM slave information and update EMIF slave next offset.
localparam bpf_hello_fim_slv_baseaddress = 'h16000; // New\nlocalparam bpf_hello_fim_slv_address_width = 12; // New\nlocalparam bpf_emif_slv_next_dfh_offset = 'h1000; // Old value: 'hB000\nlocalparam bpf_hello_fim_slv_next_dfh_offset = 'hA000; // New\nlocalparam bpf_hello_fim_slv_eol = 'b0; // New\n
-
Modify $OFS_ROOTDIR/src/top/top.sv
-
Add bpf_hello_fim_slv_if to AXI interfaces
// AXI4-lite interfaces\n...\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width), .ARADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width)) bpf_hello_fim_slv_if();\n
-
Add Hello FIM instantiation
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (fabric_width_pkg::bpf_hello_fim_slv_address_width),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`else\ndummy_csr #(\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif\n
-
Add interfaces for Hello FIM slv to bpf instantiation
bpf bpf (\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\n);\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt to add the hello_fim module as a slave to the apf.
# NAME FABRIC BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 18 fme,pcie,pmci,qsfp0,qsfp1,emif,hssi,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute helper script to generate BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\n\nsh gen_fabrics.sh\n
-
Once the script completes, the following new IP is created: $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip.
-
[OPTIONAL] You may verify the BPF changes have been made correctly by opening bpf.qsys to analyze the BPF.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\n\nqsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qpf\n
Find the bpf_hello_fim_slv instance:
-
Compile the Hello FIM design
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p fseries-dk work_fseries-dk_hello_fim\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/dfh_walker/test_csr_defs.sv
-
Add HELLO_FIM_IDX entry to t_dfh_idx enumeration.
...\ntypedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX, // New\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nVUART_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n...\n
-
Add HELLO_FIM_DFH to get_dfh_names function.
...\nfunction automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\n...\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\"; // New\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\n...\nreturn dfh_names;\n...\n
-
Add expected DFH value for Hello FIM to the get_dfh_values function.
...\nfunction automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\n...\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_xxxxxx_1012;\ndfh_values[PMCI_DFH_IDX][39:16] = fabric_width_pkg::bpf_pmci_slv_next_dfh_offset;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_xxxxxx_0100; // New\ndfh_values[HELLO_FIM_DFH_IDX][39:16] = fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset; // New\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_xxxxxx_0014;\ndfh_values[ST2MM_DFH_IDX][39:16] = fabric_width_pkg::apf_st2mm_slv_next_dfh_offset;\n...\nreturn dfh_values;\n...\n
-
Generate simulation files
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\n./gen_sim_files.sh fseries-dk\n
-
Run DFH Walker Simulation
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\nsh run_sim.sh TEST=dfh_walker\n
-
Verify that the test passes, and that the output shows the Hello FIM in the DFH sequence
********************************************\n Running TEST(0) : test_dfh_walking\n********************************************\n...\n\nREAD64: address=0x00015000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000010001009\n\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nREAD64: address=0x00016000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x30000000a0000100\n\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000000a0000100)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000200001012\n\nPMCI_DFH\n Address (0x20000)\nDFH value (0x3000000200001012)\n...\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356791250000\nV C S S i m u l a t i o n R e p o r t\nTime: 356791250000 fs\nCPU Time: 61.560 seconds; Data structure size: 47.4Mb\nTue Aug 15 16:29:45 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#414-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
[OPTIONAL] In the work directory where the FIM was compiled, determine the PR Interface ID of your design. You can use this value at the end of the walkthrough to verify that the design has been configured to the FPGA.
cd $OFS_ROOTDIR/<work_directory>/syn/board/fseries-dk/syn_top/\n\ncat fme-ifc-id.txt\n
Example output:
98ed516f-d24d-5b71-ae12-e78cd641e4be\n
-
Switch to your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Run fpgainfo to determine the PCIe B:D.F of your board, and to verify the PR Interface ID matches the ID you found in Step 1.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
-
Initialize opae.io
sudo opae.io init -d <B:D.F>\n
For example:
sudo opae.io init -d b1:00.0\n
-
Run DFH walker. Note the value read back from offset 0x16000 indicates the DFH ID is 0x100 which matches the Hello FIM module.
sudo opae.io walk -d <B:D.F>\n
For example:
sudo opae.io walk -d b1:00.0\n
Example output:
...\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000000a0000100\n dfh: id = 0x100, rev = 0x0, next = 0xa000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000200001012\n dfh: id = 0x12, rev = 0x1, next = 0x20000, eol = 0x0, reserved = 0x0, feature_type = 0x3\n...\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d b1:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d b1:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d b1:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:b1:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration JTAG PCI Development Kit (Driver: dfl-pci)\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#415-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.5 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Perform the following steps in your development environment:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Synthesize the design using the -e build script option. You may skip this step if you are using a pre-synthesized design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -e fseries-dk work_hello_fim_with_stp\n
-
Open the design in Quartus. The Compilation Dashboard should show that the Analysis & Synthesis step has completed.
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/ofs_top.qpf\n
-
Open Tools -> Signal Tap Logic Analyzer
-
Select the Default template and click Create
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note that the clock selected should correspond to the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the block of interest and open the design instance for review.
-
In the Signal Configuration -> Clock box of the Signal Tap Logic Analyzer window, click the \"...\" button
-
In the Node Finder tool that opens, type clock into the Named field, and select hello_fim_top_inst in the Look in field, then click Search. Select the hello_fim_top_inst|clk signal from the Matching Nodes box and click the \">\" button to move it to the Nodes Found box. Click OK to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM.
-
In the Named field, enter csr_lite_if*. In the Look In field, select hello_fim_top_inst.
-
Select the nets that appear in the Matching Nodes box. Click the > button to add them to the Nodes Found box.
-
Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto_signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, stp_for_hello_fim.
-
Save the newly created Signal Tap file, in the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/ directory, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will aurtomatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE stp_for_hello_fim.stp\nset_global_assignment -name SIGNALTAP_FILE stp_for_hello_fim.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
./ofs-common/scripts/common/syn/build_top.sh -p -k fseries-dk work_hello_fim_with_stp\n
-
Ensure that the compile completes successfully and meets timing:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: fseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the fseries-dk. Refer to the Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the fseries-dk via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open the Quartus Signal Tap GUI
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap Logic Analyzer window, select File -> Open, and choose the stp_for_hello_fim.stp file.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable for the fseries-dk. In the Device: selection box select the Agilex device.
-
If the Agilex Device is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
Create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'stp_for_hello_fim' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, walk the DFH list or peek/poke the Hello FIM registers.
opae.io init -d 0000:b1:00.0\nopae.io walk -d 0000:b1:00.0\nopae.io release -d 0000:b1:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":"In the FIM build, the standard AFUs in the port gasket can be replaced with PIM-based AFUs, wrapped by the ofs_plat_afu() top-level module. Before invoking build_top.sh, set the environment variable AFU_WITH_PIM to the path of a sources text file -- the same sources file from examples such as sources.txt in hello_world:
export AFU_WITH_PIM=<path to>/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt\n
When set during the build_top.sh setup stage, the FIM build is configured to load the specified AFU. PR will continue to work, if enabled. The only difference with AFU_WITH_PIM is the change to the AFUs present at power-on. Keep in mind that the default exercisers were chosen to attach to all devices and force reasonable routing decisions during the FIM build across the PR boundary. AFU_WITH_PIM is best used for FIMs that disable PR.
Configuration of the user clock from an AFU's JSON file is ignored in a FIM build.
The AFU_WITH_PIM setting matters only during the setup stage of build_top.sh. It is not consumed by later stages.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#422-compiling-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2 Compiling the FIM in preparation for designing your AFU","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" blocks. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination, order or all can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4221-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p fseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_fseries-dk\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to adjust the PR region to fit the additional logic into the static region. Similarly, if you reduce logic in the FIM region, then you can adjust the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions as reported in the following file:
$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/fseries-dk/syn_top/output_files/ofs_top.fit.rpt\n
An example report of the resources usage by partitions defined for the FIM is shown as follows:
+------------------------------------------------------------------------------------------+\n; Logic Lock Region Constraints ;\n+--------------------------------------+-------------------------+-------------------------+\n; Name ; Place Region Constraint ; Route Region Constraint ;\n+--------------------------------------+-------------------------+-------------------------+\n; afu_top|port_gasket|pr_slot|afu_main ; (90, 40) to (350, 220) ; (0, 0) to (385, 329) ;\n+--------------------------------------+-------------------------+-------------------------+\n+----------------------------------------------------------------------------------------------+\n; Logic Lock Region Usage Summary ;\n+-------------------------------------------------------+--------------------------------------+\n; Statistic ; afu_top|port_gasket|pr_slot|afu_main ;\n+-------------------------------------------------------+--------------------------------------+\n; ALMs needed [=A-B+C] ; 48011.2 / 351140 ( 13 % ) ;\n; [A] ALMs used in final placement ; 53324.4 / 351140 ( 15 % ) ;\n; [B] Estimate of ALMs recoverable by dense packing ; 5452.3 / 351140 ( 1 % ) ;\n; [C] Estimate of ALMs unavailable ; 139.0 / 351140 ( < 1 % ) ;\n; ALMs used for memory ; 450.0 ;\n; Combinational ALUTs ; 67166 ;\n; Dedicated Logic Registers ; 87533 / 1404560 ( 6 % ) ;\n; I/O Registers ; 0 ;\n; Block Memory Bits ; 1737568 ;\n; M20Ks ; 137 / 5049 ( 2 % ) ;\n; DSP Blocks needed [=A-B] ; 0 / 3439 ( 0 % ) ;\n; [A] DSP Blocks used in final placement ; 0 / 3439 ( 0 % ) ;\n; [B] Estimate of DSPs recoverable by dense merging ; 0 / 3439 ( 0 % ) ;\n; Pins ; 0 ;\n; IOPLLs ; 0 ;\n; ; ;\n; Region Placement ; (90, 40) to (350, 220) ;\n+-------------------------------------------------------+--------------------------------------+\n
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to first analyze the PR logic lock regions in a default FIM design, then resize the PR region:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
The OFS flow provides the TCL file $OFS_ROOTDIR/syn/board/fseries-dk/setup/pr_assignments.tcl which defines the PR partition where the AFU is allocated. Each region is a rectangle defined by the origin coordinate (X0, Y0) and the top right corner coordinate (X1, Y1).
#####################################################\n# Main PR Partition -- green_region\n#####################################################\nset_instance_assignment -name PARTITION green_region -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name CORE_ONLY_PLACE_REGION ON -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name RESERVE_PLACE_REGION ON -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name PARTIAL_RECONFIGURATION_PARTITION ON -to afu_top|port_gasket|pr_slot|afu_main\n\nset_instance_assignment -name PLACE_REGION \"X90 Y40 X350 Y220\" -to afu_top|port_gasket|pr_slot|afu_main\nset_instance_assignment -name ROUTE_REGION \"X0 Y0 X385 Y329\" -to afu_top|port_gasket|pr_slot|afu_main\n
-
[OPTIONAL] Use Quartus Chip Planner to visualize the default PR region allocation.
-
Compile the design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk\n
-
Open the project in the Quartus GUI
quartus $OFS_ROOTDIR/work_fseries-dk/syn/board/fseries-dk/syn_top/ofs_top.qpf\n
-
Switch to the ofs_top view if necessary.
-
Click Tools -> Chip Planner to open the Chip Planner.
-
Analyze the regions shown. Note that the regions are made of rectangles described by an origin coordinate, region height, and region width. If you are modifying the regions, you will need to identify the coordinates of your desired region.
-
Close the Quartus GUI.
-
Make changes to the partial reconfiguraton region in the $OFS_ROOTDIR/syn/board/fseries-dk/setup/pr_assignments.tcl file. You can modify the size and location of existing lock regions or add new ones and assign them to the AFU PR partition.
-
Recompile your FIM and create the PR relocatable build tree using the following commands.
cd $OFS_ROOTDIR ofs-common/scripts/common/syn/build_top.sh -p fseries-dk work_fseries-dk_resize_pr\n
-
Analyze the resource utilization report $OFS_ROOTDIR/work_fseries-dk/syn/board/fseries-dk/syn_top/output_files/ofs_top.fit.rpt produced after recompiling the FIM.
-
Perform further modification to the PR regions until the results are satisfactory. Make sure timing constraints are met.
For more information on how to optimize the floor plan of your Partial Reconfiguration design refer to the following online documentation.
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3 - Floorplan the Design
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe sub-system IP and PF/VF MUX can be modified either using the OFSS flow or the IP Presets flow. The OFSS flow supports a subset of all available PCIe Sub-system settings, while the IP Preset flow can make any available PCIe Sub-system settings change. With PCIe-SS modifcations related to the PFs and VFs, the PF/VF MUX logic is automatically configured based on the PCIe-SS configuration.
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section.
The sections below describe each modification.
- PCIe Configuration Using OFSS
- PCIe Configuration Using IP Presets
- [PCIe Sub-System Target IP]
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS PCIe Attach FIM for the n6001 can support up to 8 PFs and 2000 VFs distributed accross all PFs.
For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 (if at least 1 VF present) | 2 (if no VFs present) Max # of PFs 8 Min # of VFs 1 on PF0 (if only 1 PF present) | 0 (if 2PFs present) Max # of VFs 2000 distributed across all PFs Note that as the number of VFs goes up, timing closure can become more difficult.
The scripts provided in ${OFS_ROOTDIR}/ofs-common/tools/ofss_config allows you to easily reconfigure the number of PFs and VFs, bar addresses, vendor/device ID values and more. The PCIe Subsystem IP parameters that can be modified can be seen by reviewing ${OFS_ROOTDIR}/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py
New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe-SS configuration registers contain the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00000001 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00000001"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4431-walkthrough-modify-pcie-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify PCIe Configuration Using OFSS","text":"Perform the following steps to use OFSS files to configure the PCIe Sub-system and PF/VF MUX. In this example, we will add a PF with 2 VFs to the default configuration.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment to test the design. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
View the default OFS PCIe Attach FIM for the fseries-dk PF/VF configuration in the the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n
-
Create a new PCIe OFSS file from the existing pcie_host.ofss file
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_pfvf_mod.ofss\n
-
Configure the new pcie_pfvf_mod.ofss with your desired PCIe settings. In this example we will add PF5 with 2 VFs.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n[pf5]\nnum_vfs = 2\n
-
Compile the FIM with the new PCIe OFSS file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_pfvf_mod.ofss fseries-dk work_fseries-dk_pfvf_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_pfvf_mod/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Verify the number of VFs on the newly added PF5. In this example, we defined 2 VFs on PF5 in Step 5.
sudo lspci -vvv -s b1:00.5 | grep VF\n
Example output:
Initial VFs: 2, Total VFs: 2, Number of VFs: 0, Function Dependency Link: 05\nVF offset: 4, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
-
Verify communication with the newly added PF5. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
-
Initialize the driver on PF5
sudo opae.io init -d b1:00.5\n
-
Read the GUID for the PF5 CSR stub.
sudo opae.io -d b1:00.5 -r 0 peek 0x8\nsudo opae.io -d b1:00.5 -r 0 peek 0x10\n
Example output:
0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#444-pcie-configuration-using-ip-presets","title":"4.4.4 PCIe Configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4441-walkthrough-modify-pcie-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Configuration Using IP Presets","text":"Perform the following steps to use an IP Preset file to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID of PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment to test the design. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script using your desired OFSS configration to create a working directory for the fseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup fseries-dk work_fseries-dk\n
-
Open the PCIe-SS in the work directory using Quartus Parameter Editor.
qsys-edit $OFS_ROOTDIR/work_fseries-dk/ipss/pcie/qip/pcie_ss.ip\n
-
In the IP Parameter Editor window, scroll down and select the PCIe Interfaces Port Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab. Modify the settings as desired. In this case, we are changing the Revision ID to 0x00000002. You may make any desired modifications.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, fseries-dk-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 6.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = fseries-dk-rev2\n
-
Compile the design with the modified new pcie_host_mod_preset.ofss file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_mod_preset.ofss fseries-dk work_fseries-dk_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_pcie_mod/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Determing the PCIe B:D.F of your board. You may use the OPAE command fpgainfo fme to determine this.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
-
Use lspci to verify that the PCIe changes have been implemented. In this example, the Rev for PF0 is 02.
lspci -nvmms b1:00.0\n
Example output:
Slot: b1:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 0001\nPhySlot: 1-1\nRev: 01\nNUMANode: 1\nIOMMUGroup: 190\n
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the Intel FPGA IP Subsystem for PCI Express which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. You must also reduce the IOPLL frequency to a maximum of 470 MHz.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#445-pcie-sub-system-target-ip","title":"**4.4.5 PCIe Sub-System Target IP **","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#4451-walkthrough-change-the-pcie-sub-system-target-ip","title":"4.4.5.1 Walkthrough: Change the PCIe Sub-System Target IP","text":"Perform the following steps to change the target PCIe IP from AXI Streaming Intel FPGA IP for PCI Express to Intel FPGA IP Subsystem for PCI Express.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Add the following line to the [settings] section of the PCIe OFSS file you are using. In this example, it will be added to the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file, which is the default PCIe OFSS file used for fseries-dk.
ip_component = pcie_ss\n
Note: To change back to using the AXI Streaming Intel FPGA IP for PCI Express, simply remove this line.
-
Build the FIM using the 470 MHz IOPLL and the PCIe OFSS file that was modified in step 3.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/iopll/iopll_470MHz.ofss,tools/ofss_config/pcie/pcie_host.ofss fseries-dk work_fseries-dk\n
-
When the build completes, the output files can be found in $OFS_ROOTDIR/work_fseries-dk/syn/board/fseries-dk/output_files.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#45-minimal-fim","title":"4.5 Minimal FIM","text":"In a minimal FIM, the exercisers and Ethernet subsystem are removed from the design, and the PF/VF configuration is reduced to either 2PF/0VF or 1PF/1VF.
There are two pre-provided PCIe configurations that can be used to create minimal FIM:
- 2PF: this PCIe configuration has two physical functions, with the APF/BPF on PF0, and the AFU PR region on PF1.
-
1PF/1VF: This PCIe configuration has a single physical function, with the APF/BPF on PF0, and the AFU PR region on PF0-VF0.
The FIM source repository contains OFSS file that can be used to configure the PCIe IP for the minimal FIM.
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2pf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#451-walkthrough-create-a-minimal-fim","title":"4.5.1 Walkthrough: Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment for hardware testing. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with Null HEs compile option, the No HSSI compile option, and the desired PCIe PF/VF configuration.
cd $OFS_ROOTDIR\n
-
2PF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2pf.ofss fseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_fseries-dk_minimal_fim\n
-
1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss fseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_fseries-dk_minimal_fim\n
-
Review the $OFS_ROOTDIR/work_fseries-dk_minimal_fim/syn/board/fseries-dk/syn_top/output_files/ofs_top.fit.rpt utilization report to see the utilization statistics for the Minimal fim. Refer to Appendix A: Resource Utilization Tables Table A-4 for the expected utilization for this Minimal FIM.
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_minimal_fim/syn/board/fseries-dk/syn_top/output_files/ofs_top_hps.sof image to your deployment environmenment.
-
Switch to your deployment environment, if different than your development environment.
-
Program the ofs_top_hps.sof image to the fseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Use lspci to verify that the PCIe changes have been implemented.
sudo lspci -vvv -s b1:00.0 | grep VF\n
Example output for a 1PF/1VF image:
Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 1, stride: 1, Device ID: bcce\nVF Migration: offset: 00000000, BIR: 0\n
-
You may wish to adjust the PR logic lock regions to maximize the resources available for the AFU. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#46-migrating-to-a-different-agilex-device-number","title":"4.6 Migrating to a Different Agilex Device Number","text":"The following instructions enable a user to change the device part number of the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). Please be aware that this release works with Agilex\u00ae 7 FPGA devices that have F-tile for PCIe and Ethernet. Other tiles will take further work.
You may wish to change the device part number for the following reasons
- Migrate to same device package but with a different density
- Migrate to a different package and with a different or same density
The default device for the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is AGFB027R24C2E2VR2
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"Perform the following steps to migrate to a different Agilex Device. In this example, we will migrate from the default AGFB027R24C2E2VR2 device to AGFB027R31C2E2V. The package will change from R24C to R31C.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/fseries-dk_base.ofss file to use AGFB027R31C2E2V.
[ip]\ntype = ofs\n\n[settings]\nplatform = n6001\nfamily = agilex\nfim = base_x16\npart = AGFB027R31C2E2V device_id = 6001\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf file to use AGFB027R31C2E2V.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY \"Agilex 7\"\nset_global_assignment -name DEVICE AGFB027R31C2E2V
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_pr_afu.qsf file to use AGFB027R31C2E2V.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex 7\nset_global_assignment -name DEVICE AGFB027R31C2E2V
-
If the device you are migrating to uses the same package and pinout, you do not need to modify the pinout constraints. In this example, because we are migrating from package R24C to R31C, we need to modify the pinout to match the new device. If you would like Quartus to attempt to pin out the design automatically, you may remove all pin assignments. Typically, you will still need to set certain pins manually in order to guide Quartus for a successful compile (e.g. transceiver reference clocks).
-
Commment out all pin assignments in the following files: * $OFS_ROOTDIR/syn/board/fseries-dk/setup/emif_loc.tcl * $OFS_ROOTDIR/syn/board/fseries-dk/setup/hps_loc.tcl * $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl
-
Identify the pins in the design that will be constrained. In this example we will manually constrain the QSFP reference clock and the PCIe reference clock to help guide the fitter. The Device Migration Pinout Table shows th pin assignments that will be set, along with the pin number for both the old R24C package and the new R31C package.
Net Name Pin Name AGF 027 R24C Pin # AGF 027 R31C Pin # \"qsfp_ref_clk(n)\" REFCLK_FGTL12C_Q1_RX_CH2n AV48 BD57 qsfp_ref_clk REFCLK_FGTL12C_Q1_RX_CH2p AW49 BB57 \"PCIE_REFCLK0(n)\" REFCLK_FGTR13A_Q1_RX_CH2n BU7 BP13 PCIE_REFCLK0 REFCLK_FGTR13A_Q1_RX_CH2p BR7 BN14 -
Re-pin the pins identified in the previous step in the $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl file for the new pinout for the AGF 027 R31C package.
set_location_assignment PIN_BB57 -to qsfp_ref_clk\nset_location_assignment PIN_BD57 -to \"qsfp_ref_clk(n)\"\nset_location_assignment PIN_BP13 -to \"PCIE_REFCLK0(n)\"\nset_location_assignment PIN_BN14 -to PCIE_REFCLK0\n
-
Uncomment the instance assignments related to he QSFP reference clock in the $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl file.
set_instance_assignment -name IO_STANDARD \"CURRENT MODE LOGIC (CML)\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=156250000\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=TRUE\" -to qsfp_ref_clk\n
-
Compile a flat design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh fseries-dk:flat work_fseries-dk\n
-
Verify that the build completes successfuly. If the compilation fails with errors relating to the pinout, review the error messages and modify the pinout accordingly. If there are timing violations, try building with a different seed. Refer to the Change the Compilation Seed section for instructions on changing the build seed.
-
When you are satisfied with the pinout, preserve it by hard-coding the desired pinout back to the followig files:
$OFS_ROOTDIR/syn/board/fseries-dk/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/fseries-dk/setup/hps_loc.tcl$OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl
-
When you are ready to re-incorporate PR into the design, modify the PR region to be compatible with the new device. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#471-walkthrough-add-or-remove-the-memory-sub-system","title":"4.7.1 Walkthrough: Add or remove the Memory Sub-System","text":"The Memory Sub-System can be added or removed to the FIM design using the INCLUDE_DDR4 macro defined in the ofs_top.qsf file. The Memory-SS is enabled by default in the fseries-dk.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the ofs_top.qsf file to either enable or disable the INCLUDE_DDR4 macro. Comment out this macro assignemnt if you wish to remove the Memory-SS.
Note: The default Memory-SS has connections to the HPS. When enabling the Memory-SS, you must either ensure that the INCLUDE_HPS and INCLUDE_UART macros are also enabled, or you must remove the connections from the Memory-SS to the HPS. Refer to the Remove the HPS section for step-by-step instructions on removing the HPS from the design.
-
Compile the design. Refer to the Compile OFS FIM section for step-by-step instructions.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System.
Note: The default HSSI-SS configuration for the fseries-dk is 2x4x25GbE.
Note: The fseries-dk does not support 2x200GbE or 1x400GbE configurations.
Note: Due to Ethernet differential pair routing on the ES version of the Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit, some differential pairs were swapped to improve signal routing. To account for the pair swap, there is a requirement to run a script to invert the differential traces. Refer to the \"HE-HSSI\" Section of the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on swapping the polarity of Ethernet differential pairs at run-time.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#481-walkthrough-modify-the-ethernet-sub-system-to-2x4x10gbe-using-ip-presets","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System to 2x4x10GbE Using IP Presets","text":"Perform the following steps to modify the Ethernet Sub-System to 2x4x10GbE. In this walkthrough, we will create a new IP presets file that will be used during the build flow. Note that this modification can be done with OFSS, but the presets flow for demonstration purposes.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script to create a work directry for the fseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss fseries-dk work_fseries-dk\n
-
Open the HSSI-SS IP in the work directory using qsys-edit
qsys-edit $OFS_ROOTDIR/work_fseries-dk/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Change the PORT_PROFILE parameter from 25GbE to 10GbE for Ports 0 through 7.
-
Create an IP Presets file for this Ethernet Subsystem IP configuration.
-
Click New in the bottom right corner of the Presets window.
-
In the New Preset window, set the Preset name. In this example, we will name it 2x4x10g-fseries-dk.
-
Click the ... button to select the save location of the Preset file.
-
In the Save As window, navigate to the $OFS_ROODIR/ipss/hssi/qip/hssi_ss/presets directory. Set the File Name to 2x4x10g-fseries-dk.qprs. Then click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close out of the IP Parameter Editor GUI. Do not save or generate the IP.
-
Create a new HSSI OFSS file named $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_2x4x10_ftile.ofss with the following contents. This will call the IP Presets file generated in Step 6.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\n num_channels = 8\ndata_rate = 10GbE\n preset = 2x4x10g-fseries-dk\n
-
Run the FIM build script with the new HSSI OFSS file created in Step 8.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_2x4x10_ftile.ofss fseries-dk work_fseries-dk_eth_2x4x10\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#482-walkthrough-modify-the-ethernet-sub-system-using-ofss","title":"4.8.2 Walkthrough: Modify the Ethernet Sub-System Using OFSS","text":"Perform the following steps to modify the Ethernet Sub-System to 3x4x25GbE. In this walkthrough, we will create HSSI OFSS file that will be used during the build flow.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a new HSSI OFSS file for the 3x4x25GbE configuration.
-
Create the hssi_3x4x25_ftile.ofss file
vim $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_3x4x25_ftile.ofss\n
-
Copy the following into the new HSSI OFSS file. Note that num_channels is set to 12:
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 12\ndata_rate = 25GbE\n\neth_f_rsfec = IEEE_802.3_RS_528_514\nclient_interface = MAC Avalon ST\n
-
Edit the pinout constraints to support the new channels.
-
Open the top_loc.tcl file
vim $OFS_ROOTDIR/syn/board/fseries-dk/setup/top_loc.tcl\n
-
Add hssi_if[11:8] connections, and change the qsfp_ref_clk pin to the global ref clock which can be accessed by all transceivers in the F-Tile.
set_location_assignment PIN_AW55 -to hssi_if[8].rx_p\nset_location_assignment PIN_BC55 -to hssi_if[9].rx_p\nset_location_assignment PIN_BG55 -to hssi_if[10].rx_p\nset_location_assignment PIN_BL55 -to hssi_if[11].rx_p\n\nset_location_assignment PIN_AY54 -to hssi_if[8].rx_n\nset_location_assignment PIN_BD54 -to hssi_if[9].rx_n\nset_location_assignment PIN_BH54 -to hssi_if[10].rx_n\nset_location_assignment PIN_BM54 -to hssi_if[11].rx_n\n\nset_location_assignment PIN_BB52 -to hssi_if[8].tx_p\nset_location_assignment PIN_BF52 -to hssi_if[9].tx_p\nset_location_assignment PIN_BK52 -to hssi_if[10].tx_p\nset_location_assignment PIN_BL49 -to hssi_if[11].tx_p\n\nset_location_assignment PIN_BA51 -to hssi_if[8].tx_n\nset_location_assignment PIN_BE51 -to hssi_if[9].tx_n\nset_location_assignment PIN_BJ51 -to hssi_if[10].tx_n\nset_location_assignment PIN_BM48 -to hssi_if[11].tx_n\n\nset_location_assignment PIN_AW49 -to qsfp_ref_clk\nset_location_assignment PIN_AV48 -to \"qsfp_ref_clk(n)\"\n
-
Change the number of QSFP ports from 2 to 3 in the $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv file.
localparam NUM_QSFP_PORTS_USED = 3; // Number of QSFP cages on board used by target hssi configuration\n
-
Edit $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/hssi_wrapper.sv so that the QSFP LED signals use NUM_QSFP_PORTS_USED defined in the previous step.
// Speed and activity LEDS\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_green, // Link up in Nx25G or 2x56G or 1x100G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_yellow, // Link up in Nx10G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_green, // Link up and activity seen\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_red // LOS, TX Fault etc\n
-
Build the flat FIM. It is recommended to build the flat FIM when making changes such as this to reduce complexity until the changes are finalized.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_3x4x25_ftile.ofss fseries-dk:flat work_fseries-dk_eth_12x25g\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#49-modifying-the-hps","title":"4.9 Modifying the HPS","text":"This section describes ways to modify the HPS.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#491-walkthrough-remove-the-hps","title":"4.9.1 Walkthrough: Remove the HPS","text":"Perform the following steps to remove the HPS from the FIM design.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script to create a work directry for the fseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup fseries-dk work_fseries-dk\n
-
Create a Memory Sub-system IP presets file with the connection to the HPS removed.
-
In the work directory created in Step 3, open the mem_ss.ip
qsys-edit $OFS_ROOTDIR/work_fseries-dk/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
In the IP Parameter Editor window that opens, remove the entries corresponding to the HPS (row #2) in the Memory Interfaces and Application Interfaces sections. To do this, click the row to be removed, then click the minus (-) button.
-
In the Presets pane, click New... to create a new IP presets file.
-
In the New Preset window, create a unique preset name. For example, fseries-dk-mem-no-hps.
-
Click the ... button to select the save location of the IP presets file. In the Save As window, set the Look In field to the memory IP presets directory $OFS_ROOTDIR/ipss/mem/qip/presets. Set the File Name field to match the name selected in Step 4. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Edit the Memory OFSS file $OFS_ROOTDIR/tools/ofss_config/memory/memory_ftile.ofss to use the IP presets file generated in Step 4.
[ip]\ntype = memory\n\n[settings]\noutput_name = mem_ss\npreset = fseries-dk-mem-no-hps\n
-
Edit $OFS_ROOTDIR/syn/board/fseries-dk/syn_top/ofs_top.qsf to comment out the INCLUDE_HPS macro definition.
#set_global_assignment -name VERILOG_MACRO \"INCLUDE_HPS\"\n
-
Build the FIM.
./ofs-common/scripts/common/syn/build_top.sh -p fseries-dk work_fseries-dk_no_hps\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the fseries-dk is done by programming a SOF image to the FPGA via JTAG using Quartus Programer.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"The Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) has an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG.
Perform the following steps to establish a JTAG connection with the fseries-dk.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
Steps:
-
Refer to the following figure for Steps 2 and 3.
-
Locate Single DIP Switch SW2 and 4-position DIP switch SW3 on the fseries-dk. These switches control the JTAG setup for the board. Ensure that both SW2 and SW3.3 are set to ON.
-
Locate the J10 Micro-USB port on the fseries-dk. Connect a Micro-USB to USB-A cable between the J10 port and the workstation that has Quartus Prime Pro tools installed.
-
Use the jtagconfig tool to check that the JTAG chain contains the AGFB027R24C2E2VR2 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
1) Agilex F-Series FPGA Dev Kit [1-6]\n0343B0DD AGFB027R24C(.|R2|R0)/..\n020D10DD VTAP10\n
This concludes the walkthrough for establishing a JTAG connection on the fseries-dk.
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the fseries-dk. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is connected via JTAG. The Quartus programmer version must be the same as the version of Quartus used to build the design.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the fseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
Note: the Quartus programmer version must be the same as the version of Quartus used to build the design.
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the fseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB027R24C2E2VR2 device. The JTAG chain should show the device.
-
Right click the AGFB027R24C2E2VR2 row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGFB027R24C2E2VR2 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : user\n
"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#appendix","title":"Appendix","text":""},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#appendix-a-resource-utilization-tables","title":"Appendix A: Resource Utilization Tables","text":"Table A-1 Default FIM Resource Utilization with PR
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 189241.5 20.73 585 4.41 PCIE_RST_CTRL.rst_ctrl 463.5 0.05 0 0.0 afu_top 129761.2 14.22 320 2.41 auto_fab_0 1295.2 0.14 8 0.06 bpf 661.2 0.07 0 0.0 fme_top 638.1 0.07 6 0.05 hps_ss 0.0 0.0 0 0.0 hssi_wrapper 15902.6 1.74 80 0.6 local_mem_wrapper 5517.6 0.6 30 0.23 ofs_top_auto_tiles 14591.8 1.6 20 0.15 pcie_wrapper 18469.5 2.02 113 0.85 pmci_dummy_csr 684.8 0.08 0 0.0 qsfp_0 626.1 0.07 4 0.03 qsfp_1 626.0 0.07 4 0.03 sys_pll 0.5 0.0 0 0.0 Table A-2 Minimal FIM Resource Utilization
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 82271.4 9.01 345 2.6 PCIE_RST_CTRL.rst_ctrl 351.4 0.04 0 0.0 afu_top 45522.5 4.99 190 1.43 auto_fab_0 1281.0 0.14 8 0.06 bpf 775.6 0.08 0 0.0 fme_top 727.8 0.08 6 0.05 hps_ss 0.0 0.0 0 0.0 hssi_dummy_csr 689.2 0.08 0 0.0 local_mem_wrapper 5512.2 0.6 30 0.23 ofs_top_auto_tiles 6994.1 0.77 10 0.08 pcie_wrapper 18365.5 2.01 101 0.76 pmci_dummy_csr 682.7 0.07 0 0.0 qsfp0_dummy_csr 684.0 0.07 0 0.0 qsfp1_dummy_csr 684.3 0.07 0 0.0 sys_pll 0.5 0.0 0 0.0"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/ftile_devkit/doc_modules/ftile_wt_program_fpga_via_jtag/","title":"Ftile wt program fpga via jtag","text":"This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)))) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the fseries-dk. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
sudo fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010202A8769764\nBitstream Version : 5.0.1\nPr Interface Id : b541eb7c-3c7e-5678-a660-a54f71594b34\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the fseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the fseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB027R24C2E2VR2 device. The JTAG chain should show the device.
-
Right click the AGFB027R24C2E2VR2 row and selct Change File.
-
In the Select New Programming File window that opens, select ofs_top_hps.sof and click Open.
-
Check the Program/Configure box for the AGFB027R24C2E2VR2 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
"},{"location":"hw/ftile_devkit/doc_modules/ftile_wt_set_up_jtag/","title":"Ftile wt set up jtag","text":"The Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) has an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG.
Perform the following steps to establish a JTAG connection with the fseries-dk.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))] for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
Steps:
-
Refer to the following figure for Steps 2 and 3.
-
Locate Single DIP Switch SW2 and 4-position DIP switch SW3 on the fseries-dk. These switches control the JTAG setup for the board. Ensure that both SW2 and SW3.3 are set to ON.
-
Locate the J10 Micro-USB port on the fseries-dk. Connect a Micro-USB to USB-A cable between the J10 port and the workstation that has Quartus Prime Pro tools installed.
-
Use the jtagconfig tool to check that the JTAG chain contains the AGFB027R24C2E2VR2 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
1) Agilex F-Series FPGA Dev Kit [1-6]\n0343B0DD AGFB027R24C(.|R2|R0)/..\n020D10DD VTAP10\n
This concludes the walkthrough for establishing a JTAG connection on the fseries-dk.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/","title":"Getting Started Guide: Open FPGA Stack for Intel Agilex 7 FPGAs Targeting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)","text":"Last updated: November 19, 2024
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in evaluating the 2024.2-1 version of the PCIe Attach release targeting the F-Series Development Kit. This document will not cover Board Installation Guidelines or OFS Software Installation. Instead it will recommend you use a software installer to allow for fast evaluation. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Load and verify firmware targeting the FIM and AFU regions of the AGFB027R24C2E2VR2 FPGA
- Verify full stack functionality offered by the PCIe Attach OFS solution
- Learn where to find additional information on other PCIe Attach ingredients
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell targeting the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile). This platform is a Development Kit intended to be used as a starting point for evaluation and development of the Intel Agilex 7 FPGA F-Series with two F-Tiles.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#13-references-and-versions","title":"1.3 References and Versions","text":""},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-2-software-and-component-version-summary-for-ofs-pcie-attach-targeting-the-f-series-development-kit","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach targeting the F-Series Development Kit","text":"The OFS 2024.2-1 PCIe Attach release targeting the F-Series Development Kit is built upon tightly coupled software and Operating System version(s). The repositories listed below are used to manually build the Shell and the AFU portion of any potential workloads. Use this section as a general reference for the versions which compose this release. Specific instructions on building the FIM or AFU are discussed in their respective documents, but are shown here for the sake of completion.
Component Version Download Link Quartus Quartus Prime Pro Version 24.1 https://www.intel.com/content/www/us/en/software-kit/794624/intel-quartus-prime-pro-edition-design-software-version-24-1-for-linux.html, patches: 0.18, 0.26 Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.10/x86_64/product-software OneAPI-ASP ofs-2024.2-2 https://github.com/OFS/oneapi-asp/releases/tag/ofs-2024.2-1, patches: 0.02 OFS Platform AFU BBB ofs-2024.2-1 https://github.com/OFS/ofs-platform-afu-bbb/releases/tag/ofs-2024.2-1 OFS FIM Common Resources 2024.2-1 https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.2-1 AFU Examples tag: ofs-2024.2-1 https://github.com/OFS/examples-afu/releases/tag/ofs-2024.2-1 OPAE-SIM tag: 2.13.0-2 https://github.com/OPAE/opae-sim OFS SW Installer OFS 2024.2-1 Release Page"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-3-hardware-bkc-for-ofs-pcie-attach-targeting-the-f-series-development-kit","title":"Table 3: Hardware BKC for OFS PCIe Attach targeting the F-Series Development Kit","text":"The following table highlights the hardware which composes the Best Known Configuration (BKC) for the OFS 2024.2-1 PCIe Attach release targeting F-Series Development Kit.
Note: The Dell R750 server product line is known not to work with this release.
Component Link Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/agilex/agf027-and-agf023.html Intel FPGA Download Cable II https://www.intel.com/content/www/us/en/products/sku/215664/intel-fpga-download-cable-ii/specifications.html SuperMicro SYS-220HE-FTNR https://www.supermicro.com/en/products/system/hyper/2u/sys-220he-ftnr"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#14-reference-documents","title":"1.4 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.1-1/.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
OFS is a hardware and software infrastructure that provides an efficient approach to developing a customer FPGA-based platform or workload using an Intel, 3rd party, or custom board.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM), or shell of the FPGA provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The OFS architecture for Intel Agilex 7 FPGA provides modularity, configurability, and scalability. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem - a hierarchical design that targets the P-tile PCIe hard IP and is configured to support Gen4 speeds and Arm AXI4-Stream Data Mover functional mode.
- Ethernet Subsystem - provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
- Memory Subsystem - composed of 5 DDR4 channels; two HPS DDR4 banks, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each, and four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB
- Hard Processor System - 64-bit quad core ARM\u00ae Cortex*-A53 MPCore with integrated peripherals.
- Reset Controller
- FPGA Management Engine - Provides a way to manage the platform and enable acceleration functions on the platform.
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration. Each feature of the FME exposes itself to the kernel-level OFS drivers on the host through a Device Feature Header (DFH) register that is placed at the beginning of Control Status Register (CSR) space. Only one PCIe link can access the FME register space in a multi-host channel design architecture at a time.
Note: For more information on the FIM and its external connections, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required to support partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Like the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your entire AFU resides in a partial reconfiguration region of the FPGA.
- The AFU is part of the static region and is compiled as a flat design.
- Your AFU contains both static and PR regions.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components: ST2MM module, Virtio LB stub, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), Memory Host Exerciser (HE-MEM), Traffic Generator to memory (HE-MEM-TG), Port Gasket (PRG) and HPS Copy Engine.
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Basic HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
The AFU has the option to consume native packets from the host or interface channels or to instantiate a shim provided by the Platform Interface Manager (PIM) to translate between protocols.
Note: For more information on the Platform Interface Manager and AFU development and testing, refer to the Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#30-board-installation-and-server-requirements","title":"3.0 Board Installation and Server Requirements","text":"Instructions detailing the board installation guidelines for an F-Tile Dev Kit including server BIOS settings and regulatory information can be found in the Board Installation Guidelines: Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). This document also covers the installation of a JTAG cable, which is required for shell programming.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#40-f-series-development-kit-jtag-driver-setup","title":"4.0 F-Series Development Kit JTAG Driver Setup","text":"A specific JTAG driver needs to be installed on the host OS. Follow the instructions under the driver setup for Red Hat 5+ on Intel\u00ae FPGA Download Cable (formerly USB-Blaster) Driver for Linux*.
View the JTAG Chain after installing the proper driver and Quartus Programmer.
cd ~/intelFPGA_pro/quartus/bin\n./jtagconfig -D\n1) AGI FPGA Development Kit [1-13]\n(JTAG Server Version 23.3.0 Build 104 09/20/2023 SC Pro Edition)\n034BB0DD AGIB027R29A(.|R2|R3)/.. (IR=10)\n020D10DD VTAP10 (IR=10)\nDesign hash 27AA3E0B7CE0A5B9F366\n + Node 08586E00 (110:11) #0\n+ Node 0C006E00 JTAG UART #0\n+ Node 19104600 Nios II #0\n+ Node 30006E02 Signal Tap #2\n+ Node 30006E01 Signal Tap #1\n+ Node 30006E00 Signal Tap #0\nCaptured DR after reset = (0069761BB020D10DD) [65]\nCaptured IR after reset = (000D55) [21]\nCaptured Bypass after reset = (2) [3]\nCaptured Bypass chain = (0) [3]\nJTAG clock speed auto-adjustment is enabled. To disable, set JtagClockAutoAdjust parameter to 0\nJTAG clock speed 24 MHz\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#41-upgrading-the-f-series-development-kit-fim-via-jtag","title":"4.1 Upgrading the F-Series Development Kit FIM via JTAG","text":"Intel provides a pre-built FIM that can be used out-of-box for platform bring-up. This shell design is available on the OFS 2024.2-1 Release Page. After programming the shell and installing both the OPAE SDK and Backport Linux DFL kernel drivers as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach, you can confirm the correct FIM has been configured by checking the output of fpgainfo fme against the following table:
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-4-fim-version","title":"Table 4: FIM Version","text":"Identifier Value Pr Interface ID 98ed516f-d24d-5b71-ae12-e78cd641e4be Bitstream ID 360571656856467345 You will need to download and unpack the artifact images for this release before upgrading your device. You also need to set up the F-Series Development Kit as outlined in the Board Installation Guidelines: Intel Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) and Intel Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). The file ofs_top_hps.sof is the base OFS FIM file. This file is loaded into the FPGA using the development kit built in USB Blaster. Please be aware this FPGA is not loaded into non-volatile storage. If the server is power cycled you will need to reload the FPGA .sof file.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/fseries-dk-images_ofs-2024-2-1.tar.gz\ntar xf fseries-dk-images.tar.gz\ncd fseries-dk-images/\n
This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)))) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the fseries-dk. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
sudo fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x5010202A8769764\nBitstream Version : 5.0.1\nPr Interface Id : b541eb7c-3c7e-5678-a660-a54f71594b34\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the fseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the fseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB027R24C2E2VR2 device. The JTAG chain should show the device.
-
Right click the AGFB027R24C2E2VR2 row and selct Change File.
-
In the Select New Programming File window that opens, select ofs_top_hps.sof and click Open.
-
Check the Program/Configure box for the AGFB027R24C2E2VR2 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nboard_n6000.c:306:read_bmcfw_version() **ERROR** : Failed to get read object\nboard_n6000.c:482:print_board_info() **ERROR** : Failed to read bmc version\nboard_n6000.c:332:read_max10fw_version() **ERROR** : Failed to get read object\nboard_n6000.c:488:print_board_info() **ERROR** : Failed to read max10 version\nBoard Management Controller NIOS FW version:\nBoard Management Controller Build version:\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : 98ed516f-d24d-5b71-ae12-e78cd641e4be\nBoot Page : N/A\n
Note: The errors related to the BMC are the result of the OFS BMC not being present on the fseries-dk design. These will be removed in a future release.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#50-ofs-software-installation-from-script","title":"5.0 OFS Software Installation from Script","text":"An overview of the OFS software stack responsibilities and components can be found in the Software Installation Guide: Open FPGA Stack for PCIe Attach. In this document, we will instead use the provided PCIe Attach software installation script to quickly bring up the platform.
Before running the software installer, it is recommended you perform the following steps to lock your OS version to 8.10 and enable related repositories:
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
Download the OFS PCIe Attach installation script from the OFS 2024.2-1 Release Page. Unpack the files:
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/pcieattach_sw_installer_2024.1.zip\n\nunzip pcieattach_sw_installer_2024.1.zip\n
Executing the script without any input arguments will cause it to set up your local environment and will pull down and install pre-built OFS software artifacts. You can explore its other functionality as looking at the help output ofs_sw_install.py -h or though the included README. Execute the script.
./ofs_sw_install.py\n
You can check that all OPAE packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\n
To verify the Backport DFL kernel and driver stack have been installed, perform a warm reboot and check with uname.
Note: Your kernel version and tags may differ from the below, depending on script arguments
uname -r\n4.18.0-553.5.1.el8_10.x86_64\n
You can also check the contents of /usr/lib/modules/<kernel name>/kernel/drivers, or run modinfo <driver name>. Below is a list of the drivers which are loaded as a part of the F Tile Dev Kit Shell design:
lsmod | grep dfl\n\nqsfp_mem_dfl 16384 0\nqsfp_mem_core 20480 1 qsfp_mem_dfl\nuio_dfl 20480 0\nuio 28672 1 uio_dfl\ndfl_fme_mgr 20480 1\ndfl_emif 16384 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_afu 36864 0\ndfl_fme 49152 0\ndfl_pci 20480 0\ndfl 40960 7 dfl_pci,uio_dfl,dfl_fme,dfl_fme_br,qsfp_mem_dfl,dfl_afu,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 24576 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#51-memlock-limit","title":"5.1 Memlock limit","text":"Depending on the requirements of any loaded applications, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using:
ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#60-opae-tools-overview","title":"6.0 OPAE Tools Overview","text":"The following section offers a brief introduction including expected output values for the utilities included with OPAE. A full explanation of each command with a description of its syntax is available in the opae-sdk GitHub repo.
A list of all tools included in the OPAE SDK release can be found on the OPAE FPGA Tools tab of ofs.github.io.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#601-board-management-with-fpgainfo","title":"6.0.1 Board Management with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files.
Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Note: Your Bitstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo. As the F-Series Development Kit does not contain a traditional BMC as used by other OFS products, those lines in fpgainfo's output will not return valid objects. The subcommand fpgainfo bmc will likewise fail to report telemetry data.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x501020241BF165B\nBitstream Version : 5.0.1\nPr Interface Id : b4eda250-cdb7-5891-a06e-13d28d09bc32\nBoot Page : N/A\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#602-updating-with-fpgasupdate","title":"6.0.2 Updating with fpgasupdate","text":"The fpgasupdate tool is used to program AFU workloads into an open slot in a FIM. The fpgasupdate tool only accepts images that have been formatted using PACsign.
As the F-Series Development Kit does not contain a traditional BMC, you do not have access to a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL. Only the programming of a GBS workload is supported for this release.
The process of programming a SOF with a new FIM version is shown in section 4.0 Upgrading the F-Series Development Kit FIM via JTAG
sudo fpgasupdate ofs_pr_afu.gbs <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:42:31.58] [INFO ] updating from file ofs_pr_afu.gbs with size 19928064 [2022-04-14 16:42:31.60] [INFO ] waiting for idle [2022-04-14 16:42:31.60] [INFO ] preparing image file [2022-04-14 16:42:38.61] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] [2022-04-14 16:42:54.63] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] [2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:49:11.03] [INFO ] Secure update OK [2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#603-verify-fme-interrupts-with-hello_events","title":"6.0.3 Verify FME Interrupts with hello_events","text":"The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors.
Sample output from sudo hello_events.
sudo hello_events\nWaiting for interrupts now...\ninjecting error\nFME Interrupt occurred\nSuccessfully tested Register/Unregister for FME events!\nclearing error\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#61-host-exerciser-modules","title":"6.1 Host Exerciser Modules","text":"The reference FIM and unchanged FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. There are three HEMs present in the Intel OFS Reference FIM - HE-LPBK, HE-MEM, and HE-HSSI. These exercisers are tied to three different VFs that must be enabled before they can be used. Execution of these exercisers requires you bind specific VF endpoint to vfio-pci. The host-side software looks for these endpoints to grab the correct FPGA resource.
Refer to the Intel Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for a full description of these modules.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-5-module-pfvf-mappings","title":"Table 5: Module PF/VF Mappings","text":"Module PF/VF ST2MM PF0 HE-MEM PF0-VF0 HE-HSSI PF0-VF1 HE-MEM_TG PF0-VF2 HE-LB Stub PF1-VF0 HE-LB PF2 VirtIO LB Stub PF3 HPS Copy Engine PF4"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#611-he-mem-he-lb","title":"6.1.1 HE-MEM / HE-LB","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. The Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: - Latency (AFU to Host memory read) - MMIO latency (Write+Read) - MMIO BW (64B MMIO writes) - BW (Read/Write, Read only, Wr only)
The Host Exerciser Loopback Memory (HE-MEM) AFU is used to exercise use of FPGA connected DDR, data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host.
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. When using the F-Series Development Kit SmartNIC Platform, optimal performance requires the exercisers be run at 400 MHz.
Execution of these exercisers requires you to bind specific VF endpoint to vfio-pci. The following commands will bind the correct endpoint for a device with B/D/F 0000:b1:00.0 and run through a basic loopback test.
Note: While running the opae.io init command listed below, the command has failed if no output is present after completion. Double check that Intel VT-D and IOMMU have been enabled in the kernel as discussed in section 3.0 OFS DFL Kernel Drivers.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci Binding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci iommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188 Assigning /dev/vfio/188 to user Changing permissions for /dev/vfio/188 to rw-rw----\n\n$ sudo host_exerciser --clock-mhz 400 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 3002\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 8.732 GB/s\n Test lpbk(1): PASS\n
The following example will run a loopback throughput test using one cache line per request.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\n\n$ sudo host_exerciser --clock-mhz 400 --mode trput --cls cl_1 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2014\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 13.016 GB/s\n Test lpbk(1): PASS\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#612-traffic-generator-afu-test-application","title":"6.1.2 Traffic Generator AFU Test Application","text":"Beginning in OPAE version 2.0.11-1+ the TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank. The supported arguments for test configuration are:
- Number of test loops: --loops
- Number of read transfers per test loop: -r,--read
- Number of write transfers per test loop: -w,--write
- Burst size of each transfer: -b,--bls
- Address stride between each transfer: --stride
- Target memory TG: -m,--mem-channel
Below are some example commands for how to execute the test application. To run the preconfigured write/read traffic test on channel 0:
mem_tg tg_test\n
Target channel 1 with a 1MB single-word write only test for 1000 iterations
mem_tg --loops 1000 -r 0 -w 2000 -m 1 tg_test\n
Target channel 2 with 4MB write/read test of max burst length for 10 iterations
mem_tg --loops 10 -r 8 -w 8 --bls 255 -m 2 tg_test\n
sudo mem_tg --loops 1000 -r 2000 -w 2000 --stride 2 --bls 2 -m 1 tg_test\n[2022-07-15 00:13:16.349] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nTG PASS\nMem Clock Cycles: 17565035\nWrite BW: 4.37232 GB/s\nRead BW: 4.37232 GB/s\n
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#613-he-hssi","title":"6.1.3 HE-HSSI","text":"HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic.
The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode- specific options, and displays the ethernet statistics by invoking ethtool --statistics INTERFACE.
Due to Ethernet differential pair routing on the ES version of the Intel Agilex\u00ae 7 F-Series FPGA (Two F-Tiles) Development Kit, some differential pairs were swapped to improve signal routing. To account for the pair swap, there is a requirement to run a script to invert the differential traces. If you run the command \u201cfpgainfo phy B:d.f\u201d when the Ethernet ports are connected to known good sources and observe the following three ports are down as shown below:
$ sudo fpgainfo phy b1:00.0\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102020CB5B309\nBitstream Version : 5.0.1\nPr Interface Id : d0f93544-a487-5a92-8632-d43f27dbccee\nQSFP0 : Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
Create the following script called \u201cset_tx_inverse_polarity.sh\u201d to set make Transceiver PAM register settings:
#!/bin/sh\n# Port 3\nbase_addr=$(printf \"%08d\" \"0x500000\")\naddress=`expr $base_addr + 589884`\n#address=`expr $base_addr + 589884`\noffset=`expr $address/4`\nhex_number=$(printf \"0x%06x\" \"$(($offset))\")\necho $hex_number\ncmd_sts=$(printf \"%32x\" \"$(($offset2))\")\ncsraddr=\"${hex_number}0500000002\"\ncsraddr1=\"${hex_number}0600000001\"\ndata=1a040\necho $csraddr\nsudo opae.io poke 0x140b0 0x0001a26500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\nsleep 1\nsudo opae.io poke 0x140b0 0x0001226500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\n# Port 6\nbase_addr=$(printf \"%08d\" \"0xb00000\")\naddress=`expr $base_addr + 589884`\n#address=`expr $base_addr + 589884`\noffset=`expr $address/4`\nhex_number=$(printf \"0x%06x\" \"$(($offset))\")\necho $hex_number\ncmd_sts=$(printf \"%32x\" \"$(($offset2))\")\ncsraddr=\"${hex_number}0500000002\"\ncsraddr1=\"${hex_number}0600000001\"\ndata=1a040\necho $csraddr\nsudo opae.io poke 0x140b0 0x0001a16500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\nsleep 1\nsudo opae.io poke 0x140b0 0x0001216500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\n# Port 7\nbase_addr=$(printf \"%08d\" \"0x1100000\")\naddress=`expr $base_addr + 589884`\n#address=`expr $base_addr + 589884`\noffset=`expr $address/4`\nhex_number=$(printf \"0x%06x\" \"$(($offset))\")\necho $hex_number\ncmd_sts=$(printf \"%32x\" \"$(($offset2))\")\ncsraddr=\"${hex_number}0500000002\"\ncsraddr1=\"${hex_number}0600000001\"\ndata=1a040\necho $csraddr\nsudo opae.io poke 0x140b0 0x0001a26500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\nsleep 1\nsudo opae.io poke 0x140b0 0x0001226500000000\nsleep 1\nsudo opae.io poke 0x140a8 $csraddr\nsleep 1\nsudo opae.io peek 0x140a8\n
The script set_tx_inverse_polarity.sh requires the VFIO driver on PF0 to access the Transceiver registers. You will use the opae.io command prior to running set_tx_inverse_polarity.sh to bind the VFIO driver. Once the script completes, release the VFIO driver with opae.io release.
The listing below shows the script being run:
sudo opae.io init -d 0000:b1:00.0 $USER\nUnbinding (0x8086,0xbcce) at 0000:b1:00.0 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.0 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.0 is 8\nsh set_tx_inverse_polarity.sh\n0x16400f\n0x16400f0500000002\n0x16400f0500000006\n0x16400f0500000006\n0x2e400f\n0x2e400f0500000002\n0x2e400f0500000006\n0x2e400f0500000006\n0x46400f\n0x46400f0500000002\n0x46400f0500000006\n0x46400f0500000006\n\nsudo opae.io release -d 0000:b1:00.0\nReleasing (0x8086,0xbcce) at 0000:b1:00.0 from vfio-pci\nRebinding (0x8086,0xbcce) at 0000:b1:00.0 to dfl-pci\n\nsudo fpgainfo phy b1:00.0\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x501020241BF165B\nBitstream Version : 5.0.1\nPr Interface Id : 767712e5-b1d0-5777-aea9-592572a6817f\nQSFP0 : Connected\nQSFP1 : Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE UP\nPort1 :25GbE UP\nPort2 :25GbE UP\nPort3 :25GbE UP\nPort4 :25GbE UP\nPort5 :25GbE UP\nPort6 :25GbE UP\nPort7 :25GbE UP\n
The following example walks through the process of binding the VF corresponding with the HE-HSSI exerciser to vfio-pci, sending traffic, and verifying that traffic was received.
"},{"location":"hw/ftile_devkit/user_guides/ug_qs_ofs_ftile/ug_qs_ofs_ftile/#table-6-accelerator-pfvf-and-guid-mappings","title":"Table 6: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID F Series Dev Kit base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to user\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to user\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
-
Check Ethernet PHY settings with fpgainfo.
sudo fpgainfo phy -B 0xb1 Intel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102020CB5B309\nBitstream Version : 5.0.1\nPr Interface Id : d0f93544-a487-5a92-8632-d43f27dbccee\nQSFP0 : Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
-
Set loopback mode.
sudo hssiloopback --loopback enable --pcie-address 0000:b1:00.0 args Namespace(loopback='enable', pcie_address='0000:b1:00.0', port=0)\nsbdf: 0000:b1:00.0\nFPGA dev: {'segment': 0, 'bus': 177, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:b1:00.0'}\nargs.hssi_grps{0: ['dfl_dev.6', ['/sys/bus/pci/devices/0000:b1:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.6/uio/uio0']]}\nfpga uio dev:dfl_dev.6\n\n--------HSSI INFO START-------\nDFH :0x3000000010002015\nHSSI ID :0x15\nDFHv :0.5\nguidl :0x99a078ad18418b9d\nguidh :0x4118a7cbd9db4a9b\nHSSI version :1.0\nFirmware Version :1\nHSSI num ports :8\nPort0 :25GbE\nPort1 :25GbE\nPort2 :25GbE\nPort3 :25GbE\nPort4 :25GbE\nPort5 :25GbE\nPort6 :25GbE\nPort7 :25GbE\n--------HSSI INFO END-------\n\nhssi loopback enabled to port0\n
-
Send traffic through the 10G AFU.
$sudo hssi --pci-address b1:00.6 hssi_10g --num-packets 100 10G loopback test\nTx/Rx port: 99\nTx port: 0\nRx port: 0\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth:\n\nNo eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.\n0x40000 ETH_AFU_DFH: 0x1000010000001000\n0x40008 ETH_AFU_ID_L: 0xbb370242ac130002\n0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea\n0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000\n0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000\n0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000\n0x40048 AFU_SCRATCHPAD: 0x0000000045324511\n\n0x3c00 number_packets: 0x00000064\n0x3c01 random_length: 0x00000000\n0x3c02 random_payload: 0x00000000\n0x3c03 start: 0x00000000\n0x3c04 stop: 0x00000000\n0x3c05 source_addr0: 0x44332211\n0x3c06 source_addr1: 0x00006655\n0x3c07 dest_addr0: 0xaa998877\n0x3c08 dest_addr1: 0x0000ccbb\n0x3c09 packet_tx_count: 0x00000064\n0x3c0a rnd_seed0: 0x5eed0000\n0x3c0b rnd_seed1: 0x5eed0001\n0x3c0c rnd_seed2: 0x00025eed\n0x3c0d pkt_length: 0x00000040\n0x3cf4 tx_sta_tstamp: 0x02ab5f6d\n0x3cf5 tx_end_tstamp: 0x02ab63dd\n0x3d00 num_pkt: 0xffffffff\n0x3d01 pkt_good: 0x00000064\n0x3d02 pkt_bad: 0x00000000\n0x3d07 avst_rx_err: 0x00000000\n0x3d0b rx_sta_tstamp: 0x02ab6058\n0x3d0c rx_end_tstamp: 0x02ab64dc\n0x3e00 mac_loop: 0x00000000\n\nHSSI performance:\n Selected clock frequency : 402.832 MHz\n Latency minimum : 583.37 ns\n Latency maximum : 633.018 ns\n Achieved Tx throughput : 15.8863 GB/s\n Achieved Rx throughput : 15.6115 GB/s\n
The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. hssi_loopback tests both external and internal loopbacks.
The hssistats tool provides the MAC statistics.
"},{"location":"hw/iseries_devkit/","title":"I-Series (2xR-tile and 1xF-Tile) Development Kit Collateral for OFS","text":"This folder contains applicable collateral for OFS PCIe Attach reference shell targeting the I-Series (2xR-tile and 1xF-Tile) Development Kit DK-DEV-AGI027RBES.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/","title":"FPGA Interface Manager Developer Guide for Open FPGA Stack: Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a guide for OFS Agilex PCIe Attach developers targeting the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). The following topics are covered in this guide:
- Compiling the OFS Agilex PCIe Attach FIM design
- Simulating the OFS Agilex PCIe Attach design
- Customizing the OFS Agilex PCIe Attach FIM design
- Configuring the FPGA with an OFS Agilex PCIe Attach FIM design
The FIM Development Walkthroughs Table lists all of the walkthroughs provided in this guide. These walkthroughs provide step-by-step instructions for performing different FIM Development tasks.
Table: FIM Development Walkthroughs
Walkthrough Name Category Install Quartus Prime Pro Software Setup Clone FIM Repository Setup Set Development Environment Variables Setup Set Up Development Environment Setup Compile OFS FIM Compilation Manually Generate OFS Out-Of-Tree PR FIM Compilation Change the Compilation Seed Compilation Running Individual Unit Level Simulation Simulation Running Regression Unit Level Simulation Simulation Add a new module to the OFS FIM Customization Modify and run unit tests for a FIM that has a new module Customization Hardware test a FIM that has a new module Customization Debug the FIM with Signal Tap Customization Compile the FIM in preparation for designing your AFU Customization Resize the Partial Reconfiguration Region Customization Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Customization Modify PCIe Configuration Using IP Presets Customization Migrate to a Different Agilex Device Number Customization Modify the Ethernet Sub-System to 1x400GbE Customization Set up JTAG Configuration Program the FPGA via JTAG Configuration"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#111-knowledge-pre-requisites","title":"1.1.1 Knowledge Pre-Requisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex 7 PCIe Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Signal Tap Logic Analyzer tool software.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex PCIe Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex PCIe Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new acceleration card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":""},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1211-top-level","title":"1.2.1.1 Top Level","text":"Figure: OFS Agilex PCIe Attach iseries-dk FIM PCIe 1x16 Top-Level Diagram
Figure: OFS Agilex PCIe Attach iseries-dk FIM PCIe 2x8 Top-Level Diagram
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex PCIe Attach design are listed in the Release Capabilities Table. It describes the capabilities of the iseries-dk hardware as well as the capabilities of the default OFS Agilex PCIe Attach design targeting the iseries-dk.
Table: Release Capabilities
Interface iseries-dk Hardware Capabilities1 OFS Agilex PCIe Attach Provided Design Implementation Host Interface PCIe Gen5x16 \u2022 PCIe 1xGen5x16 (Default) \u2022 Bifurcated PCIe 2xGen5x8\u2022 PCIe 1xGen4x16 Network Interface 2 - QSFP-DD 3 Build Options: 1. QSFP 1,0 = 25 GbE2. QSFP 1,0 = 200 GbE 3. QSFP 0 = 400 GbE External Memory 2 - board mounted independent single rank DDR4-2666 8GB (1 Gb x 64 + 8b ECC)2 - DIMM sockets where each socket is single memory channels or independent channels (Check Dev Kit OPN for support option)2. 4xDDR4-2666 - 8GB (1Gb x 64 bits) - ECC not implemented by default 1 The iseries-dk FIM design was validated on DK-DEV-AGI027RBES with Agilex 7 device number AGIB027R29A1E2VR3. If you wish to use the production develpment kit (DK-DEV-AGI027-RA), you will need to migrate to device number AGIB027R29A1E1VB. Refer to the Migrate to a Different Agilex Device Number section for general instructions on this process.
2 The iseries-dk was validated with 2 Micron MTA8ATF1G64AZ-2G6E1 DDR4 SDRAM modules in the DIMM sockets.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex PCIe Attach iseries-dk FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem AXI Streaming IP for PCI Express User Guide 790711 Memory Subsystem Memory Subsystem Intel FPGA IP User Guide for Intel Agilex OFS 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workload in the OFS Agilex PCIe Attach iseries-dk FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI Used to exercise and characterize HSSI interfaces. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex PCIe Attach iseries-dk FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the host software and AFU and board peripherals. The APF Address Map Table describes the address mapping of the APF, followed by the BPF Address Map Table which describes the address mapping of the BPF.
Table: APF Address Map
Address Size (Bytes) Feature 0x00000\u20130x3FFFF 256K Board Peripherals (See BPF Address Map table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART (not used) 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket:4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x80000 \u2013 0x80FFF 4K AFU Error Reporting Table: BPF Address Mapping
Address Size (Bytes) Feature 0x00000 - 0x0FFFF 64K FME 0x10000 - 0x10FFF 4K PCIe 0x11000 - 0x11FFF 4K Reserved 0x12000 - 0x12FFF 4K QSFP0 0x13000 - 0x13FFF 4K QSFP1 0x14000 - 0x14FFF 4K HSSI 0x15000 - 0x15FFF 4K EMIF 0x20000 - 0x3FFFF 128K PMCI (note, PMCI is not implemented)"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customizations Table lists the general user flows for OFS Agilex PCIe Attach iseries-dk FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customizations
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Configuration Using IP Presets Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 1x400GbE"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA acceleration card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Please see the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up the environment for deployment machines.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 N/A Quartus Prime Software Quartus Prime Pro Version 24.1 for Linux + Patches 0.18, 0.26 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A git 1.8.3.1 or later Section 1.3.1.2 FIM Source Files ofs-2024.2-1 Section 1.3.2.1"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"The source files for the OFS Agilex PCIe Attach FIM are provided in the following repository: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
Some essential directories in the repository are described as follows:
ofs-agx7-pcie-attach\n| syn // Contains files related to synthesis\n| | board // Contains synthesis files for several cards, including the iseries-dk | | | iseries-dk // Contains synthesis files for iseries-dk\n| | | | setup // Contains setup files, including pin constraints and location constraints\n| | | | syn_top // Contains Quartus project files\n| verification // Contains files for UVM testing\n| ipss // Contains files for IP Sub-Systems\n| | qsfp // Contains source files for QSFP Sub-System\n| | hssi // Contains source files for HSSI Sub-System\n| | pmci // Contains source files for PMCI Sub-System (not used in F-Tile FIM)\n| | pcie // Contains source files for PCIe Sub-System\n| | mem // Contains source files for Memory Sub-System\n| sim // Contains simulation files\n| | unit_test // Contains files for all unit tests\n| | | scripts // Contains script to run regression unit tests\n| license // Contains Quartus patch\n| ofs-common // Contains files which are common across OFS platforms\n| | verification // Contains common UVM files\n| | scripts // Contains common scripts\n| | | common\n| | | | syn // Contains common scripts for synthesis, including build script\n| | | | sim // Contains common scripts for simulation\n| | tools // Contains common tools files\n| | | mk_csr_module // Contains common files for CSR modules\n| | | fabric_generation // Contains common files for APF/BPF fabric generation\n| | | ofss_config // Contains common files for OFSS configuration tool\n| | | | ip_params // Contains default IP parameters for certain Sub-Systems when using OFSS\n| | src // Contains common source files, including host exercisers\n| tools //\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| | | hssi // Contains OFSS files for Ethernet-SS configuraiton\n| | | memory // Contains OFSS files for Memory-SS configuration\n| | | pcie // Contains OFSS files for PCIe-SS configuration\n| | | iopll // Contains OFSS files for IOPLL configuration\n| src // Contains source files for Agilex PCIe Attach FIM\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | includes // Contains source file header files\n| | top // Contains top-level source files, including design top module\n| | afu_top // Contains top-level source files for AFU\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 PCIe Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n
-
Check out the correct tag of the repository
cd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.2-1)\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-agx7-pcie-attach\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.ISERIES_DK_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.ISERIES_DK_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 24.1 for Linux with Agilex FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 24.1 Build 94 06/14/2023 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
git 1.8.3.1 or later.
-
Verify version number
git --version\n
Example output:
git version 1.8.3.1\n
-
Clone the ofs-agx7-pcie-attach repository. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.18, 0.26. All required patches are provided in the Assets of the OFS FIM Release Tag: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
-
Extract and unzip the patch-agx7-2024-1.tar.gz file.
tar -xvzf patch-agx7-2024-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-23.4-0.17-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
- Change the Compilation Seed
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This section describes the theory behind FIM compilation.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2121-top-level-ofss-file","title":"2.1.2.1 Top Level OFSS File","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2122-base-ofss-file","title":"2.1.2.2 Base OFSS File","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2123-pcie-ofss-file","title":"2.1.2.3 PCIe OFSS File","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2124-iopll-ofss-file","title":"2.1.2.4 IOPLL OFSS File","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2125-memory-ofss-file","title":"2.1.2.5 Memory OFSS File","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is: $OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/iseries-dk/syn_top/output_files.
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes the images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Name Description ofs_top.sof The FIM design SRAM Object File; a binary file of the compiled FIM image."},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <PATH_OF_RELOCATABLE_PR_TREE> <BOARD_TARGET> <WORK_DIRECTORY>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <PATH_OF_RELOCATABLE_PR_TREE> String Specifies the location of the relocatable PR directory tree to be created. <BOARD_TARGET> iseries-dk Specifies the name of the board target. <WORK_DIRECTORY> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u2514\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#224-he_null-fim","title":"2.2.4 HE_NULL FIM","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex PCIe Attach FIM for iseries-dk:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. The following is used build the default iseries-dk design:
./ofs-common/scripts/common/syn/build_top.sh iseries-dk work_iseries-dk\n
The build command options allow for many modifications to the shell design at build time. The following tool is provided to help you conveniently get the build command for a specific shell configuration:
OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command. Note: If no OFSS file is specified, the build script will use the <target>.ofss file by default.
- Once the build script is complete, the build summary should report that the build is complete and passes timing. For example:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: iseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 1\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#226-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM. This can be automatically done for you if you run the build script with the -p option. This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the iseries-dk OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
./ofs-common/scripts/common/syn/build_top.sh iseries-dk work_iseries-dk\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_iseries-dk/pr_build_template iseries-dk work_iseries-dk\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
set_global_assignment -name SEED 2\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#3-fim-simulation","title":"3. FIM Simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config> OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> iseries-dk Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional Refer to the Running Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files automatically; refer to the [Run Regression Unit Level Simulation] section for step-by-step instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#321-walkthrough-running-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Running Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the iseries-dk
./gen_sim_files.sh iseries-dk\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"You may use the regression script regress_run.py to run some or all of the unit tests available with a single command. The regression script is in the following location:
$OFS_ROOTDIR/sim/unit_test/scripts/regress_run.py\n
The usage of the regression script is as follows:
regress_run.py [-h] [-l] [-n <num_procs>] [-k <test_package>] [-s <simulator>] [-g] [--ofss <ip_config>] [-b <board_name>] [-e]\n
The Regression Unit Test Script Options table describes the options for the regress_run.py script.
Table: Regression Unit Test Script Options
Field Options Description -h | --help N/A Show the help message and exit -l | --local N/A Run regression locally -n | --n_procs Integer Maximum number of processes/tests to run in parallel when run locally. This has no effect on farm runs. -k | --pack all | fme | he | hssi | list | mem | pmci Test package to run during regression. The \"list\" option will look for a text file named \"list.txt\" in the \"unit_test\" directory for a text list of tests to run (top directory names). The default test package is all. -s | --sim vcs | vcsmx | msim Specifies the simulator used for the regression tests. The default simulator is vcs -g | --gen_sim_files N/A Generate simulation files. This only needs to be done once per repo update. This is the equivalent of running the gen_sim_files.sh script. -o | --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. -b | --board_name iseries-dk Specifies the board target -e | --email_list String Specifies email list to send results to multiple recipients The log for each unit test that is run by the regression script is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#331-walkthrough-running-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Running Regression Unit Level Simulation","text":"Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a test list file to only run the unit level simulations that are supported for the iseries-dk FIM.
touch $OFS_ROOTDIR/sim/unit_test/list.txt\n
Copy the following list into the new file. You may remove tests from this list as desired.
./bfm_test/set_params.sh\n./csr_test/set_params.sh\n./dfh_walker/set_params.sh\n./flr/set_params.sh\n./fme_csr_access/set_params.sh\n./fme_csr_directed/set_params.sh\n./he_lb_test/set_params.sh\n./he_mem_lb_test/set_params.sh\n./he_null_test/set_params.sh\n./hssi_csr_test/set_params.sh\n./hssi_kpi_test/set_params.sh\n./hssi_test/set_params.sh\n./indirect_csr/set_params.sh\n./mem_ss_csr_test/set_params.sh\n./mem_ss_rst_test/set_params.sh\n./mem_tg_test/set_params.sh\n./pcie_ats_basic_test/set_params.sh\n./pcie_csr_test/set_params.sh\n./pcie_ss_axis_components/set_params.sh\n./pf_vf_access_test/set_params.sh\n./port_gasket_test/set_params.sh\n./qsfp_test/set_params.sh\n./remote_stp_test/set_params.sh\n./uart_csr/set_params.sh\n
-
Navigate to the unit test scripts directory.
cd $OFS_ROOTDIR/sim/unit_test/scripts\n
-
Run regression test with the your desired options. For example, to simulate with the options to generate simulation files, run locally, use 8 processes, run the specified test list, use VCS simulator, and target the iseries-dk:
python regress_run.py -g -l -n 8 -k list -s vcs -b iseries-dk\n
Note: You may run all available tests by using -k all instead of creating and using -k list, however not all tests are supported depending on the target board.
-
Once all tests are complete, check that the tests have passed.
Passing Unit Tests:24/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n bfm_test:............... PASS -- Time Elapsed:0:01:14.600452\n csr_test:............... PASS -- Time Elapsed:0:01:30.972054\n dfh_walker:............. PASS -- Time Elapsed:0:01:15.179127\n flr:.................... PASS -- Time Elapsed:0:01:14.579890\n fme_csr_access:......... PASS -- Time Elapsed:0:00:48.545993\n fme_csr_directed:....... PASS -- Time Elapsed:0:00:54.702789\n he_lb_test:............. PASS -- Time Elapsed:0:02:11.371956\n he_mem_lb_test:......... PASS -- Time Elapsed:0:41:32.226191\n he_null_test:........... PASS -- Time Elapsed:0:01:11.791063\n hssi_csr_test:.......... PASS -- Time Elapsed:0:44:10.611323\n hssi_kpi_test:.......... PASS -- Time Elapsed:2:28:24.465424\n hssi_test:.............. PASS -- Time Elapsed:2:23:52.603328\n indirect_csr:........... PASS -- Time Elapsed:0:01:02.535460\n mem_ss_csr_test:........ PASS -- Time Elapsed:0:23:37.683859\n mem_ss_rst_test:........ PASS -- Time Elapsed:0:45:19.603426\n mem_tg_test:............ PASS -- Time Elapsed:0:28:59.823955\n pcie_ats_basic_test:.... PASS -- Time Elapsed:0:01:10.104139\n pcie_csr_test:.......... PASS -- Time Elapsed:0:01:10.891950\n pcie_ss_axis_components: PASS -- Time Elapsed:0:02:04.448343\n pf_vf_access_test:...... PASS -- Time Elapsed:0:01:09.465886\n port_gasket_test:....... PASS -- Time Elapsed:0:01:11.912088\n qsfp_test:.............. PASS -- Time Elapsed:0:05:10.887379\n remote_stp_test:........ PASS -- Time Elapsed:0:01:14.684407\n uart_csr:............... PASS -- Time Elapsed:0:01:34.763679\nFailing Unit Tests: 0/24 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n Number of Unit test results captured: 24\nNumber of Unit test results passing.: 24\nNumber of Unit test results failing.: 0\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4-fim-customization","title":"4. FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Table: FIM Customization Walkthroughs
Customization Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Configuration Using IP Presets Migrate to a Different Agilex Device Number Modify the Ethernet Sub-System to 1x400GbE"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a information for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example will add a hello_fim module to the design. The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the Host. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the iseries-dk card. The process for these are described in this section.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can be mastered by the Host. The BPF is an interconnect generated by Platform Designer. The Hello FIM BPF Interface Diagram figure shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
Figure: Hello FIM BPF Interface Diagram
The BPF fabric is defined in $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt.
We will add the Hello FIM module to an un-used address space in the MMIO region. The Hello FIM MMIO Address Layout table below shows the MMIO region for the Host with the Hello FIM module added at base address 0x16000.
Table: Hello FIM MMIO Address Layout
Offset Feature CSR set 0x00000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 HSSI Interface 0x15000 EMIF Interface 0x16000 Hello FIM"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the Hello FIM CSR table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Table: Hello FIM CSR
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Make hello_fim source directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create hello_fim_top.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_top.sv\n
Copy the following code into hello_fim_top.sv:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : September 2023\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create hello_fim_com.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_com.sv\n
Copy the following code to hello_fim_com.sv:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Create hello_fim_design_files.tcl file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_design_files.tcl\n
Copy the following code into hello_fim_design_files.tcl
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qsf to include Hello FIM module
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top_sources.tcl to include Hello FIM design files
############################################\n# Design Files\n############################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_design_files.tcl\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/fabric_design_files.tcl to include BPF Hello FIM Slave IP.
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE $::env(BUILD_ROOT_REL)/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify $OFS_ROOTDIR/src/includes/fabric_width_pkg.sv to add Hello FIM slave information and update EMIF slave next offset.
localparam bpf_hello_fim_slv_baseaddress = 'h16000; // New\nlocalparam bpf_hello_fim_slv_address_width = 12; // New\nlocalparam bpf_emif_slv_next_dfh_offset = 'h1000; // Old value: 'hB000\nlocalparam bpf_hello_fim_slv_next_dfh_offset = 'hA000; // New\nlocalparam bpf_hello_fim_slv_eol = 'b0; // New\n
-
Modify $OFS_ROOTDIR/src/top/top.sv
-
Add bpf_hello_fim_slv_if to AXI interfaces
// AXI4-lite interfaces\n...\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width), .ARADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width)) bpf_hello_fim_slv_if();\n
-
Add Hello FIM instantiation
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (fabric_width_pkg::bpf_hello_fim_slv_address_width),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`else\ndummy_csr #(\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif\n
-
Add interfaces for Hello FIM slv to bpf instantiation
bpf bpf (\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\n);\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt to add the hello_fim module as a slave to the apf.
# NAME FABRIC BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 18 fme,pcie,pmci,qsfp0,qsfp1,emif,hssi,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute helper script to generate BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\n\nsh gen_fabrics.sh\n
-
Once the script completes, the following new IP is created: $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip.
-
[OPTIONAL] You may verify the BPF changes have been made correctly by opening bpf.qsys to analyze the BPF.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\n\nqsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
Find the bpf_hello_fim_slv instance:
-
Compile the Hello FIM design
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p iseries-dk work_iseries-dk_hello_fim\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/dfh_walker/test_csr_defs.sv
-
Add HELLO_FIM_IDX entry to t_dfh_idx enumeration.
...\ntypedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX, // New\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nVUART_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n...\n
-
Add HELLO_FIM_DFH to get_dfh_names function.
...\nfunction automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\n...\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\"; // New\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\n...\nreturn dfh_names;\n...\n
-
Add expected DFH value for Hello FIM to the get_dfh_values function.
...\nfunction automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\n...\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_xxxxxx_1012;\ndfh_values[PMCI_DFH_IDX][39:16] = fabric_width_pkg::bpf_pmci_slv_next_dfh_offset;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_xxxxxx_0100; // New\ndfh_values[HELLO_FIM_DFH_IDX][39:16] = fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset; // New\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_xxxxxx_0014;\ndfh_values[ST2MM_DFH_IDX][39:16] = fabric_width_pkg::apf_st2mm_slv_next_dfh_offset;\n...\nreturn dfh_values;\n...\n
-
Generate simulation files
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\n./gen_sim_files.sh iseries-dk\n
-
Run DFH Walker Simulation
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\nsh run_sim.sh TEST=dfh_walker\n
-
Verify that the test passes, and that the output shows the Hello FIM in the DFH sequence
********************************************\n Running TEST(0) : test_dfh_walking\n********************************************\n\n...\n\nTX: Tag Search Comparison: CPLD Tag: 00a Active Tag: 00a\nTX: PU Completion ADDED to transaction!\nTX: Match found for PU CPLD!\nRead64 Method Data Transaction: Address: 0000_0000_0001_5000 Data: 3000_0000_1000_1009\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nTX: Tag Search Comparison: CPLD Tag: 00b Active Tag: 00b\nTX: PU Completion ADDED to transaction!\nTX: Match found for PU CPLD!\nRead64 Method Data Transaction: Address: 0000_0000_0001_6000 Data: 3000_0000_a000_0100\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000000a0000100)\nTX: Tag Search Comparison: CPLD Tag: 00c Active Tag: 00c\nTX: PU Completion ADDED to transaction!\nTX: Match found for PU CPLD!\nRead64 Method Data Transaction: Address: 0000_0000_0002_0000 Data: 3000_0002_0000_1012\nPMCI_DFH\n Address (0x20000)\nDFH value (0x3000000200001012)\n...\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\n$finish called from file \"/home/applications.fpga.ofs.fim-n6001/sim/unit_test/dfh_walker/unit_test.sv\", line 236.\n$finish at simulation time 13720.141ns\n V C S S i m u l a t i o n R e p o r t\nTime: 13720141000 fs\nCPU Time: 3.290 seconds; Data structure size: 5.8Mb\nMon Dec 4 09:27:50 2023\nTotal of 5 minutes elapsed for dfh_walker\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#414-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
[OPTIONAL] In the work directory where the FIM was compiled, determine the PR Interface ID of your design. You can use this value at the end of the walkthrough to verify that the design has been configured to the FPGA.
cd $OFS_ROOTDIR/<work_directory>/syn/board/iseries-dk/syn_top/\n\ncat fme-ifc-id.txt\n
Example output:
b7855572-6ca9-58b8-bd11-44e1f1ab329f\n
-
Switch to your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Run fpgainfo to determine the PCIe B:D.F of your board, and to verify the PR Interface ID matches the ID you found in Step 1.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
-
Initialize opae.io
sudo opae.io init -d <B:D.F>\n
For example:
sudo opae.io init -d b1:00.0\n
-
Run DFH walker. Note the value read back from offset 0x16000 indicates the DFH ID is 0x100 which matches the Hello FIM module.
sudo opae.io walk -d <B:D.F>\n
For example:
sudo opae.io walk -d b1:00.0\n
Example output:
...\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000000a0000100\n dfh: id = 0x100, rev = 0x0, next = 0xa000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000200001012\n dfh: id = 0x12, rev = 0x1, next = 0x20000, eol = 0x0, reserved = 0x0, feature_type = 0x3\n...\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d b1:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d b1:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d b1:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d b1:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d b1:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:b1:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration JTAG PCI Development Kit (Driver: dfl-pci)\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#415-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.5 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Perform the following steps in your development environment:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Synthesize the design using the -e build script option. You may skip this step if you are using a pre-synthesized design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -e iseries-dk work_hello_fim_with_stp\n
-
Open the design in Quartus. The Compilation Dashboard should show that the Analysis & Synthesis step has completed.
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
-
Open Tools -> Signal Tap Logic Analyzer
-
Select the Default template and click Create
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note that the clock selected should correspond to the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the block of interest and open the design instance for review.
-
In the Signal Configuration -> Clock box of the Signal Tap Logic Analyzer window, click the \"...\" button
-
In the Node Finder tool that opens, type clock into the Named field, and select hello_fim_top_inst in the Look in field, then click Search. Select the hello_fim_top_inst|clk signal from the Matching Nodes box and click the \">\" button to move it to the Nodes Found box. Click OK to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM.
-
In the Named field, enter csr_lite_if*. In the Look In field, select hello_fim_top_inst.
-
Select the nets that appear in the Matching Nodes box. Click the > button to add them to the Nodes Found box.
-
Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto_signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, stp_for_hello_fim.
-
Save the newly created Signal Tap file, in the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/ directory, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will aurtomatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE STP_For_Hello_FIM.stp\nset_global_assignment -name SIGNALTAP_FILE STP_For_Hello_FIM.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
./ofs-common/scripts/common/syn/build_top.sh -p -k iseries-dk work_hello_fim_with_stp\n
-
Ensure that the compile completes successfully and meets timing:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: iseries-dk\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 1\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the iseries-dk. Refer to Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the iseries-dk via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/iseries-dk/syn_top/output_files/ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open the Quartus Signal Tap GUI
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap Logic Analyzer window, select File -> Open, and choose the stp_for_hello_fim.stp file.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable for the iseries-dk. In the Device: selection box select the Agilex device.
-
If the Agilex Device is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
Create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'stp_for_hello_fim' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, walk the DFH list or peek/poke the Hello FIM registers.
opae.io init -d 0000:b1:00.0\nopae.io walk -d 0000:b1:00.0\nopae.io release -d 0000:b1:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":"In the FIM build, the standard AFUs in the port gasket can be replaced with PIM-based AFUs, wrapped by the ofs_plat_afu() top-level module. Before invoking build_top.sh, set the environment variable AFU_WITH_PIM to the path of a sources text file -- the same sources file from examples such as sources.txt in hello_world:
export AFU_WITH_PIM=<path to>/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt\n
When set during the build_top.sh setup stage, the FIM build is configured to load the specified AFU. PR will continue to work, if enabled. The only difference with AFU_WITH_PIM is the change to the AFUs present at power-on. Keep in mind that the default exercisers were chosen to attach to all devices and force reasonable routing decisions during the FIM build across the PR boundary. AFU_WITH_PIM is best used for FIMs that disable PR.
Configuration of the user clock from an AFU's JSON file is ignored in a FIM build.
The AFU_WITH_PIM setting matters only during the setup stage of build_top.sh. It is not consumed by later stages.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#422-compiling-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2 Compiling the FIM in preparation for designing your AFU","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" blocks. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination, order or all can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4221-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_iseries-dk\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to adjust the PR region to fit the additional logic into the static region. Similarly, if you reduce logic in the FIM region, then you can adjust the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions as reported in the following file:
$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/iseries-dk/syn_top/output_files/ofs_top.fit.rpt\n
An example report of the resources usage by partitions defined for the FIM is shown as follows:
+------------------------------------------------------------------------------------------+\n; Logic Lock Region Constraints ;\n+--------------------------------------+-------------------------+-------------------------+\n; Name ; Place Region Constraint ; Route Region Constraint ;\n+--------------------------------------+-------------------------+-------------------------+\n; afu_top|port_gasket|pr_slot|afu_main ; (90, 40) to (350, 220) ; (0, 0) to (385, 329) ;\n+--------------------------------------+-------------------------+-------------------------+\n+----------------------------------------------------------------------------------------------+\n; Logic Lock Region Usage Summary ;\n+-------------------------------------------------------+--------------------------------------+\n; Statistic ; afu_top|port_gasket|pr_slot|afu_main ;\n+-------------------------------------------------------+--------------------------------------+\n; ALMs needed [=A-B+C] ; 48011.2 / 351140 ( 13 % ) ;\n; [A] ALMs used in final placement ; 53324.4 / 351140 ( 15 % ) ;\n; [B] Estimate of ALMs recoverable by dense packing ; 5452.3 / 351140 ( 1 % ) ;\n; [C] Estimate of ALMs unavailable ; 139.0 / 351140 ( < 1 % ) ;\n; ALMs used for memory ; 450.0 ;\n; Combinational ALUTs ; 67166 ;\n; Dedicated Logic Registers ; 87533 / 1404560 ( 6 % ) ;\n; I/O Registers ; 0 ;\n; Block Memory Bits ; 1737568 ;\n; M20Ks ; 137 / 5049 ( 2 % ) ;\n; DSP Blocks needed [=A-B] ; 0 / 3439 ( 0 % ) ;\n; [A] DSP Blocks used in final placement ; 0 / 3439 ( 0 % ) ;\n; [B] Estimate of DSPs recoverable by dense merging ; 0 / 3439 ( 0 % ) ;\n; Pins ; 0 ;\n; IOPLLs ; 0 ;\n; ; ;\n; Region Placement ; (90, 40) to (350, 220) ;\n+-------------------------------------------------------+--------------------------------------+\n
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to first analyze the PR logic lock regions in a default FIM design, then resize the PR region:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
The OFS flow provides the TCL file $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl which defines the PR partition where the AFU is allocated. Each region is a rectangle defined by the origin coordinate (X0, Y0) and the top right corner coordinate (X1, Y1).
#####################################################\n# Main PR Partition -- green_region\n#####################################################\nset_instance_assignment -name PARTITION green_region -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name CORE_ONLY_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name RESERVE_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name PARTIAL_RECONFIGURATION_PARTITION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n\nset_instance_assignment -name PLACE_REGION \"X90 Y40 X350 Y220\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name ROUTE_REGION \"X0 Y0 X385 Y329\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n
-
[OPTIONAL] Use Quartus Chip Planner to visualize the default PR region allocation.
-
Compile the design.
./ofs-common/scripts/common/syn/build_top.sh iseries-dk work_iseries-dk\n
-
Once the design is compiled, open it in Quartus.
quartus $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
-
Switch to the ofs_top view if necessary.
-
Click Tools -> Chip Planner to open the Chip Planner.
-
Analyze the regions shown. Note that the regions are made of rectangles described by an origin coordinate, region height, and region width. If you are modifying the regions, you will need to identify the coordinates of your desired region.
-
Close the Quartus GUI.
-
Make changes to the partial reconfiguraton region in the $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl file. You can modify the size and location of existing lock regions or add new ones and assign them to the AFU PR partition.
-
Recompile your FIM and create the PR relocatable build tree using the following commands.
cd $OFS_ROOTDIR ofs-common/scripts/common/syn/build_top.sh -p iseries-dk work_iseries-dk_resize_pr\n
-
Analyze the resource utilization report $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/output_files/ofs_top.fit.rpt produced after recompiling the FIM.
-
Perform further modification to the PR regions until the results are satisfactory. Make sure timing constraints are met.
For more information on how to optimize the floor plan of your Partial Reconfiguration design refer to the following online documentation.
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3 - Floorplan the Design
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe Subsystem can be easily modified using OFS provided script and the PCIe subsystem IP core. In this section both the PCIe SR-IOV configuration and PCIe configuration registers will be modified. You can use this process for setting up your specific settings.
The PCIe configuration for the iseries-dk can be set to 1x16 or bifurcated 2x8. You must ensure that the server you are using is set according to the design you are using. You can use OFSS to select which configuration to build with. Refer to the PCIe Configuration Using OFSS section for details. The following OFSS files are provided to you in the source repository:
- PCIe 1xGen5x16: $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss
- PCIe 2xGen5x8: $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link.ofss
- PCIe 1xGen4x16: $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_gen4.ofss
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section. Note that the older IP does not support the PCIe Gen5x16 configuration; it only supports Gen5 2x8 or Gen4 1x16. A walkthrough for these changes is provided later in this section.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS PCIe Attach FIM for the n6001 can support up to 8 PFs and 2000 VFs distributed accross all PFs.
For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 (if at least 1 VF present) | 2 (if no VFs present) Max # of PFs 8 Min # of VFs 1 on PF0 (if only 1 PF present) | 0 (if 2PFs present) Max # of VFs 2000 distributed across all PFs Note that as the number of VFs goes up, timing closure can become more difficult.
The scripts provided in ${OFS_ROOTDIR}/ofs-common/tools/ofss_config allows you to easily reconfigure the number of PFs and VFs, bar addresses, vendor/device ID values and more. The PCIe Subsystem IP parameters that can be modified can be seen by reviewing ${OFS_ROOTDIR}/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py
New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe-SS configuration registers contain the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. pcie_gen 4 | 5 N/A Specifies the PCIe generation pcie_instances 1 | 2 N/A Specifies the number of PCIe instances. When set to 1 the PCIe configuration will be 1x16. When set to 2 the PCIe configuration will be x8; the pcie_instances_enabled parameter will determine if it is 2x8 or 1x8. pcie_instances_enabled 1 | 2 N/A Specifies the number of PCIe instances that are enabled. When set to 2 the PCIe configuration will be 2x8 if; when set to 1 the PCIe configuration will be 1x16. Use only when the pcie_instances parameter is set to 2. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00000001 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00000001"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4431-walkthrough-modify-the-pcie-sub-system-and-pfvf-mux-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS","text":"Perform the following steps to use OFSS files to configure the PCIe Sub-system and PF/VF MUX. In this example both the PCIe SR-IOV configuration and PCIe configuration registers will be modified.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- To demonstrate updated PCIe PF/VF in hardware, use an OFS Agilex I-Series PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
View the default OFS PCIe Attach FIM for the iseries-dk PF/VF configuration in the the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n
-
Create a new PCIe OFSS file from the existing pcie_host.ofss file
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_pfvf_mod.ofss\n
-
Configure the new pcie_pfvf_mod.ofss with your desired PCIe settings. In this example we will add PF5 with 2 VFs.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n[pf5]\nnum_vfs = 2\n
-
Compile the FIM with the new PCIe OFSS file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_pfvf_mod.ofss iseries-dk work_iseries-dk_pfvf_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_iseries-dk_pfvf_mod/syn/board/iseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Verify the number of VFs on the newly added PF5. In this example, we defined 2 VFs on PF5 in Step 5.
sudo lspci -vvv -s b1:00.5 | grep VF\n
Example output:
Initial VFs: 2, Total VFs: 2, Number of VFs: 0, Function Dependency Link: 05\nVF offset: 4, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
-
Verify communication with the newly added PF5. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
-
Initialize the driver on PF5
sudo opae.io init -d b1:00.5\n
-
Read the GUID for the PF5 CSR stub.
sudo opae.io -d b1:00.5 -r 0 peek 0x8\nsudo opae.io -d b1:00.5 -r 0 peek 0x10\n
Example output:
0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4432-walkthrough-build-the-2x8-pcie-sub-system","title":"4.4.3.2 Walkthrough: Build the 2x8 PCIe Sub-System","text":"Perform the following steps to use OFSS files to build the FIM with a 2xGen5x8 PCIe configuration.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- To demonstrate updated PCIe PF/VF in hardware, use an OFS Agilex I-Series PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Build the FIM with the 2xGen5x16 PCIe configuration.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link.ofss iseries-dk work_iseries-dk\n
-
When the build script completes, copy the resulting $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/output_files/ofs_top.sof to your deployment environment if it is different than your development environment.
-
Switch to your deployment environment.
Note: Ensure that your server is configured for PCIe 2x8 bifurcated mode before continuing.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Verify the 6 PFs (3 for each link) are present with proper device ID. Not the PCIe B:D.F for each as this will be used later.
$ lspci | grep bcce\n
Example output:
ac:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nac:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nac:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\nad:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nad:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)\nad:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)\n
-
Verify the new VFs can be added. Use the OPAE SDK command pci_device to create VFs. Verify PF 0 has the proper number of VFs and have device ID of bccf.
sudo pci_device ac:00.0 vf 3\nsudo pci_device ad:00.3 vf 3\n
-
Verify that the VFs have been added.
```bash sudo lspci -vvv -s ac:00.0 | grep VF sudo lspci -vvv -s ad:00.0 | grep VF
Example Outputs:\n\n```bash\nInitial VFs: 3, Total VFs: 3, Number of VFs: 3, Function Dependency Link: 00\nVF offset: 3, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n\nInitial VFs: 3, Total VFs: 3, Number of VFs: 3, Function Dependency Link: 00\nVF offset: 3, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#444-pcie-configuration-using-ip-presets","title":"4.4.4 PCIe Configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4441-walkthrough-modify-pcie-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Configuration Using IP Presets","text":"Perform the following steps to use OFSS files to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID of PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- To demonstrate updated PCIe PF/VF in hardware, use an OFS Agilex I-Series PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script using your desired OFSS configration to create a working directory for the iseries-dk design.
./ofs-common/scripts/common/syn/build_top.sh --stage setup iseries-dk work_iseries-dk\n
-
Open the PCIe-SS in the work directory using Quartus Parameter Editor.
qsys-edit $OFS_ROOTDIR/work_iseries-dk/ipss/pcie/qip/pcie_ss.ip\n
-
In the IP Parameter Editor window, scroll down and select the PCIe Interfaces Port Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab. Modify the settings as desired. In this case, we are changing the Revision ID to 0x00000002. You may make any desired modifications.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, iseries-dk-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 8.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = iseries-dk-rev2\n
-
Compile the design with the modified new pcie_host_mod_preset.ofss file.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_mod_preset.ofss iseries-dk work_iseries-dk_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_fseries-dk_pcie_mod/syn/board/fseries-dk/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Determing the PCIe B:D.F of your board. You may use the OPAE command fpgainfo fme to determine this.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
-
Use lspci with the PCIe B:D.F of your board to verify that the PCIe changes have been implemented. In this example, the Rev for PF0 is 02.
lspci -nvmms 84:00.0\n
Example output:
Slot: 84:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 0001\nPhySlot: 1-1\nRev: 01\nNUMANode: 1\nIOMMUGroup: 190\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#445-pcie-sub-system-target-ip","title":"**4.4.5 PCIe Sub-System Target IP **","text":"As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section. Note that the older IP does not support the PCIe Gen5x16 configuration; it only supports Gen5 2x8 or Gen4 1x16.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#4451-walkthrough-change-the-pcie-sub-system-target-ip","title":"4.4.5.1 Walkthrough: Change the PCIe Sub-System Target IP","text":"Perform the following steps to change the target PCIe IP from AXI Streaming Intel FPGA IP for PCI Express to Intel FPGA IP Subsystem for PCI Express.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Add the following line to the [settings] section of the PCIe OFSS file you are using. In this example, it will be added to the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link.ofss file. Note that the older IP does not support the PCIe Gen5x16 configuration; it only supports Gen5 2x8 or Gen4 1x16. By using the pcie_host_2link.ofss file, the PCIe configuration will be Gen5 2x8.
ip_component = pcie_ss\n
Note: To change back to using the AXI Streaming Intel FPGA IP for PCI Express, simply remove this line.
-
Build the FIM using the 470 MHz IOPLL and the PCIe OFSS file that was modified in step 3.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/iopll/iopll_470MHz.ofss,tools/ofss_config/pcie/pcie_host_2link.ofss iseries-dk work_${{ env.ISERIS_DK_MODEL}}\n
-
When the build completes, the output files can be found in $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/output_files.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#45-minimal-fim","title":"4.5 Minimal FIM","text":"In a minimal FIM, the exercisers and Ethernet subsystem are removed from the design, the PF/VF configuration is reduced to either 2PF/0VF or 1PF/1VF, and the AFU PR area is expanded to make use of the available area previously used by the removed components.
There are two pre-provided PCIe configurations that can be used to create minimal FIM.
- 2PF: this PCIe configuration has two physical functions, with the APF/BPF on PF0, and the AFU PR region on PF1.
- 1PF/1VF: This PCIe configuration has a single physical function, with the APF/BPF on PF0, and the AFU PR region on PF0-VF0.
Block diagrams for the PCIe 2x8 and PCIe 1x16 configurations are shown below.
-
PCIe 1x16 Minimal Fim
-
PCIe 2x8 Minimal Fim
The FIM source repository contains OFSS file that can be used to configure the PCIe IP for the minimal FIM.
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2pf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2link_1pf_1vf.ofss
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#451-create-a-minimal-fim","title":"4.5.1 Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment for hardware testing. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
The OFS FIM repo provides a PR assignments TCL file which optimizes the PR region for the minimal FIM. Copy the minimal PR assignments TCL file into the pr_assignments.tcl file location for use in the FIM build process.
-
Rename the current pr_assignments.tcl file to pr_assignments_base.tcl for future use
mv $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments_base.tcl\n
-
Copy the pr_assignments_slim.tcl file to pr_assignments.tcl to be used in the current build
cp $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments_slim.tcl $OFS_ROOTDIR/syn/board/iseries-dk/setup/pr_assignments.tcl\n
-
Compile the FIM with Null HEs compile option, the No HSSI compile option, and the desired PCIe PF/VF configuration.
cd $OFS_ROOTDIR\n
-
PCIe 1x16 2PF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2pf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim\n
-
PCIe 1x16 1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim\n
-
PCIe 2x8 1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2link_1pf_1vf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim\n
-
Review the $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries-dk/syn_top/output_files/ofs_top.fit.rpt utilization report to see the utilization statistics for the Minimal fim. Refer to Appendix A: Resource Utilization Tables Table A-4 for the expected utilization for this Minimal FIM.
-
Copy the resulting $OFS_ROOTDIR/work_iseries-dk/syn/board/iseries/syn_top/output_files/ofs_top.sof image to your deployment environmenment.
-
Switch to your deployment environment, if different than your development environment.
-
Program the ofs_top.sof image to the iseries-dk FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step programming instructions.
-
Use lspci to verify that the PCIe changes have been implemented.
sudo lspci -vvv -s b1:00.0 | grep VF\n
Example output for a 1PF/1VF image:
Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 1, stride: 1, Device ID: bcce\nVF Migration: offset: 00000000, BIR: 0\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#46-migrating-to-a-different-agilex-device-number","title":"4.6 Migrating to a Different Agilex Device Number","text":"The following instructions enable a user to change the device part number of the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). Please be aware that this release works with Agilex\u00ae 7 FPGA devices that have F-tile for PCIe and Ethernet. Other tiles will take further work.
You may wish to change the device part number for the following reasons
- Migrate to same device package but with a different density
- Migrate to a different package and with a different or same density
The default device for the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) is AGIB027R29A1E2VR3
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"Perform the following steps to migrate to a different Agilex Device. In this example, we will migrate from the default AGIB027R29A1E2VR3 device to AGIB027R31A1E2VB. The package will change from R29A to R31A.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/iseries-dk_base.ofss file to use AGIB027R31A1E2VB.
[ip]\ntype = ofs\n\n[settings]\nplatform = n6001\nfamily = agilex\nfim = base_x16\npart = AGIB027R31A1E2VB device_id = 6001\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qsf file to use AGIB027R31A1E2VB.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY \"Agilex 7\"\nset_global_assignment -name DEVICE AGIB027R31A1E2VB
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_pr_afu.qsf file to use AGIB027R31A1E2VB.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex 7\nset_global_assignment -name DEVICE AGIB027R31A1E2VB
-
If the device you are migrating to uses the same package and pinout, you do not need to modify the pinout constraints. In this example, because we are migrating from package R29A to R31A, we need to modify the pinout to match the new device. If you would like Quartus to attempt to pin out the design automatically, you may remove all pin assignments. Typically, you will still need to set certain pins manually in order to guide Quartus for a successful compile (e.g. transceiver reference clocks).
-
Commment out all pin assignments in the following files: * $OFS_ROOTDIR/syn/board/iseries-dk/setup/emif_loc.tcl * $OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl
-
Identify the pins in the design that will be constrained. In this example we will manually constrain the QSFP reference clock and the PCIe reference clock to help guide the fitter. The Device Migration Pinout Table shows the pin assignments that will be set, along with the pin number for both the old R24C package and the new R31C package.
Net Name R29A Pin Name R31A Pin Name AGI 027 R29A Pin # AGI 027 R31A Pin # qsfp_ref_clk REFCLK_FGTL12A_Q2_RX_CH4p REFCLK_FGTL12C_Q2_RX_CH4p JD74 AM57 PCIE_REFCLK0 REFCLK_GXRL14C_CH0p REFCLK_GXRL14A_CH0p DR68 BU56 PCIE_REFCLK1 REFCLK_GXRL14C_CH1p REFCLK_GXRL14A_CH1p CU68 BP57 -
Re-pin the pins identified in the previous step in the $OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl file for the new pinout for the AGF 027 R31C package.
set_location_assignment PIN_AM57 -to qsfp_ref_clk\n\nset_location_assignment PIN_BU56 -to PCIE_REFCLK0\nset_location_assignment PIN_BP57 -to PCIE_REFCLK1\n
-
Uncomment the instance assignments related to he QSFP and PCIe reference clocks in the $OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl file.
set_instance_assignment -name IO_STANDARD \"CURRENT MODE LOGIC (CML)\" -to qsfp_ref_clk\n\nset_instance_assignment -name IO_STANDARD HCSL -to PCIE_REFCLK0\nset_instance_assignment -name IO_STANDARD HCSL -to PCIE_REFCLK1\n
-
Compile a flat design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh ${{ env.ISERIS_DK_MODEL }}:flat work_${{ env.ISERIS_DK_MODEL }}\n
-
Verify that the build completes successfuly. If the compilation fails with errors relating to the pinout, review the error messages and modify the pinout accordingly. If there are timing violations, try building with a different seed. Refer to the Change the Compilation Seed section for instructions on changing the build seed.
-
When you are satisfied with the pinout, preserve it by hard-coding the desired pinout back to the followig files:
$OFS_ROOTDIR/syn/board/iseries-dk/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/iseries-dk/setup/top_loc.tcl
-
When you are ready to re-incorporate PR into the design, modify the PR region to be compatible with the new device. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM. This section provides an example walkthrough for modifiying the Memory-SS.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#471-walkthrough-modify-the-memory-sub-system-using-ip-presets-with-ofss","title":"4.7.1 Walkthrough: Modify the Memory Sub-System Using IP Presets With OFSS","text":"This walkthrough will go through the flow of modifying the Memory-SS in the OFS FIM. In this example, we will enable ECC on memory channels 0-3.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script to create a work directory with the Memory-SS configured for the iseries-dk
./ofs-common/scripts/common/syn/build_top.sh --stage setup iseries-dk work_iseries-dk\n
-
Open the Memory Subsystem mem_ss.ip in IP Parameter Editor
qsys-edit $OFS_ROOTDIR/work_iseries-dk/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
The Memory Subsystem IP will open in IP Parameter Editor. Click Dive Into Packaged Subsystem.
-
The Platform Designer mem_ss view opens. All of the EMIFs are shown in the Filter window.
-
Click each EMIF 0 through 3 and perform the following actions.
-
In the Parameters window, click the Memory tab and change the DQ width to 72.
-
In the Parameters window, click the Controller tab. Scroll down and check the box for Enable Error Detection and Correction Logic with ECC.
-
Once Step 6 has been done for each EMIF 0-3, click File -> Save. Close the Platform Designer window.
-
In the IP Parameter Editor Presets window, click New to create an IP Presets file.
-
In the New Preset window, set the Name for the preset to iseries-dk.
-
Click the ... button to select the location for the Preset file.
-
In the Save As window, change the save location to $OFS_ROOTDIR/ipss/mem/qip/presets and change the File Name to mem_presets.qprs to overwrite the existing presets file. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close the IP Parameter Editor. You do not need to generate or save the IP.
-
Edit the $OFS_ROOTDIR/syn/board/iseries-dk/setup/emif_loc.tcl file to add pin assignments for the new signals supporting ECC on Channels 0-3.
# CH0 DQS8\nset_location_assignment PIN_BC29 -to ddr4_mem[0].dbi_n[8]\nset_location_assignment PIN_AT30 -to ddr4_mem[0].dqs_n[8]\nset_location_assignment PIN_AV29 -to ddr4_mem[0].dqs[8]\nset_location_assignment PIN_AT28 -to ddr4_mem[0].dq[64]\nset_location_assignment PIN_AV27 -to ddr4_mem[0].dq[66]\nset_location_assignment PIN_AT32 -to ddr4_mem[0].dq[65]\nset_location_assignment PIN_BC31 -to ddr4_mem[0].dq[67]\nset_location_assignment PIN_BF32 -to ddr4_mem[0].dq[68]\nset_location_assignment PIN_AV31 -to ddr4_mem[0].dq[69]\nset_location_assignment PIN_BC27 -to ddr4_mem[0].dq[70]\nset_location_assignment PIN_BF28 -to ddr4_mem[0].dq[71]\n# CH1 DQS8\nset_location_assignment PIN_BC59 -to ddr4_mem[1].dbi_n[8]\nset_location_assignment PIN_AT60 -to ddr4_mem[1].dqs_n[8]\nset_location_assignment PIN_AV59 -to ddr4_mem[1].dqs[8]\nset_location_assignment PIN_BC61 -to ddr4_mem[1].dq[64]\nset_location_assignment PIN_AT62 -to ddr4_mem[1].dq[65]\nset_location_assignment PIN_AV61 -to ddr4_mem[1].dq[66]\nset_location_assignment PIN_BC57 -to ddr4_mem[1].dq[67]\nset_location_assignment PIN_BF62 -to ddr4_mem[1].dq[68]\nset_location_assignment PIN_AT58 -to ddr4_mem[1].dq[69]\nset_location_assignment PIN_BF58 -to ddr4_mem[1].dq[70]\nset_location_assignment PIN_AV57 -to ddr4_mem[1].dq[71]\n# CH2 DQS8\nset_location_assignment PIN_MD30 -to ddr4_mem[2].dbi_n[8]\nset_location_assignment PIN_MK31 -to ddr4_mem[2].dqs_n[8]\nset_location_assignment PIN_MH30 -to ddr4_mem[2].dqs[8]\nset_location_assignment PIN_MD32 -to ddr4_mem[2].dq[64]\nset_location_assignment PIN_MH32 -to ddr4_mem[2].dq[65]\nset_location_assignment PIN_MK33 -to ddr4_mem[2].dq[66]\nset_location_assignment PIN_MC33 -to ddr4_mem[2].dq[67]\nset_location_assignment PIN_MD28 -to ddr4_mem[2].dq[68]\nset_location_assignment PIN_MH28 -to ddr4_mem[2].dq[69]\nset_location_assignment PIN_MK29 -to ddr4_mem[2].dq[70]\nset_location_assignment PIN_MC29 -to ddr4_mem[2].dq[71]\n# CH3 DQS8\nset_location_assignment PIN_MD42 -to ddr4_mem[3].dbi_n[8]\nset_location_assignment PIN_MK43 -to ddr4_mem[3].dqs_n[8]\nset_location_assignment PIN_MH42 -to ddr4_mem[3].dqs[8]\nset_location_assignment PIN_MH40 -to ddr4_mem[3].dq[64]\nset_location_assignment PIN_MD40 -to ddr4_mem[3].dq[65]\nset_location_assignment PIN_MC41 -to ddr4_mem[3].dq[66]\nset_location_assignment PIN_MK41 -to ddr4_mem[3].dq[67]\nset_location_assignment PIN_MH44 -to ddr4_mem[3].dq[68]\nset_location_assignment PIN_MK45 -to ddr4_mem[3].dq[69]\nset_location_assignment PIN_MD44 -to ddr4_mem[3].dq[70]\nset_location_assignment PIN_MC45 -to ddr4_mem[3].dq[71]\n
-
Compile the design.
./ofs-common/scripts/common/syn/build_top.sh -p iseries-dk work_iseries-dk_mem_ecc_preset\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#472-walkthrough-add-or-remove-the-memory-sub-system","title":"4.7.2 Walkthrough: Add or remove the Memory Sub-System","text":"The Memory Sub-System can be added or removed to the FIM design using the INCLUDE_DDR4 macro defined in the ofs_top.qsf file. The Memory-SS is enabled by default in the iseries-dk.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the ofs_top.qsf file to either enable or disable the INCLUDE_DDR4 macro. Comment out this macro assignemnt if you wish to remove the Memory-SS.
-
Compile the design. Refer to the Compile OFS FIM section for step-by-step instructions.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System.
Note: The default HSSI-SS configuration for the iseries-dk is 2x4x25GbE.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#481-walkthrough-modify-the-ethernet-sub-system-to-1x400gbe","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System to 1x400GbE","text":"OFS provides a preconfigured ofss file so the build script produces a FIM with a 1x400GbE Ethernet subsystem set for 400 GAUI-8. You can build this system with the following:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Build the I-Series FIM with 1x400GbE FIM:
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_1x400_ftile.ofss iseries-dk work_iseries-dk_400\n
The resulting FIM with 1x400GbE has the following MAC settings:
Table: 1x400GbE MAC Settings
MAC Setting Value Client Interface Segmented FEC mode IEEE 802.3 RS(544.514) (CL 134) Auto Negotiation and link training Disabled Maximum Frame Size 1518 You can change the MAC settings by opening the Ethernet Subsystem IP in IP Parameter Editor, update the setting and then save the update as a new preset. You will then edit the ofss_config/hssi/hssi_1x400_ftile.ofss to use the new preset.
The following steps describe the steps to change the 400 GbE MAC frame size to 9600 bytes. Note, the 2x200 and 8x25 GbE MAC implementations can be changed using this process.
-
Invoke IP Parameter editor to make changes to the Ethernet Subsystem IP.
cd $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss\nqsys-edit hssi_ss.ip --quartus-project=$OFS_ROOTDIR/syn/board/iseries-dk/syn_top/ofs_top.qpf\n
-
In IP Parameter editor, load the 400g-fseries-dk preset by selecting and then click Apply
-
In the Device 0 Configuration tab, go to the F-Tile IP Configuration tab and scroll down to P8 MAC Options - P8 Basic and change the TX and RX maximum framesize to 9600.
-
Click New in the Presets panel and in the New Preset pop up window, Name the new preset 400g-fseries-dk-9600 and in File enter $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets/400g-fseries-dk-9600.qprs and click Save
-
Close IP Parameter Editor and do not save changes to hssi_ss.ip. The new preset captured the changes and this new preset will be used in the following updates to re-generate the Ethernet IP subsystem with the updated frame size.
-
Create a new ofss file for the new preset.
cp $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_1x400_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_1x400_ftile_9600.ofss\n
-
Edit $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_1x400_ftile_9600.ofss to use the new 400g-fseries-dk-9600 preset as listed below:
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 1\ndata_rate = 400GAUI-8\npreset = 400g-fseries-dk-9600\n
-
Build the FIM with 9600 byte frame size by using the new hssi_1x400_ftile_9600.ofss file
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_1x400_ftile_9600.ofss iseries-dk work_iseries-dk_400_9600\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the iseries-dk is done by programming a SOF image to the FPGA the embedded USvia JTAG using Quartus Programer.
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"The Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) has an on-board FPGA Download Cable II module which is used to program the FPGA via JTAG.
Perform the following steps to establish a JTAG connection with the iseries-dk.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
Steps:
-
Refer to the following figure showing the location of the on-board Intel FPGA Download Cable II micro USB connector.
-
Verify all switches are set to default as defined in Agilex\u00ae 7 FPGA I-Series Development Kit User Guide.
-
Connect a Micro-USB to USB-A cable between the front panel J8 micro USB port and either the deployment server or an external computer that has Quartus Prime Pro Programming tools installed.
-
Use the jtagconfig tool to check that the JTAG chain contains the AGIB027R29A1E2VR3 device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
1) AGI FPGA Development Kit [1-13]\n034BB0DD AGIB027R29A(.|R2|R3)/..\n 020D10DD VTAP10\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"This walkthrough describes the steps to program the Agilex FPGA on the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile)) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the iseries-dk. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) is connected via JTAG. The Quartus programmer version must be the same as the version of Quartus used to build the design.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device 84:00.0 unplug\n
-
Switch to the machine with JTAG connection to the iseries-dk, if different than your deployment machine.
-
Open the Quartus programmer GUI
Note: the Quartus programmer version must be the same as the version of Quartus used to build the design.
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the iseries-dk.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGIB027R29A1E2VR3 device. The JTAG chain should show the device.
-
Right click the AGIB027R29A1E2VR3 row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGIB027R29A1E2VR3 row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI.
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device 84:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345 Bitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : user\n
"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#appendix","title":"Appendix","text":""},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#appendix-a-resource-utilization-tables","title":"Appendix A: Resource Utilization Tables","text":"Table A-1 Default FIM Resource Utilization with PR (PCIe 1x16)
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 261647.7 28.66 821 6.19 PCIE_RST_CTRL.rst_ctrl 465.6 0.05 0 0.0 afu_top 177715.5 19.47 393 2.96 auto_fab_0 1278.7 0.14 8 0.06 bpf 732.3 0.08 0 0.0 fme_top 625.1 0.07 6 0.05 hssi_wrapper 15867.4 1.74 80 0.6 local_mem_wrapper 10712.7 1.17 60 0.45 ofs_top_auto_tiles 7619.0 0.83 10 0.08 pcie_wrapper 44687.6 4.9 256 1.93 pmci_dummy_csr 682.6 0.07 0 0.0 qsfp_0 629.3 0.07 4 0.03 qsfp_1 629.0 0.07 4 0.03 sys_pll 0.5 0.0 0 0.0 Table A-2 Default FIM Resource Utilization with PR (PCIe 2x8)
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 239719.6 26.26 840 6.33 PCIE_RST_CTRL.rst_ctrl 458.3 0.05 0 0.0 PCIE_RST_CTRL.rst_ctrl 79.0 0.01 0 0.0 afu_top 157249.2 17.23 309 2.33 auto_fab_0 1288.1 0.14 8 0.06 bpf 703.5 0.08 0 0.0 fme_top 626.6 0.07 6 0.05 hssi_wrapper 16164.1 1.77 80 0.6 local_mem_wrapper 10901.9 1.19 60 0.45 ofs_top_auto_tiles 7563.1 0.83 10 0.08 pcie_wrapper 42755.4 4.68 359 2.7 pmci_dummy_csr 676.5 0.07 0 0.0 qsfp_0 627.2 0.07 4 0.03 qsfp_1 624.0 0.07 4 0.03 sys_pll 0.5 0.0 0 0.0 Table A-3 Minimal FIM Resource Utilization (PCIe 2x8, 2PF)
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 122157.7 13.38 598 4.51 PCIE_RST_CTRL.rst_ctrl 366.5 0.04 0 0.0 PCIE_RST_CTRL.rst_ctrl 65.4 0.01 0 0.0 afu_top 62450.5 6.84 190 1.43 auto_fab_0 1273.8 0.14 8 0.06 bpf 835.0 0.09 0 0.0 fme_top 614.6 0.07 6 0.05 hssi_dummy_csr 704.7 0.08 0 0.0 local_mem_wrapper 11229.5 1.23 60 0.45 pcie_wrapper 42541.4 4.66 334 2.52 pmci_dummy_csr 688.1 0.08 0 0.0 qsfp0_dummy_csr 697.4 0.08 0 0.0 qsfp1_dummy_csr 689.8 0.08 0 0.0 sys_pll 0.5 0.0 0 0.0"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/iseries_devkit/dev_guides/fim_dev/ug_ofs_iseries_dk_fim_dev/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/","title":"Getting Started Guide: Open FPGA Stack for Intel Agilex 7 FPGAs Targeting the Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)","text":"Last updated: November 19, 2024
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in evaluating the 2024.2-1 version of the PCIe Attach release targeting the I-Series Development Kit. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Load and verify firmware targeting the FIM and AFU regions of the Agilex FPGA
- Verify full stack functionality offered by the PCIe Attach OFS solution
- Learn where to find additional information on other PCIe Attach ingredients
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell targeting Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile). This platform is a Development Kit intended to be used as a starting point for evaluation and development.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#13-references-and-versions","title":"1.3 References and Versions","text":""},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-2-software-and-component-version-summary-for-ofs-pcie-attach-targeting-the-i-series-development-kit","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach targeting the I-Series Development Kit","text":"The OFS 2024.2-1 PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are used to manually build the Shell and the AFU portion of any potential workloads. Use this section as a general reference for the versions which compose this release. Specific instructions on building the FIM or AFU are discussed in their respective documents.
Component Version Quartus https://www.intel.com/content/www/us/en/software-kit/794624/intel-quartus-prime-pro-edition-design-software-version-24-1-for-linux.html, patches: 0.18, 0.26 Host Operating System https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.10/x86_64/product-software OneAPI-ASP https://github.com/OFS/oneapi-asp/releases/tag/ofs-2024.2-1, patches: 0.02 OFS Platform AFU BBB https://github.com/OFS/ofs-platform-afu-bbb/releases/tag/ofs-2024.2-1 OFS FIM Common Resources https://github.com/OFS/ofs-fim-common/releases/tag/ofs-2024.2-1 AFU Examples https://github.com/OFS/examples-afu/releases/tag/ofs-2024.2-1 OPAE-SIM https://github.com/OPAE/opae-sim"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-3-programmable-firmware-version-summary-for-ofs-pcie-attach-targeting-the-i-series-development-kit","title":"Table 3: Programmable Firmware Version Summary for OFS PCIe Attach Targeting the I-Series Development Kit","text":"OFS releases include pre-built binaries for the FPGA, OPAE SDK and Linux DFL which can be programmed out-of-box (OOB) and include known identifiers shown below. Installation of artifacts provided with this release will be discussed in their relevant sections.
Component Version Link FIM (shell) Pr Interface ID: b7855572-6ca9-58b8-bd11-44e1f1ab329f https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Host OPAE SDK https://github.com/OPAE/opae-sdk, tag: 2.13.0-3 https://github.com/OFS/opae-sdk/releases/tag/2.13.0-3 Host Linux Backport DFL Drivers https://github.com/OPAE/linux-dfl, tag: intel-1.11.0-2 https://github.com/OFS/linux-dfl-backport/releases/tag/intel-1.11.0-2"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-4-hardware-bkc-for-ofs-pcie-attach","title":"Table 4: Hardware BKC for OFS PCIe Attach","text":"The following table highlights the hardware which composes the Best Known Configuration (BKC) for the OFS 2024.2-1 PCIe Attach release. The Intel FPGA Download Cable II is not required when using the I Series Dev Kit, as the device has an on-board blaster.
Component Link Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/agilex/agi027.html (optional) Intel FPGA Download Cable II https://www.intel.com/content/www/us/en/products/sku/215664/intel-fpga-download-cable-ii/specifications.html"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#14-board-installation-and-server-settings","title":"1.4 Board Installation and Server Settings","text":"Instructions detailing the board installation guidelines for an I-Series Dev Kit including server BIOS settings and regulatory information can be found in the [Board Installation Guide: OFS for Agilex\u00ae 7 PCIe Attach Development Kits]. This document also covers the installation of a JTAG cable, which is required for shell programming.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#15-reference-documents","title":"1.5 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.1-1/.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#16-i-series-development-kit-jtag-driver-setup","title":"1.6 I-Series Development Kit JTAG Driver Setup","text":"A specific JTAG driver needs to be installed on the host OS. Follow the instructions under the driver setup for Red Hat 5+ on Intel\u00ae FPGA Download Cable (formerly USB-Blaster) Driver for Linux*.
View the JTAG Chain after installing the proper driver and Quartus Programmer.
cd ~/intelFPGA_pro/quartus/bin\n./jtagconfig -D\n1) AGI FPGA Development Kit [1-13]\n(JTAG Server Version 23.3.0 Build 104 09/20/2023 SC Pro Edition)\n034BB0DD AGIB027R29A(.|R2|R3)/.. (IR=10)\n020D10DD VTAP10 (IR=10)\nDesign hash 27AA3E0B7CE0A5B9F366\n + Node 08586E00 (110:11) #0\n+ Node 0C006E00 JTAG UART #0\n+ Node 19104600 Nios II #0\n+ Node 30006E02 Signal Tap #2\n+ Node 30006E01 Signal Tap #1\n+ Node 30006E00 Signal Tap #0\nCaptured DR after reset = (0069761BB020D10DD) [65]\nCaptured IR after reset = (000D55) [21]\nCaptured Bypass after reset = (2) [3]\nCaptured Bypass chain = (0) [3]\nJTAG clock speed auto-adjustment is enabled. To disable, set JtagClockAutoAdjust parameter to 0\nJTAG clock speed 24 MHz\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#17-upgrading-the-i-series-development-kit-fim-via-jtag","title":"1.7 Upgrading the I-Series Development Kit FIM via JTAG","text":"Intel provides a pre-built FIM that can be used out-of-box for platform bring-up. This shell design is available on the OFS 2024.2-1 Release Page. After programming the shell and installing both the OPAE SDK and Backport Linux DFL kernel drivers as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach, you can confirm the correct FIM has been configured by checking the output of fpgainfo fme against the following table:
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-5-fim-version","title":"Table 5: FIM Version","text":"Identifier Value Pr Interface ID b7855572-6ca9-58b8-bd11-44e1f1ab329f Bitstream ID 360571656856467345 -
Download and unpack the artifacts from the 2024.2-1 release page. The file ofs_top_hps.sof is the base OFS FIM file. This file is loaded into the FPGA using the development kit built in USB Blaster. Please be aware this FPGA is not loaded into non-volatile storage, therefore if the server is power cycled, you will need to reload the FPGA .sof file.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/iseries-dk-images_ofs-2024-2-1.tar.gz\ntar xf iseries-dk-images.tar.gz\ncd iseries-dk-images/\n
-
Remove the card from the PCIe bus to prevent a surprise link down. This step is not necessary if no image is currently loaded.
sudo pci_device <PCIe BDF> unplug\n
-
Start the Quartus Prime Programmer GUI interface, quartus_pgmw &, located in the bin directory of your Quartus installation. Select \"Hardware Setup\", double click the AGI FPGA Development Kit hardware item and change the hardware frequency to 16MHz.
-
On the home screen select \"Auto Detect\" and accept the default device.
-
Left click on the line for device AGIB027R29A (the Agilex device) and hit \"Change File\". Load ofs_top.sof from the artifacts directory and check \"Program / Configure\". Hit start.
-
Re-add the card to the PCIe bus. This step is not necessary if no image was loaded beforehand.
sudo pci_device <BDF> plug\n
-
If this is the first time you've loaded an image into the board, you will need to restart (warm boot) the server (not power cycle / cold boot).
-
Verify the PR Interface ID for your image matches expectation. When loading a FIM from the pre-compiled binary included in the artifacts archive, this ID will match the one listed in Table 4.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
OFS is a hardware and software infrastructure that provides an efficient approach to developing a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM), or shell of the FPGA provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The OFS architecture for Intel Agilex 7 FPGA provides modularity, configurability, and scalability. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem - a hierarchical design that targets the P-tile PCIe hard IP and is configured to support bifurcated Gen 5 speeds
- Ethernet Subsystem - provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
- Memory Subsystem - 2 x 8 GB DDR4 DIMMs, supporting 2666 MHz speeds, 64-bit width (no ECC)
- Reset Controller
- FPGA Management Engine - Provides a way to manage the platform and enable acceleration functions on the platform.
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration. Each feature of the FME exposes itself to the kernel-level OFS drivers on the host through a Device Feature Header (DFH) register that is placed at the beginning of Control Status Register (CSR) space. Only one PCIe link can access the FME register space in a multi-host channel design architecture at a time.
Note: For more information on the FIM and its external connections, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required to support partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Like the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your entire AFU resides in a partial reconfiguration region of the FPGA.
- The AFU is part of the static region and is compiled as a flat design.
- Your AFU contains both static and PR regions.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components: ST2MM module, Virtio LB stub, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), Memory Host Exerciser (HE-MEM), Traffic Generator to memory (HE-MEM-TG), Port Gasket (PRG) and HPS Copy Engine.
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Basic HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
The AFU has the option to consume native packets from the host or interface channels or to instantiate a shim provided by the Platform Interface Manager (PIM) to translate between protocols.
Note: For more information on the Platform Interface Manager and AFU development and testing, refer to the Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#22-ofs-software-overview","title":"2.2 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the [Software Reference Manual: Open FPGA Stack], on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS Backport DFL driver software provides the bottom-most API to FPGA platforms for this release. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on the wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The Backport DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
You can also build and install the software stack yourself from source as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You may choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
Instructions on building and installing the OPAE SDK from source can be found in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#41-opae-tools-overview","title":"4.1 OPAE Tools Overview","text":"The following section offers a brief introduction including expected output values for the utilities included with OPAE. A full explanation of each command with a description of its syntax is available in the opae-sdk GitHub repo.
A list of all tools included in the OPAE SDK release can be found on the OPAE FPGA Tools tab of ofs.github.io.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#411-board-management-with-fpgainfo","title":"4.1.1 Board Management with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files. As this release targets a development kit platform, it does not have access to an on-board BMC. Subcommands that target specific BMC functionality (such as reading temeletry) are not supported for this release.
Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Note: Your Bitstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo. As the I-Series Development Kit does not contain a traditional BMC as used by other OFS products, those lines in fpgainfo's output will not return valid objects. The subcommand fpgainfo bmc will likewise fail to report telemetry data.
Intel Acceleration JTAG PCI Development Kit\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : b7855572-6ca9-58b8-bd11-44e1f1ab329f\nBoot Page : N/A\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#412-updating-with-fpgasupdate","title":"4.1.2 Updating with fpgasupdate","text":"The fpgasupdate tool is used to program AFU workloads into an open slot in a FIM. The fpgasupdate tool only accepts images that have been formatted using PACsign.
As the I-Series Development Kit does not contain a traditional BMC, you do not have access to a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL. Only the programming of a GBS workload is supported for this release.
The process of programming a SOF with a new FIM version is shown in section 1.5 Upgrading the I-Series Development Kit FIM via JTAG
sudo fpgasupdate ofs_pr_afu.gbs <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:42:31.58] [INFO ] updating from file ofs_pr_afu.gbs with size 19928064 [2022-04-14 16:42:31.60] [INFO ] waiting for idle [2022-04-14 16:42:31.60] [INFO ] preparing image file [2022-04-14 16:42:38.61] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] [2022-04-14 16:42:54.63] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] [2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:49:11.03] [INFO ] Secure update OK [2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#413-verify-fme-interrupts-with-hello_events","title":"4.1.3 Verify FME Interrupts with hello_events","text":"The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors.
Sample output from sudo hello_events.
sudo hello_events\nWaiting for interrupts now...\ninjecting error\nFME Interrupt occurred\nSuccessfully tested Register/Unregister for FME events!\nclearing error\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#414-host-exerciser-modules","title":"4.1.4 Host Exerciser Modules","text":"The reference FIM and unchanged FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. There are three HEMs present in the Intel OFS Reference FIM - HE-LPBK, HE-MEM, and HE-HSSI. These exercisers are tied to three different VFs that must be enabled before they can be used. Execution of these exercisers requires you bind specific VF endpoint to vfio-pci. The host-side software looks for these endpoints to grab the correct FPGA resource.
Refer to the Intel Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for a full description of these modules.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-6-module-pfvf-mappings","title":"Table 6: Module PF/VF Mappings","text":"Module PF/VF ST2MM PF0 HE-MEM PF0-VF0 HE-HSSI PF0-VF1 HE-MEM_TG PF0-VF2 HE-LB Stub PF1-VF0 HE-LB PF2 VirtIO LB Stub PF3 HPS Copy Engine PF4"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#4141-he-mem-he-lb","title":"4.1.4.1 HE-MEM / HE-LB","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: - Latency (AFU to Host memory read) - MMIO latency (Write+Read) - MMIO BW (64B MMIO writes) - BW (Read/Write, Read only, Wr only)
Host Exerciser Loopback Memory (HE-MEM) AFU is used to exercise use of FPGA connected DDR, data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host.
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. When using the I-Series Development Kit SmartNIC Platform, optimal performance requires the exercisers be run at 400 MHz.
Execution of these exercisers requires you to bind specific VF endpoint to vfio-pci. The following commands will bind the correct endpoint for a device with B/D/F 0000:b1:00.0 and run through a basic loopback test.
Note: While running the opae.io init command listed below, the command has failed if no output is present after completion. Double check that Intel VT-D and IOMMU have been enabled in the kernel as discussed in section 3.0 OFS DFL Kernel Drivers.
sudo pci_device 0000:b1:00.0 vf 3\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci Binding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci iommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188 Assigning /dev/vfio/188 to user Changing permissions for /dev/vfio/188 to rw-rw----\n\n$ sudo host_exerciser --clock-mhz 400 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 128 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2166\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 12.103 GB/s\n Test lpbk(1): PASS\n
The following example will run a loopback throughput test using one cache line per request.
sudo pci_device 0000:b1:00.0 vf 3\nsudo opae.io init -d 0000:b1:00.2 user:user\n\nsudo host_exerciser --clock-mhz 400 --mode trput --cls cl_1 lpbk\n starting test run, count of 1\nAPI version: 4\nBus width: 128 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 256\nHost Exerciser numWrites: 257\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2127\nTotal number of Reads sent: 256\nTotal number of Writes sent: 256\nBandwidth: 12.325 GB/s\n Test lpbk(1): PASS\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#4142-traffic-generator-afu-test-application","title":"4.1.4.2 Traffic Generator AFU Test Application","text":"Beginning in OPAE version 2.0.11-1+ the TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank. The supported arguments for test configuration are:
- Number of test loops: --loops
- Number of read transfers per test loop: -r,--read
- Number of write transfers per test loop: -w,--write
- Burst size of each transfer: -b,--bls
- Address stride between each transfer: --stride
- Target memory TG: -m,--mem-channel
Below are some example commands for how to execute the test application. To run the preconfigured write/read traffic test on channel 0:
mem_tg tg_test\n[2024-06-27 21:07:11.753] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 0:\nTG PASS\nMem Clock Cycles: 117\nDEBUG: wcnt_ 1\nDEBUG: rcnt_ 1\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1\nDEBUG: num_ticks 117\nWrite BW: 0.164103 GB/s\nRead BW: 0.164103 GB/s\n\nThread on channel 0 exited with status 0\n[2024-06-27 21:07:11.754] [tg_test] [info] Test tg_test(1): PASS\n
Target channel 1 with a 1MB single-word write only test for 1000 iterations
mem_tg --loops 1000 -r 0 -w 2000 -m 1 tg_test\n[2024-06-27 21:07:28.601] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 2116468\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 0\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1000\nDEBUG: num_ticks 2116468\nWrite BW: 18.1434 GB/s\nRead BW: 0 GB/s\n\nThread on channel 1 exited with status 0\n[2024-06-27 21:07:28.608] [tg_test] [info] Test tg_test(1): PASS\n
Target channel 2 with 4MB write/read test of max burst length for 10 iterations
mem_tg --loops 10 -r 8 -w 8 --bls 255 -m 2 tg_test\n[2024-06-27 21:07:41.537] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 2:\nTG PASS\nMem Clock Cycles: 43398\nDEBUG: wcnt_ 8\nDEBUG: rcnt_ 8\nDEBUG: bcnt_ 255\nDEBUG: loop_ 10\nDEBUG: num_ticks 43398\nWrite BW: 9.0253 GB/s\nRead BW: 9.0253 GB/s\n\nThread on channel 2 exited with status 0\n[2024-06-27 21:07:41.539] [tg_test] [info] Test tg_test(1): PASS\n
sudo mem_tg --loops 1000 -r 2000 -w 2000 --stride 2 --bls 2 -m 1 tg_test\n[2024-06-27 21:07:54.841] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 8508637\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 2000\nDEBUG: bcnt_ 2\nDEBUG: loop_ 1000\nDEBUG: num_ticks 8508637\nWrite BW: 9.02612 GB/s\nRead BW: 9.02612 GB/s\n\nThread on channel 1 exited with status 0\n[2024-06-27 21:07:54.867] [tg_test] [info] Test tg_test(1): PASS\n
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#414-he-hssi","title":"4.1.4 HE-HSSI","text":"HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic.
The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode- specific options, and displays the ethernet statistics by invoking ethtool --statistics INTERFACE.
sudo fpgainfo phy b1:00.0\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102025B1C68BA\nBitstream Version : 5.0.1\nPr Interface Id : e5ee3be6-1b34-5c65-923b-84cf652e6b93\nQSFP0 : Not Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
The following example walks through the process of binding the VF corresponding with the HE-HSSI exerciser to vfio-pci, sending traffic, and verifying that traffic was received.
"},{"location":"hw/iseries_devkit/user_guides/ug_qs_ofs_iseries/ug_qs_ofs_iseries/#table-7-accelerator-pfvf-and-guid-mappings","title":"Table 7: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID I Series Dev Kit base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to user\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to user\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
-
Check Ethernet PHY settings with fpgainfo.
sudo fpgainfo phy -B 0xb1 Intel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xF100000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102025B1C68BA\nBitstream Version : 5.0.1\nPr Interface Id : e5ee3be6-1b34-5c65-923b-84cf652e6b93\nQSFP0 : Not Connected\nQSFP1 : Not Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
-
Set loopback mode.
sudo hssiloopback --loopback enable --pcie-address 0000:b1:00.0 args Namespace(loopback='enable', ncsi_ch_sel=None, pcie_address='0000:84:00.0', port=0)\nsbdf: 0000:b1:00.0\nFPGA dev: {'segment': 0, 'bus': 132, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:b1:00.0'}\nargs.hssi_grps[('dfl_dev.3', ['/sys/bus/pci/devices/0000:b1:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.3/uio/uio0'], '0000:b1:00.0', 21)]\nfpga uio dev:dfl_dev.3\n\n--------HSSI INFO START-------\nDFH :0x3000000010003015\nHSSI ID :0x15\nDFHv :0.5\nguidl :0x99a078ad18418b9d\nguidh :0x4118a7cbd9db4a9b\nHSSI version :2.0\nFirmware Version :16\nHSSI num ports :8\nPort0 :25GbE\nPort1 :25GbE\nPort2 :25GbE\nPort3 :25GbE\nPort4 :25GbE\nPort5 :25GbE\nPort6 :25GbE\nPort7 :25GbE\n--------HSSI INFO END-------\n\nhssi loopback enabled to port0\n[DCPsupport@AN-R760-1 ~]$ sudo fpgainfo phy 0xb1\nIntel Acceleration JTAG PCI Development Kit\n//****** PHY ******//\nInterface : DFL\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:b1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x0001\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x50102025B1C68BA\nBitstream Version : 5.0.1\nPr Interface Id : 79a90b4b-b308-55d0-961e-a882ef571b2c\nQSFP0 : Not Connected\nQSFP1 : Connected\n//****** HSSI information ******//\nHSSI version : 2.0\nNumber of ports : 8\nPort0 :25GbE UP\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
-
Send traffic through the 10G AFU.
10G loopback test\nTx/Rx port: 99\nTx port: 0\nRx port: 0\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth:\n\nNo eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.\n0x40000 ETH_AFU_DFH: 0x1000010000001000\n0x40008 ETH_AFU_ID_L: 0xbb370242ac130002\n0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea\n0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000\n0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000\n0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000\n0x40048 AFU_SCRATCHPAD: 0x0000000045324511\n\n0x3c00 number_packets: 0x00000064\n0x3c01 random_length: 0x00000000\n0x3c02 random_payload: 0x00000000\n0x3c03 start: 0x00000000\n0x3c04 stop: 0x00000000\n0x3c05 source_addr0: 0x44332211\n0x3c06 source_addr1: 0x00006655\n0x3c07 dest_addr0: 0xaa998877\n0x3c08 dest_addr1: 0x0000ccbb\n0x3c09 packet_tx_count: 0x00000064\n0x3c0a rnd_seed0: 0x5eed0000\n0x3c0b rnd_seed1: 0x5eed0001\n0x3c0c rnd_seed2: 0x00025eed\n0x3c0d pkt_length: 0x00000040\n0x3cf4 tx_sta_tstamp: 0x02c1a766\n0x3cf5 tx_end_tstamp: 0x02c1abd6\n0x3d00 num_pkt: 0xffffffff\n0x3d01 pkt_good: 0x00000063\n0x3d02 pkt_bad: 0x00000000\n0x3d07 avst_rx_err: 0x00000000\n0x3d0b rx_sta_tstamp: 0x02c1a85d\n0x3d0c rx_end_tstamp: 0x02c1acd2\n0x3e00 mac_loop: 0x00000000\n\nHSSI performance:\n Selected clock frequency : 402.832 MHz\n Latency minimum : 613.159 ns\n Latency maximum : 625.571 ns\n Achieved Tx throughput : 15.8863 GB/s\n Achieved Rx throughput : 15.8167 GB/s\n
The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. hssi_loopback tests both external and internal loopbacks.
The hssistats tool provides the MAC statistics.
"},{"location":"hw/n6001/","title":"Index","text":"Repository folder for Agilex OFS PCIe Attach collateral targeting Intel FPGA SmartNIC N6001-PL.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/","title":"Shell Developer Guide for Open FPGA Stack: Intel\u00ae FPGA SmartNIC N6000-PL / Intel\u00ae FPGA SmartNIC N6001-PL PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1-introduction","title":"1. Introduction","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#11-about-this-document","title":"1.1. About This Document","text":"This document serves as a guide for OFS Agilex PCIe Attach developers targeting the Intel\u00ae FPGA SmartNIC N6000-PL and Intel\u00ae FPGA SmartNIC N6001-PL. The following topics are covered in this guide:
- Compiling the OFS Agilex PCIe Attach FIM design
- Simulating the OFS Agilex PCIe Attach design
- Customizing the OFS Agilex PCIe Attach FIM design
- Configuring the FPGA with an OFS Agilex PCIe Attach FIM design
The FIM Development Walkthroughs Table lists all of the walkthroughs provided in this guide. These walkthroughs provide step-by-step instructions for performing different FIM Development tasks.
Table: FIM Development Walkthroughs
Walkthrough Name Category Install Quartus Prime Pro Software Setup Clone FIM Repository Setup Set Development Environment Variables Setup Set Up Development Environment Setup Compile OFS FIM Compilation Manually Generate OFS Out-Of-Tree PR FIM Compilation Change the Compilation Seed Compilation Run Individual Unit Level Simulation Simulation Run Regression Unit Level Simulation Simulation Add a new module to the OFS FIM Customization Modify and run unit tests for a FIM that has a new module Customization Modify and run UVM tests for a FIM that has a new module Customization Hardware test a FIM that has a new module Customization Debug the FIM with Signal Tap Customization Compile the FIM in preparation for designing your AFU Customization Resize the Partial Reconfiguration Region Customization Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Customization Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Customization Create a Minimal FIM Customization Migrate to a Different Agilex Device Number Customization Modify the Memory Sub-System Using IP Presets With OFSS Customization Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Customization Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Customization Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Customization Modify the Ethernet Sub-System Without HSSI OFSS Customization Remove the HPS Customization Set up JTAG Configuration Program the FPGA via JTAG Configuration Program the FPGA via RSU Configuration"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#111-knowledge-pre-requisites","title":"1.1.1 Knowledge Pre-Requisites","text":"It is recommended that you have the following knowledge and skills before using this developer guide.
- Basic understanding of OFS and the difference between OFS designs. Refer to the OFS Welcome Page.
- Review the release notes for the Agilex\u00ae 7 PCIe Attach Reference Shells, with careful consideration of the Known Issues.
- Review of Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL).
- FPGA compilation flows using Quartus\u00ae Prime Pro Edition Software.
- Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus\u00ae Prime Pro Edition Software, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
- RTL (System Verilog) and coding practices to create synthesized logic.
- RTL simulation tools.
- Quartus\u00ae Prime Pro Edition Software Signal Tap Logic Analyzer tool software.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#12-fim-development-theory","title":"1.2. FIM Development Theory","text":"This section will help you understand how the OFS Agilex PCIe Attach FIM can be developed to fit your design goals.
The Default FIM Features section provides general information about the default features of the OFS Agilex PCIe Attach FIM so you can become familiar with the default design. For more detailed information about the FIM architecture, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
The Customization Options section then gives suggestions of how this default design can be customized. Step-by-step walkthroughs for many of the suggested customizations are later described in the FIM Customization section.
FIM development for a new acceleration card generally consists of the following steps:
- Install OFS and familiarize yourself with provided scripts and source code
- Develop high level design with your specific functionality
- Determine requirements and key performance metrics
- Select IP cores
- Select FPGA device
- Develop software memory map
- Select and implement FIM Physical interfaces including:
- External clock sources and creation of internal PLL clocks
- General I/O
- Ethernet modules
- External memories
- FPGA programming methodology
- Develop device physical implementation
- FPGA device pin assignment
- Create logic lock regions
- Create of timing constraints
- Create Quartus Prime Pro FIM test project and validate:
- Placement
- Timing constraints
- Build script process
- Review test FIM FPGA resource usage
- Select FIM to AFU interfaces and development of PIM
- Implement FIM design
- Develop RTL
- Instantiate IPs
- Develop test AFU to validate FIM
- Develop unit and device level simulation
- Develop timing constraints and build scripts
- Perform timing closure and build validation
- Create FIM documentation to support AFU development and synthesis
- Software Device Feature discovery
- Integrate, validate, and debug hardware/software
- Prepare for high volume production
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#121-default-fim-features","title":"1.2.1 Default FIM Features","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1211-top-level","title":"1.2.1.1 Top Level","text":"The top level block diagram for the OFS Agilex PCIe Attach reference design is shown below.
Figure: OFS Agilex PCIe Attach FIM Top-Level Diagram
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1212-interfaces","title":"1.2.1.2 Interfaces","text":"The key interfaces in the OFS Agilex PCIe Attach design are listed in the Release Capabilities tables below. They describe the capabilities of the n6001 and n6000 hardware as well as the capabilities of the default OFS Agilex PCIe Attach reference designs targeting the n6001 and the n6000.
Table: Intel\u00ae FPGA SmartNIC N6001-PL OFS Release Capabilities
Interface Intel\u00ae FPGA SmartNIC N6001-PL Hardware Capabilities n6001 OFS Agilex PCIe Attach Reference Design Implementation Host Interface PCIe Gen4x16 PCIe Gen4x16 Network Interface 2 x QSFP-28/56 cages 2x4x25GbE | 2x1x100GbE | 2x4x10GbE External Memory 5xDDR4 DIMMs sockets - 40-bits (1 available for HPS) 4xDDR4 - 2400MHz - 4GB (1Gb x 32) - 32-bits - No ECC1xDDR4 - 2400MHz - 1GB (256Mb x 32 with 256 Mb x8 ECC) - 40-bits - With ECC - For HPS Table: Intel\u00ae FPGA SmartNIC N6000-PL OFS Release Capabilities
Interface Intel\u00ae FPGA SmartNIC N6000-PL Hardware Capabilities n6000 OFS Agilex PCIe Attach Reference Design Implementation Host Interface To Agilex 7: PCIe Gen4x8 To E810: PCIe Gen4x8 To Agilex 7: PCIe Gen4x8 To E810: PCIe Gen4x8 Network Interface 2 x QSFP-28/56 cages 2x2x100GbE External Memory 5xDDR4 DIMMs sockets - 40-bits (1 available for HPS) Not enabled by default, but can support: 4xDDR4 - 2400MHz - 4GB (1Gb x 32) - 32-bits - No ECC1xDDR4 - 2400MHz - 1GB (256Mb x 32 with 256 Mb x8 ECC) - 40-bits - With ECC - For HPS"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1213-subsystems","title":"1.2.1.3 Subsystems","text":"The FIM Subsystems Table describes the Platform Designer IP subsystems used in the OFS Agilex PCIe Attach n6001 FIM.
Table: FIM Subsystems
Subsystem User Guide Document ID PCIe Subsystem AXI Streaming IP for PCI Express User Guide 790711 Memory Subsystem Memory Subsystem Intel FPGA IP User Guide for Intel Agilex OFS 686148[1] Ethernet Subsystem Ethernet Subsystem Intel FPGA IP User Guide 773413[1] [1] You must request entitled access to these documents.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1214-host-exercisers","title":"1.2.1.4 Host Exercisers","text":"The default AFU workload in the OFS Agilex PCIe Attach n6001/n6000 FIM contains several modules called Host Exercisers which are used to exercise the interfaces on the board. The Host Exerciser Descriptions Table describes these modules.
Table: Host Exerciser Descriptions
Name Acronym Description OPAE Command Host Exerciser Loopback HE-LB Used to exercise and characterize host to FPGA data transfer. host_exerciser Host Exerciser Memory HE_MEM Used to exercise and characterize host to Memory data transfer. host_exerciser Host Exerciser Memory Traffic Generator HE_MEM_TG Used to exercise and test available memory channels with a configurable traffic pattern. mem_tg Host Exerciser High Speed Serial Interface HE-HSSI n6001: Used to exercise and characterize HSSI interfaces. n6000: Used to pass through data between the E810 to the QSFPs. hssi The host exercisers can be removed from the design at compile-time using command line arguments for the build script.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1215-module-access-via-apfbpf","title":"1.2.1.5 Module Access via APF/BPF","text":"The OFS Agilex PCIe Attach n6001 FIM uses AXI4-Lite interconnect logic named the AFU Peripheral Fabric (APF) and Board Peripheral Fabric (BPF) to access the registers of the various modules in the design. The APF/BPF modules define master/slave interactions, namely between the host software and AFU and board peripherals. The APF Address Map Table describes the address mapping of the APF, followed by the BPF Address Map Table which describes the address mapping of the BPF.
Table: APF Address Map
Address Size (Bytes) Feature 0x00000\u20130x3FFFF 256K Board Peripherals (See BPF Address Map table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket:4K= PR Gasket DFH, control and status4K= Port DFH4K=User Clock52K=Remote STP 0x80000 \u2013 0x80FFF 4K AFU Error Reporting Table: BPF Address Map
Address Size (Bytes) Feature 0x00000 - 0x0FFFF 64K FME 0x10000 - 0x10FFF 4K PCIe 0x11000 - 0x11FFF 4K Reserved 0x12000 - 0x12FFF 4K QSFP0 0x13000 - 0x13FFF 4K QSFP1 0x14000 - 0x14FFF 4K HSSI 0x15000 - 0x15FFF 4K EMIF 0x20000 - 0x3FFFF 128K PMCI Controller"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#122-customization-options","title":"1.2.2 Customization Options","text":"OFS is designed to be easily customizable to meet your design needs. The OFS FIM Customization Examples Table lists the general user flows for OFS Agilex PCIe Attach n6001/n6000 FIM development, along with example customizations for each user flow, plus links to step-by-step walkthroughs where available.
Table: OFS FIM Customization Examples
Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Modify and run UVM tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Memory Sub-System Using IP Presets With OFSS Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Modify the Ethernet Sub-System Without HSSI OFSS Remove the HPS"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#13-development-environment","title":"1.3 Development Environment","text":"This section describes the components required for OFS FIM development, and provides a walkthrough for setting up the environment on your development machine.
Note that your development machine may be different than your deployment machine where the FPGA acceleration card is installed. FPGA development work and deployment work can be performed either on the same machine, or on different machines as desired. Please see the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up the environment for deployment machines.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#131-development-tools","title":"1.3.1 Development Tools","text":"The Development Environment Table describes the Best Known Configuration (BKC) for the tools that are required for OFS FIM development.
Table: Development Environment BKC
Component Version Installation Walkthrough Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 N/A Quartus\u00ae Prime Pro Edition Software Quartus Prime Pro Version 24.1 for Linux + Patches 0.18, 0.26 Section 1.3.1.1 Python 3.6.8 or later N/A GCC 8.5.0 or later N/A cmake 3.15 or later N/A FIM Source Files ofs-2024.2-1 Section 1.3.2.1"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1311-walkthrough-install-quartus-prime-pro-software","title":"1.3.1.1 Walkthrough: Install Quartus Prime Pro Software","text":"Intel Quartus Prime Pro Version 24.1 is verified to work with the latest OFS release ofs-2024.2-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.
Use RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 for compatibility with your development flow and also testing your FIM design in your platform.
Prior to installing Quartus:
-
Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel\u00ae recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.
-
Perform the following steps to satisfy the required dependencies.
$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs
Apply the following configurations.
$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 $ sudo ln -s /usr/bin/python3 /usr/bin/python\n
-
Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.
The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
-
Download your required Quartus Prime Pro Linux version here.
-
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.18, 0.26.
-
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
export PATH=<Quartus install directory>/quartus/bin:$PATH\nexport PATH=<Quartus install directory>/qsys/bin:$PATH\n
For example, if the Quartus install directory is /home/intelFPGA_pro/24.1 then the new line is:
export PATH=/home/intelFPGA_pro/24.1/quartus/bin:$PATH\nexport PATH=/home/intelFPGA_pro/24.1/qsys/bin:$PATH\n
-
Verify, Quartus is discoverable by opening a new shell:
$ which quartus\n/home/intelFPGA_pro/24.1/quartus/bin/quartus\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#132-fim-source-files","title":"1.3.2 FIM Source Files","text":"The source files for the OFS Agilex PCIe Attach FIM are provided in the following repository: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1.
Some essential directories in the repository are described as follows:
ofs-agx7-pcie-attach\n| syn // Contains files related to synthesis\n| | board // Contains synthesis files for several cards, including the n6001 and n6000\n| | | n6001 // Contains synthesis files for n6001 and n6000\n| | | | setup // Contains setup files, including pin constraints and location constraints\n| | | | syn_top // Contains Quartus project files\n| verification // Contains files for UVM testing\n| ipss // Contains files for IP Sub-Systems\n| | qsfp // Contains source files for QSFP Sub-System\n| | hssi // Contains source files for HSSI Sub-System\n| | pmci // Contains source files for PMCI Sub-System (not used in F-Tile FIM)\n| | pcie // Contains source files for PCIe Sub-System\n| | mem // Contains source files for Memory Sub-System\n| sim // Contains simulation files\n| | unit_test // Contains files for all unit tests\n| | | scripts // Contains script to run regression unit tests\n| license // Contains Quartus patch\n| ofs-common // Contains files which are common across OFS platforms\n| | verification // Contains common UVM files\n| | scripts // Contains common scripts\n| | | common\n| | | | syn // Contains common scripts for synthesis, including build script\n| | | | sim // Contains common scripts for simulation\n| | tools // Contains common tools files\n| | | mk_csr_module // Contains common files for CSR modules\n| | | fabric_generation // Contains common files for APF/BPF fabric generation\n| | | ofss_config // Contains common files for OFSS configuration tool\n| | | | ip_params // Contains default IP parameters for certain Sub-Systems when using OFSS\n| | src // Contains common source files, including host exercisers\n| tools //\n| | ofss_config // Contains top level OFSS files for each pre-made board configuration\n| | | hssi // Contains OFSS files for Ethernet-SS configuraiton\n| | | memory // Contains OFSS files for Memory-SS configuration\n| | | pcie // Contains OFSS files for PCIe-SS configuration\n| | | iopll // Contains OFSS files for IOPLL configuration\n| src // Contains source files for Agilex PCIe Attach FIM\n| | pd_qsys // Contains source files related to APF/BPF fabric\n| | includes // Contains source file header files\n| | top // Contains top-level source files, including design top module\n| | afu_top // Contains top-level source files for AFU\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1321-walkthrough-clone-fim-repository","title":"1.3.2.1 Walkthrough: Clone FIM Repository","text":"Perform the following steps to clone the OFS Agilex\u00ae 7 PCIe Attach FIM Repository:
-
Create a new directory to use as a clean starting point to store the retrieved files.
mkdir OFS_BUILD_ROOT\ncd OFS_BUILD_ROOT\nexport OFS_BUILD_ROOT=$PWD\n
-
Clone GitHub repository using the HTTPS git method
git clone --recurse-submodules https://github.com/OFS/ofs-agx7-pcie-attach.git\n
-
Check out the correct tag of the repository
cd ofs-agx7-pcie-attach\ngit checkout --recurse-submodules tags/ofs-2024.2-1\n
-
Ensure that ofs-common has been cloned as well
git submodule status\n
Example output:
ofs-common (ofs-2024.2-1)\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#133-environment-variables","title":"1.3.3 Environment Variables","text":"The OFS FIM compilation and simulation scripts require certain environment variables be set prior to execution.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#1331-walkthrough-set-development-environment-variables","title":"1.3.3.1 Walkthrough: Set Development Environment Variables","text":"Perform the following steps to set the required environment variables. These environment variables must be set prior to simulation or compilation tasks so it is recommended that you create a script to set these variables.
-
Navigate to the top level directory of the cloned OFS FIM repository.
cd ofs-agx7-pcie-attach\n
-
Set project variables
# Set OFS Root Directory - e.g. this is the top level directory of the cloned OFS FIM repository\nexport OFS_ROOTDIR=$PWD\n
-
Set variables based on your development environment
# Set proxies if required for your server\nexport http_proxy=<YOUR_HTTP_PROXY>\nexport https_proxy=<YOUR_HTTPS_PROXY>\nexport ftp_proxy=<YOUR_FTP_PROXY>\nexport socks_proxy=<YOUR_SOCKS_PROXY>\nexport no_proxy=<YOUR_NO_PROXY>\n\n# Set Quartus license path\nexport LM_LICENSE_FILE=<YOUR_LM_LICENSE_FILE>\n\n# Set Synopsys License path (if using Synopsys for simulation)\nexport DW_LICENSE_FILE=<YOUR_DW_LICENSE_FILE>\nexport SNPSLMD_LICENSE_FILE=<YOUR_SNPSLMD_LICENSE_FILE>\n\n# Set Quartus Installation Directory - e.g. $QUARTUS_ROOTDIR/bin contains Quartus executables\nexport QUARTUS_ROOTDIR=<YOUR_QUARTUS_INSTALLATION_DIRECTORY>\n\n# Set the Tools Directory - e.g. $TOOLS_LOCATION contains the 'synopsys' directory if you are using Synopsys. Refer to the $VCS_HOME variable for an example.\nexport TOOLS_LOCATION=<YOUR_TOOLS_LOCATION>\n
-
Set generic environment variables
# Set Work directory \nexport WORKDIR=$OFS_ROOTDIR\n# Set Quartus Tools variables\nexport QUARTUS_HOME=$QUARTUS_ROOTDIR\nexport QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR\nexport QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR\nexport QUARTUS_VER_AC=$QUARTUS_ROOTDIR\nexport IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip\nexport IMPORT_IP_ROOTDIR=$IP_ROOTDIR\nexport QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys/bin\n\n# Set Verification Tools variables (if running simulations)\nexport DESIGNWARE_HOME=$TOOLS_LOCATION/synopsys/vip_common/vip_Q-2020.03A\nexport UVM_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.N6001_SIM_VCS_VER_SH }}/linux64/rhel/etc/uvm\nexport VCS_HOME=$TOOLS_LOCATION/synopsys/vcsmx/${{ env.N6001_SIM_VCS_VER_SH }}/linux64/rhel\nexport MTI_HOME=$QUARTUS_ROOTDIR/../questa_fse\nexport VERDIR=$OFS_ROOTDIR/verification\nexport VIPDIR=$VERDIR\n# Set OPAE variables\nexport OPAE_SDK_REPO_BRANCH=release/2.13.0\n\n# Set PATH to include compilation and simulation tools\nexport PATH=$QUARTUS_HOME/bin:$QUARTUS_HOME/../qsys/bin:$QUARTUS_HOME/sopc_builder/bin/:$OFS_ROOTDIR/opae-sdk/install-opae-sdk/bin:$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$DESIGNWARE_HOME/bin:$VCS_HOME/bin:$PATH\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#134-walkthrough-set-up-development-environment","title":"1.3.4 Walkthrough: Set Up Development Environment","text":"This walkthrough guides you through the process of setting up your development environment in preparation for FIM development. This flow only needs to be done once on your development machine.
-
Ensure that Quartus Prime Pro Version 24.1 for Linux with Agilex\u00ae 7 FPGA device support is installed on your development machine. Refer to the Install Quartus Prime Pro Software section for step-by-step installation instructions.
- Verify version number
quartus_sh --version\n
Example Output:
Quartus Prime Shell\nVersion 24.1 SC Pro Edition\n
-
Ensure that all support tools are installed on your development machine, and that they meet the version requirements.
-
Python 3.6.8 or later
-
Verify version number
python --version\n
Example Output:
Python 3.6.8\n
-
GCC 8.5.0 or later
-
Verify version number
gcc --version\n
Example output:
gcc (GCC) 8.5.0\n
-
cmake 3.15 or later
-
Verify version number
cmake --version\n
Example output:
cmake version 3.15\n
-
Clone the ofs-agx7-pcie-attach repository. Refer to the Clone FIM Repository section for step-by-step instructions.
-
Install UART IP license patch .02.
-
Navigate to the license directory
cd $IOFS_BUILD_ROOT/license\n
-
Install Patch 0.02
sudo ./quartus-0.0-0.02iofs-linux.run\n
-
Install Quartus Patches 0.18, 0.26. All required patches are provided in the Assets of the OFS FIM Release: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1
-
Extract and unzip the patch-agx7-2024-2-1.tar.gz file.
tar -xvzf patch-agx7-2024-2-1.tar.gz\n
-
Run each patch .run file. As an example:
sudo ./quartus-24.1-0.18-linux.run\n
-
Verify that patches have been installed correctly. They should be listed in the output of the following command.
quartus_sh --version\n
-
Set required environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
This concludes the walkthrough for setting up your development environment. At this point you are ready to begin FIM development.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2-fim-compilation","title":"2. FIM Compilation","text":"This section describes the process of compiling OFS FIM designs using the provided build scripts. It contains two main sections:
- Compilation Theory - Describes the theory behind FIM compilation
- Compilation Flows - Describes the process of compiling a FIM
The walkthroughs provided in this section are:
- Compile OFS FIM
- Manually Generate OFS Out-Of-Tree PR FIM
- Change the Compilation Seed
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#21-compilation-theory","title":"2.1 Compilation Theory","text":"This section describes the theory behind FIM compilation.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#211-fim-build-script","title":"2.1.1 FIM Build Script","text":"The OFS Common Repository contains a script named build_top.sh which is used to build OFS FIM designs and generate output files that can be programmed to the board. After cloning the OFS FIM repository (with the ofs-common repository included), the build script can be found in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/syn/build_top.sh\n
The usage of the build_top.sh script is as follows:
build_top.sh [-k] [-p] [-e] [--stage=<action>] [--ofss=<ip_config>] <build_target>[:<fim_options>] [<work_dir_name>]\n
Field Options Description Requirement -k None Keep. Preserves and rebuilds within an existing work tree instead of overwriting it. Optional -p None When set, and if the FIM supports partial reconfiguration, a PR template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable and uses only relative paths. See $OFS_ROOTDIR/syn/common/scripts generate_pr_release.sh for details. Optional -e None Run only Quartus analysis and elaboration. It completes the setup stage, passes -end synthesis to the Quartus compilation flow and exits without running the finish stage. Optional --stage all | setup | compile | finish Controls which portion of the OFS build is run.\u00a0\u00a0- all: Run all build stages (default)\u00a0\u00a0- setup: Initialize a project in the work directory\u00a0\u00a0- compile: Run the Quartus compilation flow on a project that was already initialized with setup\u00a0\u00a0- finish: Complete OFS post-compilation tasks, such as generating flash images and, if -p is set, generating a release. Optional --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. This parameter is consumed during the setup stage and IP is updated only inside the work tree. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6000 | n6001 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> flat | null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg | no_hssi Used to change how the FIM is built.\u00a0\u00a0\u2022 flat - Compiles a flat design (no PR assignments). This is useful for bringing up the design on a new board without dealing with PR complexity.\u00a0\u00a0\u2022 null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0\u2022 null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0\u2022 null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0\u2022 null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. \u00a0\u00a0\u2022 no_hssi - Removes the HSSI-SS from the FIM. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:flat,null_he_lb,null_he_hssi Optional <work_dir_name> String Specifies the name of the work directory in which the FIM will be built. If not specified, the default target is $OFS_ROOTDIR/work Optional Refer to Compile OFS FIM which provides step-by-step instructions for running the build_top.sh script with some of the different available options.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2111-build-work-directory","title":"2.1.1.1 Build Work Directory","text":"The build script copies source files from the existing cloned repository into the specified work directory, which are then used for compilation. As such, any changes made in the base source files will be included in all subsequent builds, unless the -k option is used, in which case an existing work directories files are used as-is. Likewise, any changes made in a work directory is only applied to that work directory, and will not be updated in the base repository by default. When using OFSS files to modify the design, the build script will create a work directory and make the modifications in the work directory.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2112-null-host-exercisers","title":"2.1.1.2 Null Host Exercisers","text":"When using the he_null_x command command line options, the specified Host Exerciser is replaced by an he_null block. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#212-ofss-file-usage","title":"2.1.2 OFSS File Usage","text":"The OFS FIM build script uses OFSS files to configure the design IP prior to compilation using preset configurations. The OFSS files specify certain parameters for different IPs. Using OFSS is provided as a convenience feature for building different FIM configurations. You can specify the IP OFSS files you wish to use on the command line, by editing the default Platform OFSS file, or by creating a custom Platform OFSS file and calling it on the command line. Any IP OFSS file type not explicitly specified will default to the one defined in the default Platform OFSS file.
The following video describes OFS Settings files, and provides demonstrations showing how to easily customize the OFS reference shell designs and accelerate your development flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2121-top-level-ofss-file","title":"2.1.2.1 Top Level OFSS File","text":"Top-level OFSS files are OFSS files that contain a list of IP OFSS files that will be used during compilation when the Top-level OFSS file is provided to the build script. You may make your own custom Top-level OFSS files for convenient compilation. The Provided Top-level OFSS Files table describes the Top-level OFSS files that are provided to you.
Top-level OFSS files contain a [default] header, followed by all of the IP OFSS files that will be used by the build script when this Platform OFSS file is called. Ensure that any environment variables (e.g. $OFS_ROOTDIR) are set correctly. The OFSS Config tool uses breadth first search to include all of the specified OFSS files; the ordering of OFSS files does not matter.
The general structure of a Top-level OFSS file is as follows:
[default]\n<PATH_TO_BASE_OFSS_FILE>\n<PATH_TO_PCIE_OFSS_FILE>\n<PATH_TO_IOPLL_OFSS_FILE>\n<PATH_TO_MEMORY_OFSS_FILE>\n<PATH_TO_HSSI_OFSS_FILE>\n
Any IP OFSS file types that are not explicitly defined by the user will default to using the IP OFSS files specified in the default Top-level OFSS file of the target board. The default Top-level OFSS file for each target is /tools/ofss_config/<target_board>.ofss. You can use the --ofss nodefault option to prevent the build script from using the default Top-level OFSS file. You can still provide other OFSS files while using the nodefault option, e.g. --ofss nodefault tools/ofss_config/pcie/pcie_host_2link.ofss will implement the settings within pcie_host_2link.ofss, and will not use any default settings for the other IP types.
Table: Provided Top-Level OFSS Files
OFSS File Name Location Type Description Supported Board n6001.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6001. Includes the following OFSS files: \u00a0\u00a0\u2022 n6001_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory.ofss \u00a0\u00a0\u2022 hssi_8x25.ofss N6001 n6000.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for N6000. Includes the following OFSS files: \u00a0\u00a0\u2022 n6000_base.ofss \u00a0\u00a0\u2022 pcie_host_n6000.ofss \u00a0\u00a0\u2022 iopll_350MHz.ofss \u00a0\u00a0\u2022 hssi_4x100.ofss N6000 fseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for fseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 fseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_ftile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss fseries-dk iseries-dk.ofss $OFS_ROOTDIR/tools/ofss_config Top-level This is the default for iseries-dk. Includes the following OFSS files: \u00a0\u00a0\u2022 iseries-dk_base.ofss \u00a0\u00a0\u2022 pcie_host.ofss \u00a0\u00a0\u2022 iopll_500MHz.ofss \u00a0\u00a0\u2022 memory_rtile.ofss \u00a0\u00a0\u2022 hssi_8x25_ftile.ofss iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2122-base-ofss-file","title":"2.1.2.2 Base OFSS File","text":"An OFSS file with IP type ofs contains board specific information for the target board. It defines certain attributes of the design, including the platform name, device family, fim type, part number, and device ID. It can also contain settings for system information like PCIe generation and subsystem device IDs. Note that PCIe settings defined in the PCIe OFSS file will take precedence over any PCIe settings defined in the Base OFSS file.
Currently supported configuration options for an OFSS file with IP type ofs are described in the OFS IP OFSS File Options table.
Table: OFS IP OFSS File Options
Section Parameter n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type ofs ofs ofs ofs [settings] platform n6001 n6000 n6001 n6001 family agilex agilex agilex agilex fim base_x16 base_x16 base_x16 base_x16 part AGFB014R24A2E2V AGFB014R24A2E2V AGFB027R24C2E2VR2 AGIB027R29A1E2VR3 device_id 6001 6000 6001 6001 [pcie.settings] pcie_gen 4 4 4 5 [pcie] subsys_dev_id 1771 1770 1 1 exvf_subsysid 1771 1770 1 1 The Provided Base OFSS Files table describes the Base OFSS files that are provided to you.
Table: Provided Base OFSS Files
OFSS File Name Location Type Supported Board n6001_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6001 n6000_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs N6000 fseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs fseries-dk iseries-dk_base.ofss $OFS_ROOTDIR/tools/ofss_config/base ofs iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2123-pcie-ofss-file","title":"2.1.2.3 PCIe OFSS File","text":"An OFSS file with IP type pcie is used to configure the PCIe-SS and PF/VF MUX in the FIM.
The PCIe OFSS file has a special section type ([pf*]) which is used to define physical functions (PFs) in the FIM. Each PF has a dedicated section, where the * character is replaced with the PF number. For example, [pf0], [pf1], etc. For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 PF if 1 or more VFs present | 2 PFs if 0 VFs present (PFs must start at PF0) Max # of PFs 8 Min # of VFs 0 VFs if 2 or more PFs present | 1 VF if only 1 PF present Max # of VFs 2000 distributed across all PFs Currently supported configuration options for an OFSS file with IP type pcie are described in the PCIe IP OFSS File Options table.
Table: PCIe IP OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type pcie Specifies that this OFSS file configures the PCIe-SS pcie pcie pcie pcie [settings] output_name pcie_ss Specifies the output name of the PCIe-SS IP pcie_ss pcie_ss pcie_ss pcie_ss ip_component intel_pcie_ss_axi | pcie_ss Specifies the PCIe SS IP that will be used. \u00a0\u00a0\u2022 intel_pcie_ss_axi: AXI Streaming Intel FPGA IP for PCI Express \u00a0\u00a0\u2022 pcie_ss: Intel FPGA IP Subsystem for PCI Express intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi intel_pcie_ss_axi preset String OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. N/A N/A N/A N/A [pf*] num_vfs Integer Specifies the number of Virtual Functions in the current PF Variable[1] Variable[2] Variable[1] Variable[1] bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] bar4_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] vf_bar0_address_width Integer Variable[1] Variable[2] Variable[1] Variable[1] ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] vf_ats_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] prs_ext_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pasid_cap_enable 0 | 1 Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] pci_type0_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] revision_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] class_code 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_vendor_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] subsys_dev_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] sriov_vf_device_id 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] exvf_subsysid 32'h Value Variable[1] Variable[2] Variable[1] Variable[1] [1] Refer to pcie_host.ofss
[2] Refer to pcie_host_n6000.ofss
Any parameter that is not specified in the PCIe OFSS file will default to the values defined in $OFS_ROOTDIR/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py. When using a PCIe IP OFSS file during compilation, the PCIe-SS IP that is used will be defined based on the values in the PCIe IP OFSS file plus the parameters defined in pcie_ss_component_parameters.py.
The Provided PCIe OFSS Files table describes the PCIe OFSS files that are provided to you.
Table: Provided PCIe OFSS Files
OFSS File Name Location Type Description Supported Boards pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 PF0 (1 VF) N6001 | fseries-dk | iseries-dk pcie_host_2link.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_2link_1pf_1vf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration: \u00a0\u00a0\u2022 2x8 PCIe\u00a0\u00a0\u2022 PF0 (1 VF) iseries-dk pcie_host_2pf.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (0 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs) N6001 | fseries-dk | iseries-dk pcie_host_gen4.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) iseries-dk pcie_host_n6000.ofss $OFS_ROOTDIR/tools/ofss_config/pcie pcie Defines the PCIe Subsystem for the N6000 with the following configuration:\u00a0\u00a0\u2022 PF0 (3 VFs)\u00a0\u00a0\u2022 PF1 (0 VFs)\u00a0\u00a0\u2022 PF2 (0 VFs)\u00a0\u00a0\u2022 PF3 (0 VFs)\u00a0\u00a0\u2022 PF4 (0 VFs) N6001"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2124-iopll-ofss-file","title":"2.1.2.4 IOPLL OFSS File","text":"An OFSS file with IP type iopll is used to configure the IOPLL in the FIM.
The IOPLL OFSS file has a special section type ([p_clk]) which is used to define the IOPLL clock frequency.
Currently supported configuration options for an OFSS file with IP type iopll are described in the IOPLL OFSS File Options table.
Table: IOPLL OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type iopll Specifies that this OFSS file configures the IOPLL iopll iopll iopll iopll [settings] output_name sys_pll Specifies the output name of the IOPLL. sys_pll sys_pll sys_pll sys_pll instance_name iopll_0 Specifies the instance name of the IOPLL. iopll_0 iopll_0 iopll_0 iopll_0 [p_clk] freq Integer: 250 - 500 Specifies the IOPLL clock frequency in MHz. 500 350 500 500 The Provided IOPLL OFSS Files table describes the IOPLL OFSS files that are provided to you.
Table: Provided IOPLL OFSS Files
OFSS File Name Location Type Description Supported Board iopll_500MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 500 MHz N6001 | fseries-dk | iseries-dk iopll_470MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 470 MHz N6001 | fseries-dk | iseries-dk iopll_350MHz.ofss $OFS_ROOTDIR/tools/ofss_config/iopll iopll Sets the IOPLL frequency to 350 MHz N6001 | N6000 | fseries-dk | iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2125-memory-ofss-file","title":"2.1.2.5 Memory OFSS File","text":"An OFSS file with IP type memory is used to configure the Memory-SS in the FIM.
The Memory OFSS file specifies a preset value, which selects a presets file (.qprs) to configure the Memory-SS.
Currently supported configuration options for an OFSS file with IP type memory are described in the Memory OFSS File Options table.
Table: Memory OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type memory Specifies that this OFSS file configures the Memory-SS memory N/A memory memory [settings] output_name mem_ss_fm Specifies the output name of the Memory-SS. mem_ss_fm N/A mem_ss_fm mem_ss_fm preset String[1] Specifies the name of the .qprs presets file that will be used to build the Memory-SS. n6001 N/A fseries-dk iseries-dk [1] You may generate your own .qprs presets file with a unique name using Quartus.
Memory-SS presets files are stored in the $OFS_ROOTDIR/ipss/mem/qip/presets directory.
The Provided Memory OFSS Files table describes the Memory OFSS files that are provided to you.
Table: Provided Memory OFSS Files
OFSS File Name Location Type Description Supported Board memory.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as: N6001 | N6000 [1] memory_ftile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as fseries-dk fseries-dk memory_rtile.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk memory_rtile_no_dimm.ofss $OFS_ROOTDIR/tools/ofss_config/memory memory Defines the memory IP preset file to be used during the build as iseries-dk iseries-dk [1] The memory.ofss file can be used for the N6000, however, the default N6000 FIM does not implement the Memory Sub-system. Refer to Section 4.7.2 for step-by-step instructions on how to enable the Memory sub-system
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2126-hssi-ip-ofss-file","title":"2.1.2.6 HSSI IP OFSS File","text":"An OFSS file with IP type hssi is used to configure the Ethernet-SS in the FIM.
Currently supported configuration options for an OFSS file with IP type hssi are described in the HSSI OFSS File Options table.
Table: HSSI OFSS File Options
Section Parameter Options Description n6001 Default Value n6000 Default Value fseries-dk Default Value iseries-dk Default Value [ip] type hssi Specifies that this OFSS file configures the Ethernet-SS hssi hssi hssi hssi [settings] output_name hssi_ss Specifies the output name of the Ethernet-SS hssi_ss hssi_ss hssi_ss hssi_ss num_channels Integer Specifies the number of channels. 8 4 8 8 data_rate 10GbE | 25GbE | 100GCAUI-4 | 200GAUI-4 | 400GAUI-8 Specifies the data rate[1] 25GbE 100GCAUI-4 25GbE 25GbE preset None | fseries-dk | 200g-fseries-dk | 400g-fseries-dk | String[1] OPTIONAL - Selects the platform whose preset .qprs file will be used to build the Ethernet-SS. When used, this will overwrite the other settings in this OFSS file. N/A N/A N/A N/A [1] The presets file will take priority over the data_rate parameter, so this value will not take effect if using a presets file.
[2] You may generate your own .qprs presets file with a unique name using Quartus.
Ethernet-SS presets are stored in $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets directory.
The Provided HSSI OFSS Files table describes the HSSI OFSS files that are provided to you.
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#213-ofs-build-script-outputs","title":"2.1.3 OFS Build Script Outputs","text":"The output files resulting from running the the OFS FIM build_top.sh build script are copied to a single directory during the finish stage of the build script. The path for this directory is:
- For N6001:
$OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/n6001/syn_top/output_files - For N6000:
$OFS_ROOTDIR/<WORK_DIRECTORY>/syn/board/n6000/syn_top/output_files
The output files include programmable images and compilation reports. The OFS Build Script Output Descriptions table describes the images that are generated by the build script.
Table: OFS Build Script Output Descriptions
File Description ofs_top[_hps].bin This is an intermediate, raw binary file. This intermediate raw binary file is produced by taking the Quartus generated .sof file, and converting it to *.pof using quartus_pfg, then converting the *.pof to *.hexout using quartus_cpf, and finally converting the *.hexout to *.bin using objcopy. Depending on whether the FPGA design contains an HPS block, a different file will be generated. **ofs_top.bin* - Raw binary image of the FPGA generated if there is no HPS present in the design. ofs_top_hps.bin - Raw binary image of the FPGA generated if there is an HPS present in the design. ofs_top_page1.bin This is the binary of the Factory Image and is the input to PACSign utility to generate ofs_top_page1_unsigned.bin binary image file. This image will carry binary content for the HPS if it is included in the SOF image. ofs_top_page0_factory.bin This is an input file to PACSign to generate ofs_top_page0_unsigned_factory.bin. ofs_top_page0_unsigned_factory.bin This is the unsigned PACSign output generated for the Factory Image. ofs_top_page1_user1.bin This is an input file to PACSign to generate ofs_top_page1_unsigned_user1.bin. This file is created by taking the ofs_top_[hps].bin file and assigning the User1 or appending factory block information. ofs_top_page1_unsigned_user1.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User1 Image. This file is used to load the FPGA flash User1 Image using the fpgasupdate tool. ofs_top_page2_user2.bin This is an input file to PACSign to generate ofs_top_page2_unsigned_user2.bin. This file is created by taking the ofs_top_[hps].bin file and assigning the User2 or appending factory block information. ofs_top_page2_unsigned_user2.bin This is the unsigned FPGA binary image generated by the PACSign utility for the User2 Image. This file is used to load the FPGA flash User2 Image using the fpgasupdate tool. ofs_top_hps.sof If your design contains an Agilex\u00ae 7 FPGA Hard Processor System, then the build assembly process combines the FPGA ofs_top.sof programming file with u-boot-spl-dtb.hex to produce this file."},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#22-compilation-flows","title":"2.2 Compilation Flows","text":"This section provides information for using the build script to generate different FIM types. Walkthroughs are provided for each compilation flow. These walkthroughs require that the development environment has been set up as described in the Set Up Development Environment section.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#221-flat-fim","title":"2.2.1 Flat FIM","text":"A flat FIM is compiled such that there is no partial reconfiguration region, and the entire design is built as a flat design. This is useful for compiling new designs without worrying about the complexity introduced by partial reconfiguration. The flat compile removes the PR region and PR IP; thus, you cannot use the -p build flag when using the flat compile setting. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#222-in-tree-pr-fim","title":"2.2.2 In-Tree PR FIM","text":"An In-Tree PR FIM is the default compilation if no compile flags or compile settings are used. This flow will compile the design with the partial reconfiguration region, but it will not create a relocatable PR directory tree to aid in AFU development. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#223-out-of-tree-pr-fim","title":"2.2.3 Out-of-Tree PR FIM","text":"An Out-of-Tree PR FIM will compile the design with the partial reconfiguration region, and will create a relocatable PR directory tree to aid in AFU workload development. This is especially useful if you are developing a FIM to be used by another team developing AFU workloads. This is the recommended build flow in most cases. There are two ways to create the relocatable PR directory tree:
- Run the FIM build script with the
-p option. Refer to the Compile OFS FIM Section for step-by-step instructions for this flow. - Run the
generate_pr_release.sh script after running the FIM build script. Refer to the Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM Section step-by-step instructions for this flow.
In both cases, the generate_pr_release.sh is run to create the relocatable build tree. This script is located at $OFS_ROOTDIR/ofs-common/scripts/common/syn/generate_pr_release.sh. Usage for this script is as follows:
generate_pr_release.sh -t <tgt dir> [-f] <build target> <work dir name>\n
The Generate PR Release Script Options table describes the options for the generate_pr_release.sh script.
Table: Generate PR Release Script Options
Parameter Options Description <tgt dir> String Specifies the location of the relocatable PR directory tree to be created. [-f] - If exists, the script will abort unless -f is specified. <build target> n6001 | n6000 | fseries-dk | iseries-dk Specifies the name of the board target. <work dir name> String Specifies the existing work directory from which the relocatable PR directory tree will be created from. After generating the relocatable build tree, it is located in the $OFS_ROOTDIR/<WORK_DIRECTORY>/pr_build_template directory (or the directory you specified if generated separately). The contents of this directory have the following structure:
\u251c\u2500\u2500 bin\n\u251c\u2500\u2500 \u251c\u2500\u2500 afu_synth\n\u251c\u2500\u2500 \u251c\u2500\u2500 qar_gen\n\u251c\u2500\u2500 \u251c\u2500\u2500 update_pim\n\u251c\u2500\u2500 \u251c\u2500\u2500 run.sh\n\u251c\u2500\u2500 \u251c\u2500\u2500 build_env_config\n\u251c\u2500\u2500 README\n\u251c\u2500\u2500 hw\n\u251c\u2500\u2500 \u251c\u2500\u2500 lib\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 build\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-ifc-id.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 platform\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 fme-platform-class.txt\n\u251c\u2500\u2500 \u251c\u2500\u2500 blue_bits\n\u251c\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top_hps.sof\n\u2514\u2500\u2500 \u251c\u2500\u2500 \u251c\u2500\u2500 ofs_top.sof\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#224-he_null-fim","title":"2.2.4 HE_NULL FIM","text":"An HE_NULL FIM refers to a design with one, some, or all of the Host Exercisers replaced by he_null blocks. The he_null is a minimal block with CSRs that responds to PCIe MMIO requests in order to keep PCIe alive. You may use any of the build flows (flat, in-tree, out-of-tree) with the HE_NULL compile options. The HE_NULL compile options are as follows:
null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_nullnull_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_nullnull_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_nullnull_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null
The Compile OFS FIM section gives step-by-step instructions for this flow.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#225-walkthrough-compile-ofs-fim","title":"2.2.5 Walkthrough: Compile OFS FIM","text":"Perform the following steps to compile the OFS Agilex PCIe Attach FIM for n6001 or n6000:
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options. The following command would build the default n6001 design:
./ofs-common/scripts/common/syn/build_top.sh n6001 work_n6001\n
The build command options allow for many modifications to the shell design at build time. The following tool is provided to help you conveniently get the build command for a specific shell configuration:
OFS Build Command Generator Build Flow Options Build Target n6001 n6000 fseries-dk iseries-dk Partial Reconfiguration Settings Disable Partial Reconfiguration Generate Relocatable PR Tree Add/Remove Subsystems Remove HSSI-SS (Ethernet Sub-System) Add/Remove Host Exercisers Remove HE_HSSI (Ethernet Host Exerciser) Remove HE_LBK (PCIe Loopback) Remove HE_MEM (Read/Write Memory Exerciser) Remove HE_MEM_TG (Pseudo random memory traffic generator) IP Configuration HSSI default 8x10 GbE 8x25 GbE 2x100 GbE 2x200 GbE 1x400 GbE IOPLL default 500 MHz 470 MHz 350 MHz PCIe default 1x16 5PF/3VF 1x16 1PF/1VF 1x16 2PF/0VF 2x8 3PF/3VF 2x8 1PF/1VF Gen4 Gen5 Press submit to generate the build command. Note: If no OFSS file is specified, the build script will use the <target>.ofss file by default.
- Once the build script is complete, the build summary should report that the build is complete and passes timing. For example:
***********************************\n***\n*** OFS_PROJECT: n6001\n*** OFS_BOARD: n6001\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#226-walkthrough-manually-generate-ofs-out-of-tree-pr-fim","title":"2.2.6 Walkthrough: Manually Generate OFS Out-Of-Tree PR FIM","text":"This walkthrough describes how to manually generate an Out-Of-Tree PR FIM. This can be automatically done for you if you run the build script with the -p option. This process is not applicable if you run the build script with the flat option.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the root directory.
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the desired compile options using the n6001 OFSS presets. In order to create the relocatable PR tree, you may not compile with the flat option. For example:
-
N6001
./ofs-common/scripts/common/syn/build_top.sh n6001 work_n6001\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh n6000 work_n6000\n
-
Run the generate_pr_release.sh script to create the relocatable PR tree.
-
N6001
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_n6001/pr_build_template n6001 work_n6001\n
-
N6000
./ofs-common/scripts/common/syn/generate_pr_release.sh -t work_n6000/pr_build_template n6000 work_n6000\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#227-compilation-seed","title":"2.2.7 Compilation Seed","text":"You may change the seed which is used by the build script during Quartus compilation to change the starting point of the fitter. Trying different seeds is useful when your design is failing timing by a small amount.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#2271-walkthrough-change-the-compilation-seed","title":"2.2.7.1 Walkthrough: Change the Compilation Seed","text":"Perform the following steps to change the compilation seed for the FIM build.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the SEED assignment in the $OFS_ROOTDIR/syn/board/<board_name>/syn_top/ofs_top.qsf file to your desired seed value. The value can be any non-negative integer value.
For example, for n6001:
vim $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf\n
set_global_assignment -name SEED 1\n
-
Build the FIM. Refer to the Compile OFS FIM section for instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#3-fim-simulation","title":"3. FIM Simulation","text":"Unit level simulation of key components in the FIM is provided for verification of the following areas:
- Ethernet
- PCIe
- External Memory
- Core FIM
The Unit Level simulations work with Synopsys VCS/VCSMX or Mentor Graphics Questasim simulators. The scripts to run each unit level simulation are located in $OFS_ROOTDIR/sim/unit_test. Each unit test directory contains a README which describes the test in detail.
Refer to the Supported Unit Tests table for a list of the supported unit tests.
Table: Supported Unit Tests
Test Name Description n6001 n6000 fseries-dk iseries-dk bfm_test This is the unit test for PCIe BFM. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 csr_test This is the unit test for FIM CSR access and AFU memory write/readThe Verilog macro 'SIM_USE_PCIE_DUMMY_CSR' is enabled to switch to a dummy CSR instance in pcie_top.The dummy CSR implements full RW registers which is useful to test MMIO write/read burst to FIM CSR region.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 MMIO read 32-bit address and 64-bit address (FIM and AFU)\u00a0\u00a0\u2022 Simple memory loopback test using he_lb - this is similar to simple_test_pcie except that it uses a simple pcie BFM \u2713 \u2713 \u2713 \u2713 dfh_walker This is the unit test for FME DFH walking \u2713 \u2713 \u2713 \u2713 flr This is the unit test for PCIe PF/VF FLR.It covers the following test scenarios:\u00a0\u00a0\u2022 PF FLR request and response\u00a0\u00a0\u2022 VF FLR request and response \u2713 \u2713 \u2713 \u2713 fme_csr_access This is the a unit test for the register access logic for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 Ensures CSR registers do not have any unknown \"x\" bits.\u00a0\u00a0\u2022 Checks that CSR register read accesses to not return with any unknown \"x\" bits.\u00a0\u00a0\u2022 Returning read and write AXI responses to CSR register addresses are checked to make sure all return with \"RESP_OKAY\".\u00a0\u00a0\u2022 Checks that all register access types operate correctly:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Lower 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Upper 32-bit read/writes.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2022 Full 64-bit read/writes.\u00a0\u00a0\u2022 Checks all non-CSR reads return with all zeros. \u2713 \u2713 \u2713 \u2713 fme_csr_directed This is the unit test for $OFS_ROOTDIR/ofs-common/src/common/fme/fme_csr.svIt covers the following test scenarios:\u00a0\u00a0\u2022 MMIO reads to FME registers.\u00a0\u00a0\u2022 MMIO writes to FME registers.\u00a0\u00a0\u2022 Test of Register bit attributes.\u00a0\u00a0\u2022 Test of update/status values read from FME inputs through FME registers.\u00a0\u00a0\u2022 Test of update/control values written to FME registers and driven on FME outputs.\u00a0\u00a0\u2022 Test of reads/writes outside of valid register range in valid FME Ranges. \u2713 \u2713 \u2713 \u2713 he_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_mem_lb_test This is the unit test for HE_LPBK. The test uses HE-LB to perform memory loopback between FIM and the host. \u2713 \u2713 \u2713 \u2713 he_null_test This is the unit test for HE-NULL Exerciser. The test issues basic mmio Rd/Wr requests targetting HE-NULL CSRs. \u2713 \u2713 \u2713 \u2713 hssi_csr_test This is the unit test for HE_HSSI/HSSI SS CSR access test \u2713 \u2713 \u2713 \u2713 hssi_kpi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 hssi_test This is the unit test for HE_HSSI/HSSI SS CSR access and HSSI traffic test.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO write 32-bit address and 64-bit address\u00a0\u00a0\u2022 Simple ethernet traffic loopback test using HE_HSSI \u2713 \u2713 \u2713 \u2713 indirect_csr This is the unit test for axi4lite_indirect_csr_if module.It covers the following test scenarios:\u00a0\u00a0\u2022 Indirect CSR write\u00a0\u00a0\u2022 Indirect CSR read \u2713 \u2713 \u2713 \u2713 mem_ss_csr_test This is the unit test for the Mem SS CSRs. It checks the contents of the EMIF DFH and MemSS CSRs and compares them to the expected startup configuration. \u2713 \u2713 \u2713 \u2713 mem_ss_rst_test This is the unit test for the Mem SS reset sequence. It enables the reset port on the Mem SS so that a reset is performed after EMIF initialization/calibration. \u2713 \u2713 \u2713 \u2713 mem_tg_test This is the unit test for HE-MEM Traffic generators. The test exercises MMIO access to the HE-MEM TG AFU at PF2 VF2 and runs the traffic generators to test the memory interface. \u2713 \u2713 \u2713 \u2713 pcie_ats_basic_test This is a basic test of PCIe ATS messages and ATS invalidation handling.PCIe ATS must be enabled in the FIM Quartus project being simulated. If ATS is not enabled the test will pass but do nothing.The FIM has an ATS invalidation handler that generates responses for AFUs that are not holding address translations. The test begins by sending an inval to each AFU in the port gasket and confirms that the FIM responds. It then requests ATS translations on each port and confirms they are successful. After that, more ATS invalidations are sent and the test confirms that the AFUs see them and respond -- not the FIM. \u2713 \u2713 \u2713 \u2713 pcie_csr_test This is the unit test for PCIE CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PCIe CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PCIe CSR region \u2713 \u2713 \u2713 \u2713 pf_vf_access_test This is the unit test for PCIe PF/VF MMIO. Each function has a feature GUID at offset 0x8 with an associated register map. For testing CSR access we only exercise a single 64b scratchpad who's offset is determined from the GUID.It covers the following test scenarios:\u00a0\u00a0\u2022 PF MMIO request and response\u00a0\u00a0\u2022 VF MMIO request and response \u2713 \u2713 \u2713 \u2713 pmci_csr_test This is the unit test for PMCI CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to PMCI CSR\u00a0\u00a0\u2022 MMIO access 32-bit address and 64-bit address to unused PMCI CSR region \u2713 \u2713 pmci_mailbox_test This is the unit test for PMCI M10 accessible registers and RW mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 Accessing PMCI RW mailbox register through SPI loopback \u2713 \u2713 pmci_rd_default_value_test This is the unit test for PMCI Flash Write Read access.It covers the following test scenarios:\u00a0\u00a0\u2022 PMCI Flash Write Read\u00a0\u00a0\u2022 accessing PMCI mailbox register through SPI loopback \u2713 \u2713 pmci_ro_mailbox_test This is the unit test for PMCI RO mailbox registers.It covers the following test scenarios:\u00a0\u00a0\u2022 accessing PMCI RO mailbox register through SPI loopback \u2713 \u2713 pmci_vdm_b2b_drop_err_scenario_test This is the unit test for error testing of MCTP Back to back Drop scenario.It covers the following test scenarios:\u00a0\u00a0\u2022 RX payload will be sent back to back immediately to test this condition.\u00a0\u00a0\u2022 PMCI_SS requires some time to process the previous packets before sending this packet since this criteria is not met it will drop the present packet. \u2713 \u2713 pmci_vdm_len_err_scenario_test This is the unit test for Error scenario testing of MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenario related to length is tested in this testcase. Scenarios include packet length greater than MCTP_BASELINE_MTU, packet length equal to 0. \u2713 \u2713 pmci_vdm_mctp_mmio_b2b_test This is the unit test for MCTP VDM packets and CSR transactions sent back to back.It covers the following test scenarios:\u00a0\u00a0\u2022 MCTP RX transactions are done b2b with CSR transactions. \u2713 \u2713 pmci_vdm_multipkt_error_scenario_test This is the unit test for multipacket error scenarios in case of MCTP VDM messages.It covers the following test scenarios:\u00a0\u00a0\u2022 This testcase covers error scenarios for MCTP VDM multipackets.\u00a0\u00a0\u2022 Various scenarios include\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO EOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Singlepacket with NO SOM\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Middle packet having greater length than the first packet\u00a0\u00a0\u00a0\u00a0\u2022 Multipacket with Last packet having greater lenght than previous packets. \u2713 \u2713 pmci_vdm_multipkt_tlp_err_test This is the unit test for checking Error scnearios in multipacket MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 This test covers certain error scenarios for multipacket VDM messages\u00a0\u00a0\u2022 Error scenarios include:\u00a0\u00a0\u00a0\u00a0\u2022 Multipackets with different deid,seid,tag,pkt_sequence number etc \u2713 \u2713 pmci_vdm_tlp_error_scenario_test This is the unit test for covering certain tlp error for single MCTP VDM packets.It covers the following test scenarios:\u00a0\u00a0\u2022 Error scenarios include invalid tlp fields for DW0,DW1,DW3 like invalid t9,t8,tc,at,ep,attr,MCTP header version ,tag fields,invalid DEID \u2713 \u2713 pmci_vdm_tx_rx_all_random_lpbk_test This testcase is written just to cover certain fields like randomizing seid,msg_tag,target_id etc. It is functionally equivalent to pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_all_toggle_test This testcase is added for improving coverage for MCTP VDM packets TX/RX flow. Functionally same as pmci_vdm_tx_rx_lpbk_test. \u2713 \u2713 pmci_vdm_tx_rx_lpbk_test This is the unit test for MCTP VDM packets TX/RX flowIt covers the following test scenarios:\u00a0\u00a0\u2022 BMC component ( inside Testbench) will intiate a MCTP txn with 16DW in TX path.\u00a0\u00a0\u2022 This MCTP VDM packets will be formed in PMCI_SS and will be sent to PCIe top (through mctp_tx_bridge).\u00a0\u00a0\u2022 These transaction will be looped back at PCIe top (PCIe BFM) and will be sent back in the RX path.\u00a0\u00a0\u2022 RX and TX payload comparison is done at BMC side.\u00a0\u00a0\u2022 BMC->PMCI->PCIe->PMCI->BMC ( flow of packets). \u2713 \u2713 port_gasket_test This is the unit test for pg_csr block and it's connectivity to fabric. The test issues mmio Rd/Wr requests targetting the csrs in port_gasket. This test does not do any functional testing of partial reconfiguration, user clock or remote stp. \u2713 \u2713 \u2713 \u2713 qsfp_test This is the unit test for QSFP contrtoller CSR access.It covers the following test scenarios:\u00a0\u00a0\u2022 MMIO read-write to common csr with 64-bit address \u2713 \u2713 \u2713 \u2713 remote_stp_test This is the unit test for remote stp. It covers mmio read access to remote_stp registers. \u2713 \u2713 \u2713 \u2713 uart_csr This is the unit test for UART CSR accesses. \u2713 \u2713 \u2713 \u2713"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#31-simulation-file-generation","title":"3.1 Simulation File Generation","text":"The simulation files must be generated prior to running unit level simulations. The script to generate simulation files is in the following location:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/gen_sim_files.sh\n
The usage of the gen_sim_files.sh script is as follows:
gen_sim_files.sh [--ofss=<ip_config>] <build_target>[:<fim_options>] [<device>] [<family>]\n
The Gen Sim Files Script Options table describes the options for the gen_sim_files.sh script.
Table: Gen Sim Files Script Options
Field Options Description Requirement --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. Optional <build_target> n6001 | n6000 | fseries-dk | iseries-dk Specifies which board is being targeted. Required <fim_options> null_he_lb | null_he_hssi | null_he_mem | null_he_mem_tg Used to change how the FIM is built.\u00a0\u00a0- null_he_lb - Replaces the Host Exerciser Loopback (HE_LBK) with he_null.\u00a0\u00a0- null_he_hssi - Replaces the Host Exerciser HSSI (HE_HSSI) with he_null.\u00a0\u00a0- null_he_mem - Replaces the Host Exerciser Memory (HE_MEM) with he_null.\u00a0\u00a0- null_he_mem_tg - Replaces the Host Exerciser Memory Traffic Generator with he_null. More than one FIM option may be passed included in the <fim_options> list by concatenating them separated by commas. For example: <build_target>:null_he_lb,null_he_hssi Optional <device> string Specifies the device ID for the target FPGA. If not specified, the default device is parsed from the QSF file for the project. Optional <family> string Specifies the family for the target FPGA. If not specified, the default family is parsed from the QSF file for the project. Optional Refer to the Run Individual Unit Level Simulation section for an example of the simulation files generation flow.
When running regression tests, you may use the -g command line argument to generate simulation files automatically; refer to the Run Regression Unit Level Simulation section for step-by-step instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#32-individual-unit-tests","title":"3.2 Individual Unit Tests","text":"Each unit test may be run individually using the run_sim.sh script located in the following directory:
$OFS_ROOTDIR/ofs-common/scripts/common/sim/run_sim.sh\n
The usage for the run_sim.sh script is as follows:
sh run_sim.sh TEST=<test> [VCSMX=<0|1> | MSIM=<0|1>]\n
The Run Sim Script Options table describes the options for the run_sim.sh script.
Table: Run Sim Script Options
Field Options Description TEST String Specify the name of the test to run, e.g. dfh_walker VCSMX 0 | 1 When set, the VCSMX simulator will be used MSIM 0 | 1 When set, the QuestaSim simulator will be used Note: The default simulator is VCS if neither VCSMX nor MSIM are set.
The log for a unit test is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#321-walkthrough-run-individual-unit-level-simulation","title":"3.2.1 Walkthrough: Run Individual Unit Level Simulation","text":"Perform the following steps to run an individual unit test.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Navigate to the simulation directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Generate the simulation files for the target design.
-
N6001
./gen_sim_files.sh n6000\n
-
N6000
./gen_sim_files.sh n6000\n
-
Navigate to the common simulation directory
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the desired unit test using your desired simulator
-
Using VCS
sh run_sim.sh TEST=<test_name>\n
-
Using VCSMX
sh run_sim.sh TEST=<test_name> VCSMX=1\n
-
Using QuestaSim
sh run_sim.sh TEST=<test_name> MSIM=1\n
-
For example, to run the DFH walker test using VCSMX:
sh run_sim.sh TEST=dfh_walker VCSMX=1\n
-
Once the test is complete, check the output for the simulation results. Review the log for detailed test results.
Example output:
Test status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356233750000\nV C S S i m u l a t i o n R e p o r t\nTime: 356233750000 fs\nCPU Time: 57.730 seconds; Data structure size: 47.2Mb\nTue Sep 5 09:44:19 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#33-regression-unit-tests","title":"3.3 Regression Unit Tests","text":"You may use the regression script regress_run.py to run some or all of the unit tests available with a single command. The regression script is in the following location:
$OFS_ROOTDIR/sim/unit_test/scripts/regress_run.py\n
The usage of the regression script is as follows:
regress_run.py [-h] [-l] [-n <num_procs>] [-k <test_package>] [-s <simulator>] [-g] [--ofss <ip_config>] [-b <board_name>] [-e]\n
The Regression Unit Test Script Options table describes the options for the regress_run.py script.
Table: Regression Unit Test Script Options
Field Options Description -h | --help N/A Show the help message and exit -l | --local N/A Run regression locally -n | --n_procs Integer Maximum number of processes/tests to run in parallel when run locally. This has no effect on farm runs. -k | --pack all | fme | he | hssi | list | mem | pmci Test package to run during regression. The \"list\" option will look for a text file named \"list.txt\" in the \"unit_test\" directory for a text list of tests to run (top directory names). The default test package is all. -s | --sim vcs | vcsmx | msim Specifies the simulator used for the regression tests. The default simulator is vcs -g | --gen_sim_files N/A Generate simulation files. This only needs to be done once per repo update. This is the equivalent of running the gen_sim_files.sh script. -o | --ofss <ip_config>.ofss | nodefault OFS Settings. OFSS files are used to modify IP in the design. More than one .ofss file may be passed to the --ofss switch by concatenating them separated by commas. For example: --ofss config_a.ofss,config_b.ofss. If no OFSS files are provided, the script will default to using the .ofss file to configure the design. You may specify --ofss nodefault to prevent the script from using the default OFSS configuration; the resulting build will only use the source files as-is, plus any OFSS files you specify. -b | --board_name n6001 | n6000 | fseries-dk | iseries-dk Specifies the board target -e | --email_list String Specifies email list to send results to multiple recipients The log for each unit test that is run by the regression script is stored in a transcript file in the simulation directory of the test that was run.
$OFS_ROOTDIR/sim/unit_test/<TEST_NAME>/<SIMULATOR>/transcript\n
For example, the log for the DFH walker test using VCSMX would be found at:
$OFS_ROOTDIR/sim/unit_test/dfh_walker/sim_vcsmx/transcript\n
The simulation waveform database is saved as vcdplus.vpd for post simulation review.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#331-walkthrough-run-regression-unit-level-simulation","title":"3.3.1 Walkthrough: Run Regression Unit Level Simulation","text":"Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a test list file to only run the unit level simulations that are supported for the target boards.
touch $OFS_ROOTDIR/sim/unit_test/list.txt\n
Copy the following list into the new file. You may remove tests from this list as desired.
./bfm_test/set_params.sh\n./csr_test/set_params.sh\n./dfh_walker/set_params.sh\n./flr/set_params.sh\n./fme_csr_access/set_params.sh\n./fme_csr_directed/set_params.sh\n./he_lb_test/set_params.sh\n./he_mem_lb_test/set_params.sh\n./he_null_test/set_params.sh\n./hssi_csr_test/set_params.sh\n./hssi_kpi_test/set_params.sh\n./hssi_test/set_params.sh\n./indirect_csr/set_params.sh\n./mem_ss_csr_test/set_params.sh\n./mem_ss_rst_test/set_params.sh\n./mem_tg_test/set_params.sh\n./pcie_ats_basic_test/set_params.sh\n./pcie_csr_test/set_params.sh\n./pcie_ss_axis_components/set_params.sh\n./pf_vf_access_test/set_params.sh\n./pmci_csr_test/set_params.sh\n./pmci_mailbox_test/set_params.sh\n./pmci_rd_default_value_test/set_params.sh\n./pmci_ro_mailbox_test/set_params.sh\n./pmci_vdm_b2b_drop_err_scenario_test/set_params.sh\n./pmci_vdm_len_err_scenario_test/set_params.sh\n./pmci_vdm_mctp_mmio_b2b_test/set_params.sh\n./pmci_vdm_multipkt_error_scenario_test/set_params.sh\n./pmci_vdm_multipkt_tlp_err_test/set_params.sh\n./pmci_vdm_tlp_error_scenario_test/set_params.sh\n./pmci_vdm_tx_rx_all_random_lpbk_test/set_params.sh\n./pmci_vdm_tx_rx_all_toggle_test/set_params.sh\n./pmci_vdm_tx_rx_lpbk_test/set_params.sh\n./port_gasket_test/set_params.sh\n./qsfp_test/set_params.sh\n./remote_stp_test/set_params.sh\n./uart_csr/set_params.sh\n
-
Navigate to the unit test scripts directory.
cd $OFS_ROOTDIR/sim/unit_test/scripts\n
-
Run regression test with the your desired options. For example, to simulate with the options to generate simulation files, run locally, use 8 processes, run all tests, use VCS simulator, and target the desired board:
-
N6001
python regress_run.py -g -l -n 8 -k list -s vcs -b n6001\n
-
N6000
python regress_run.py -g -l -n 8 -k list -s vcs -b n6000\n
Note: You may run all available tests by using -k all instead of creating and using -k list, however not all tests are supported depending on the target board.
-
Once all tests are complete, check that the tests have passed.
Example output:
Passing Unit Tests:37/37 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n bfm_test:............................. PASS -- Time Elapsed:0:01:09.697671\n csr_test:............................. PASS -- Time Elapsed:0:01:25.629880\n dfh_walker:........................... PASS -- Time Elapsed:0:01:10.426995\n flr:.................................. PASS -- Time Elapsed:0:01:09.624240\n fme_csr_access:....................... PASS -- Time Elapsed:0:00:37.512457\n fme_csr_directed:..................... PASS -- Time Elapsed:0:00:45.458114\n he_lb_test:........................... PASS -- Time Elapsed:0:02:07.375008\n he_mem_lb_test:....................... PASS -- Time Elapsed:0:38:52.066694\n he_null_test:......................... PASS -- Time Elapsed:0:01:04.939156\n hssi_csr_test:........................ PASS -- Time Elapsed:0:06:23.029516\n hssi_kpi_test:........................ PASS -- Time Elapsed:2:29:26.521521\n hssi_test:............................ PASS -- Time Elapsed:2:26:59.511113\n indirect_csr:......................... PASS -- Time Elapsed:0:00:53.041156\n mem_ss_csr_test:...................... PASS -- Time Elapsed:0:18:43.020948\n mem_ss_rst_test:...................... PASS -- Time Elapsed:0:37:35.095748\n mem_tg_test:.......................... PASS -- Time Elapsed:0:25:07.194488\n pcie_ats_basic_test:.................. PASS -- Time Elapsed:0:01:08.777231\n pcie_csr_test:........................ PASS -- Time Elapsed:0:01:10.284642\n pcie_ss_axis_components:.............. PASS -- Time Elapsed:0:02:06.266695\n pf_vf_access_test:.................... PASS -- Time Elapsed:0:01:08.948374\n pmci_csr_test:........................ PASS -- Time Elapsed:0:01:16.863665\n pmci_mailbox_test:.................... PASS -- Time Elapsed:0:02:40.877768\n pmci_rd_default_value_test:........... PASS -- Time Elapsed:0:01:13.951152\n pmci_ro_mailbox_test:................. PASS -- Time Elapsed:0:08:40.948176\n pmci_vdm_b2b_drop_err_scenario_test:.. PASS -- Time Elapsed:0:08:53.753526\n pmci_vdm_len_err_scenario_test:....... PASS -- Time Elapsed:0:14:18.816711\n pmci_vdm_mctp_mmio_b2b_test:.......... PASS -- Time Elapsed:0:01:55.465110\n pmci_vdm_multipkt_error_scenario_test: PASS -- Time Elapsed:1:37:57.808918\n pmci_vdm_multipkt_tlp_err_test:....... PASS -- Time Elapsed:0:20:29.416693\n pmci_vdm_tlp_error_scenario_test:..... PASS -- Time Elapsed:0:19:57.978144\n pmci_vdm_tx_rx_all_random_lpbk_test:.. PASS -- Time Elapsed:0:10:54.756589\n pmci_vdm_tx_rx_all_toggle_test:....... PASS -- Time Elapsed:0:07:54.360796\n pmci_vdm_tx_rx_lpbk_test:............. PASS -- Time Elapsed:0:08:13.492743\n port_gasket_test:..................... PASS -- Time Elapsed:0:01:09.992123\n qsfp_test:............................ PASS -- Time Elapsed:0:05:48.597337\n remote_stp_test:...................... PASS -- Time Elapsed:0:01:12.008778\n uart_csr:............................. PASS -- Time Elapsed:0:01:33.400947\nFailing Unit Tests: 0/37 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n Number of Unit test results captured: 37\nNumber of Unit test results passing.: 37\nNumber of Unit test results failing.: 0\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4-fim-customization","title":"4. FIM Customization","text":"This section describes how to perform specific customizations of the FIM, and provides step-by-step walkthroughs for these customizations. Each walkthrough can be done independently. These walkthroughs require a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment. The FIM Customization Walkthroughs table lists the walkthroughs that are provided in this section. Some walkthroughs include steps for testing on hardware. Testing on hardware requires that you have a deployment environment set up. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
Table: OFS FIM Customization Examples
Walkthrough Name Add a new module to the OFS FIM Modify and run unit tests for a FIM that has a new module Modify and run UVM tests for a FIM that has a new module Hardware test a FIM that has a new module Debug the FIM with Signal Tap Compile the FIM in preparation for designing your AFU Resize the Partial Reconfiguration Region Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS Modify PCIe Sub-System and PF/VF MUX Configuration Using IP Presets Create a Minimal FIM Migrate to a Different Agilex Device Number Modify the Memory Sub-System Using IP Presets With OFSS Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications Modify the Ethernet Sub-System Without HSSI OFSS Remove the HPS"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#41-adding-a-new-module-to-the-fim","title":"4.1 Adding a new module to the FIM","text":"This section provides a information for adding a custom module to the FIM, simulating the new design, compiling the new design, implementing and testing the new design on hardware, and debugging the new design on hardware.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#411-hello-fim-theory-of-operation","title":"4.1.1 Hello FIM Theory of Operation","text":"If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.
See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.
This example will add a hello_fim module to the design. The Hello FIM example adds a simple DFH register and 64bit scratchpad register connected to the Board Peripheral Fabric (BPF) that can be accessed by the Host. You can use this example as the basis for adding a new feature to your FIM.
For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0x100 which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.
The Hello FIM design can be verified by Unit Level simulation, Universal Verification Methodology (UVM) simulation, and running in hardware on the n6001 card. The process for these are described in this section.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4111-hello-fim-board-peripheral-fabric-bpf","title":"4.1.1.1 Hello FIM Board Peripheral Fabric (BPF)","text":"The Hello FIM module will be connected to the Board Peripheral Fabric (BPF), and will be connected such that it can be mastered by the Host. The BPF is an interconnect generated by Platform Designer. The Hello FIM BPF Interface Diagram figure shows the APF/BPF Master/Slave interactions, as well as the added Hello FIM module.
Figure: Hello FIM BPF Interface Diagram
The BPF fabric is defined in $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt.
We will add the Hello FIM module to an un-used address space in the MMIO region. The Hello FIM MMIO Address Layout table below shows the MMIO region for the Host with the Hello FIM module added at base address 0x16000.
Table: Hello FIM MMIO Address Layout
Offset Feature CSR set 0x00000 FME AFU 0x10000 PCIe Interface 0x12000 QSFP Controller 0 0x13000 QSFP Controller 1 0x14000 E-Tile Ethernet Interface 0x15000 EMIF 0x16000 Hello FIM 0x20000 PMCI Controller 0x40000 ST2MM (Streaming to Memory-Mapped) 0x60000 VUART 0x70000 PR Control & Status (Port Gasket) 0x71000 Port CSRs (Port Gasket) 0x72000 User Clock (Port Gasket) 0x74000 Remote SignalTap (Port Gasket) 0x80000 AFU Errors (AFU Interface Handler)"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4112-hello-fim-csr","title":"4.1.1.2 Hello FIM CSR","text":"The Hello FIM CSR will consist of the three registers shown in the Hello FIM CSR table below. The DFH and Hello FIM ID registers are read-only. The Scratchpad register supports read and write accesses.
Table: Hello FIM CSR
Offset Attribute Description Default Value 0x016000 RO DFH(Device Feature Headers) register 0x30000006a0000100 0x016030 RW Scrachpad register 0x0 0x016038 RO Hello FIM ID register 0x6626070150000034"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#412-walkthrough-add-a-new-module-to-the-ofs-fim","title":"4.1.2 Walkthrough: Add a new module to the OFS FIM","text":"Perform the following steps to add a new module to the OFS FIM that can be accessed by the Host.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Make hello_fim source directory
mkdir $OFS_ROOTDIR/src/hello_fim\n
-
Create hello_fim_top.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_top.sv\n
Copy the following code into hello_fim_top.sv:
// ***************************************************************************\n// INTEL CONFIDENTIAL\n//\n// Copyright (C) 2023 Intel Corporation All Rights Reserved.\n//\n// The source code contained or described herein and all documents related to\n// the source code (\"Material\") are owned by Intel Corporation or its\n// suppliers or licensors. Title to the Material remains with Intel\n// Corporation or its suppliers and licensors. The Material contains trade\n// secrets and proprietary and confidential information of Intel or its\n// suppliers and licensors. The Material is protected by worldwide copyright\n// and trade secret laws and treaty provisions. No part of the Material may be\n// used, copied, reproduced, modified, published, uploaded, posted,\n// transmitted, distributed, or disclosed in any way without Intel's prior\n// express written permission.\n//\n// No license under any patent, copyright, trade secret or other intellectual\n// property right is granted to or conferred upon you by disclosure or\n// delivery of the Materials, either expressly, by implication, inducement,\n// estoppel or otherwise. Any license under such intellectual property rights\n// must be express and approved by Intel in writing.\n//\n// You will not, and will not allow any third party to modify, adapt, enhance, \n// disassemble, decompile, reverse engineer, change or create derivative works \n// from the Software except and only to the extent as specifically required by \n// mandatory applicable laws or any applicable third party license terms \n// accompanying the Software.\n//\n// -----------------------------------------------------------------------------\n// Engineer : \n// Create Date : September 2023\n// Module Name : hello_fim_top.sv\n// Project : OFS\n// -----------------------------------------------------------------------------\n//\n// Description: \n// This is a simple module that implements DFH registers and \n// AVMM address decoding logic.\nmodule hello_fim_top #(\nparameter ADDR_WIDTH = 12, parameter DATA_WIDTH = 64, parameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput logic clk,\ninput logic reset,\n// -----------------------------------------------------------\n// AXI4LITE Interface\n// -----------------------------------------------------------\nofs_fim_axi_lite_if.slave csr_lite_if\n);\nimport ofs_fim_cfg_pkg::*;\nimport ofs_csr_pkg::*;\n//-------------------------------------\n// Signals\n//-------------------------------------\nlogic [ADDR_WIDTH-1:0] csr_waddr;\nlogic [DATA_WIDTH-1:0] csr_wdata;\nlogic [DATA_WIDTH/8-1:0] csr_wstrb;\nlogic csr_write;\nlogic csr_slv_wready;\ncsr_access_type_t csr_write_type;\nlogic [ADDR_WIDTH-1:0] csr_raddr;\nlogic csr_read;\nlogic csr_read_32b;\nlogic [DATA_WIDTH-1:0] csr_readdata;\nlogic csr_readdata_valid;\nlogic [ADDR_WIDTH-1:0] csr_addr;\nlogic [63:0] com_csr_writedata;\nlogic com_csr_read;\nlogic com_csr_write;\nlogic [63:0] com_csr_readdata;\nlogic com_csr_readdatavalid;\nlogic [5:0] com_csr_address;\n// AXI-M CSR interfaces\nofs_fim_axi_mmio_if #(\n.AWID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.AWADDR_WIDTH (ADDR_WIDTH),\n.WDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),\n.ARID_WIDTH (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),\n.ARADDR_WIDTH (ADDR_WIDTH),\n.RDATA_WIDTH (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)\n) csr_if();\n// AXI4-lite to AXI-M adapter\naxi_lite2mmio axi_lite2mmio (\n.clk (clk),\n.rst_n (~reset),\n.lite_if (csr_lite_if),\n.mmio_if (csr_if)\n);\n//---------------------------------\n// Map AXI write/read request to CSR write/read,\n// and send the write/read response back\n//---------------------------------\nofs_fim_axi_csr_slave #(\n.ADDR_WIDTH (ADDR_WIDTH),\n.USE_SLV_READY (1'b1)\n) csr_slave (\n.csr_if (csr_if),\n.csr_write (csr_write),\n.csr_waddr (csr_waddr),\n.csr_write_type (csr_write_type),\n.csr_wdata (csr_wdata),\n.csr_wstrb (csr_wstrb),\n.csr_slv_wready (csr_slv_wready),\n.csr_read (csr_read),\n.csr_raddr (csr_raddr),\n.csr_read_32b (csr_read_32b),\n.csr_readdata (csr_readdata),\n.csr_readdata_valid (csr_readdata_valid)\n);\n// Address mapping\nassign csr_addr = csr_write ? csr_waddr : csr_raddr;\nassign com_csr_address = csr_addr[5:0]; // byte address\nassign csr_slv_wready = 1'b1 ;\n// Write data mapping\nassign com_csr_writedata = csr_wdata;\n// Read-Write mapping\nalways_comb\nbegin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\ncasez (csr_addr[11:6])\n6'h00 : begin // Common CSR\ncom_csr_read = csr_read;\ncom_csr_write = csr_write;\nend default: begin\ncom_csr_read = 1'b0;\ncom_csr_write = 1'b0;\nend\nendcase\nend\n// Read data mapping\nalways_comb begin\nif (com_csr_readdatavalid) begin\ncsr_readdata = com_csr_readdata;\ncsr_readdata_valid = 1'b1;\nend\nelse begin\ncsr_readdata = '0;\ncsr_readdata_valid = 1'b0;\nend\nend\nhello_fim_com #(\n.FEAT_ID (FEAT_ID),\n.FEAT_VER (FEAT_VER),\n.NEXT_DFH_OFFSET (NEXT_DFH_OFFSET),\n.END_OF_LIST (END_OF_LIST)\n) hello_fim_com_inst (\n.clk (clk ),\n.reset (reset ),\n.writedata (com_csr_writedata ),\n.read (com_csr_read ),\n.write (com_csr_write ),\n.byteenable (4'hF ),\n.readdata (com_csr_readdata ),\n.readdatavalid (com_csr_readdatavalid ),\n.address (com_csr_address )\n);\nendmodule\n
-
Create hello_fim_com.sv file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_com.sv\n
Copy the following code to hello_fim_com.sv:
module hello_fim_com #(\nparameter bit [11:0] FEAT_ID = 12'h100,\nparameter bit [3:0] FEAT_VER = 4'h0,\nparameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,\nparameter bit END_OF_LIST = 1'b0\n)(\ninput clk,\ninput reset,\ninput [63:0] writedata,\ninput read,\ninput write,\ninput [3:0] byteenable,\noutput reg [63:0] readdata,\noutput reg readdatavalid,\ninput [5:0] address\n);\nwire reset_n = !reset; reg [63:0] rdata_comb;\nreg [63:0] scratch_reg;\nalways @(negedge reset_n ,posedge clk) if (!reset_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];\nalways @(negedge reset_n , posedge clk)\nif (!reset_n) readdatavalid <= 1'b0; else readdatavalid <= read;\nwire wr = write;\nwire re = read;\nwire [5:0] addr = address[5:0];\nwire [63:0] din = writedata [63:0];\nwire wr_scratch_reg = wr & (addr[5:0] == 6'h30)? byteenable[0]:1'b0;\n// 64 bit scratch register\nalways @( negedge reset_n, posedge clk)\nif (!reset_n) begin\nscratch_reg <= 64'h0;\nend\nelse begin\nif (wr_scratch_reg) begin scratch_reg <= din; end\nend\nalways @ (*)\nbegin\nrdata_comb = 64'h0000000000000000;\nif(re) begin\ncase (addr) 6'h00 : begin\nrdata_comb [11:0] = FEAT_ID ; // dfh_feature_id is reserved or a constant value, a read access gives the reset value\nrdata_comb [15:12] = FEAT_VER ; // dfh_feature_rev is reserved or a constant value, a read access gives the reset value\nrdata_comb [39:16] = NEXT_DFH_OFFSET ; // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value\nrdata_comb [40] = END_OF_LIST ; //dfh_end_of_list\nrdata_comb [59:40] = 20'h00000 ; // dfh_rsvd1 is reserved or a constant value, a read access gives the reset value\nrdata_comb [63:60] = 4'h3 ; // dfh_feat_type is reserved or a constant value, a read access gives the reset value\nend\n6'h30 : begin\nrdata_comb [63:0] = scratch_reg; end\n6'h38 : begin\nrdata_comb [63:0] = 64'h6626_0701_5000_0034;\nend\ndefault : begin\nrdata_comb = 64'h0000000000000000;\nend\nendcase\nend\nend\nendmodule\n
-
Create hello_fim_design_files.tcl file.
touch $OFS_ROOTDIR/src/hello_fim/hello_fim_design_files.tcl\n
Copy the following code into hello_fim_design_files.tcl
# Copyright 2023 Intel Corporation.\n#\n# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE\n# COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED\n# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR\n# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n# Hello FIM Files\n#--------------------\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_com.sv\nset_global_assignment -name SYSTEMVERILOG_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_top.sv\n
-
Modify ofs_top.qsf for the target board to include Hello FIM module
-
N6001
vim $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf\n
-
N6000
vim $OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top.qsf\n
Add the following line:
######################################################\n# Verilog Macros\n######################################################\n.....\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HELLO_FIM\" # Includes Hello FIM\n
-
Modify ofs_top_sources.tcl for the target board to include Hello FIM design files
-
N6001
vim $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top_sources.tcl\n
-
N6000
vim $OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top_sources.tcl\n
Add the following line:
############################################\n# Design Files\n############################################\n...\n# Subsystems\n...\nset_global_assignment -name SOURCE_TCL_SCRIPT_FILE $::env(BUILD_ROOT_REL)/src/hello_fim/hello_fim_design_files.tcl\n
-
Modify fabric_design_files.tcl to include BPF Hello FIM Slave IP.
vim $OFS_ROOTDIR/src/pd_qsys/fabric/fabric_design_files.tcl\n
Add the following line:
#--------------------\n# BPF\n#--------------------\n...\nset_global_assignment -name IP_FILE $::env(BUILD_ROOT_REL)/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip\n
-
Modify fabric_width_pkg.sv to add Hello FIM slave information and update EMIF slave next offset.
-
Open fabric_width_pkg.sv
vim $OFS_ROOTDIR/src/includes/fabric_width_pkg.sv\n
-
Add/modify the following lines:
localparam bpf_hello_fim_slv_baseaddress = 'h16000; // New\nlocalparam bpf_hello_fim_slv_address_width = 12; // New\nlocalparam bpf_emif_slv_next_dfh_offset = 'h1000; // Old value: 'hB000\nlocalparam bpf_hello_fim_slv_next_dfh_offset = 'hA000; // New\nlocalparam bpf_hello_fim_slv_eol = 'b0; // New\n
-
Modify top.sv to support the Hello FIM module.
-
Open top.sv
vim $OFS_ROOTDIR/src/top/top.sv\n
-
Add bpf_hello_fim_slv_if to AXI interfaces
// AXI4-lite interfaces\n...\nofs_fim_axi_lite_if #(.AWADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width), .ARADDR_WIDTH(fabric_width_pkg::bpf_hello_fim_slv_address_width)) bpf_hello_fim_slv_if();\n
-
Add Hello FIM instantiation
//*******************************\n// Hello FIM Subsystem\n//*******************************\n`ifdef INCLUDE_HELLO_FIM\nhello_fim_top #(\n.ADDR_WIDTH (fabric_width_pkg::bpf_hello_fim_slv_address_width),\n.DATA_WIDTH (64),\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_top_inst (\n.clk (clk_csr),\n.reset(~rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`else\ndummy_csr #(\n.FEAT_ID (12'h100),\n.FEAT_VER (4'h0),\n.NEXT_DFH_OFFSET (fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset),\n.END_OF_LIST (fabric_width_pkg::bpf_hello_fim_slv_eol)\n) hello_fim_dummy (\n.clk (clk_csr),\n.rst_n (rst_n_csr),\n.csr_lite_if (bpf_hello_fim_slv_if)\n);\n`endif\n
-
Add interfaces for Hello FIM slv to bpf instantiation
bpf bpf (\n...\n.bpf_hello_fim_slv_awaddr (bpf_hello_fim_slv_if.awaddr ),\n.bpf_hello_fim_slv_awprot (bpf_hello_fim_slv_if.awprot ),\n.bpf_hello_fim_slv_awvalid (bpf_hello_fim_slv_if.awvalid ),\n.bpf_hello_fim_slv_awready (bpf_hello_fim_slv_if.awready ),\n.bpf_hello_fim_slv_wdata (bpf_hello_fim_slv_if.wdata ),\n.bpf_hello_fim_slv_wstrb (bpf_hello_fim_slv_if.wstrb ),\n.bpf_hello_fim_slv_wvalid (bpf_hello_fim_slv_if.wvalid ),\n.bpf_hello_fim_slv_wready (bpf_hello_fim_slv_if.wready ),\n.bpf_hello_fim_slv_bresp (bpf_hello_fim_slv_if.bresp ),\n.bpf_hello_fim_slv_bvalid (bpf_hello_fim_slv_if.bvalid ),\n.bpf_hello_fim_slv_bready (bpf_hello_fim_slv_if.bready ),\n.bpf_hello_fim_slv_araddr (bpf_hello_fim_slv_if.araddr ),\n.bpf_hello_fim_slv_arprot (bpf_hello_fim_slv_if.arprot ),\n.bpf_hello_fim_slv_arvalid (bpf_hello_fim_slv_if.arvalid ),\n.bpf_hello_fim_slv_arready (bpf_hello_fim_slv_if.arready ),\n.bpf_hello_fim_slv_rdata (bpf_hello_fim_slv_if.rdata ),\n.bpf_hello_fim_slv_rresp (bpf_hello_fim_slv_if.rresp ),\n.bpf_hello_fim_slv_rvalid (bpf_hello_fim_slv_if.rvalid ),\n.bpf_hello_fim_slv_rready (bpf_hello_fim_slv_if.rready ),\n...\n);\n
-
Modify $OFS_ROOTDIR/src/pd_qsys/fabric/bpf.txt to add the hello_fim module as a slave to the apf.
# NAME FABRIC BASEADDRESS ADDRESS_WIDTH SLAVES\napf mst n/a 18 fme,pcie,pmci,qsfp0,qsfp1,emif,hssi,hello_fim\n...\nhello_fim slv 0x16000 12 n/a\n
-
Execute helper script to generate BPF design files
cd $OFS_ROOTDIR/src/pd_qsys/fabric/\n\nsh gen_fabrics.sh\n
-
Once the script completes, the following new IP is created: $OFS_ROOTDIR/src/pd_qsys/fabric/ip/bpf/bpf_hello_fim_slv.ip.
-
[OPTIONAL] You may verify the BPF changes have been made correctly by opening bpf.qsys to analyze the BPF.
-
Navigate to the $OFS_ROOTDIR/src/pd_qsys/fabric directory.
cd $OFS_ROOTDIR/src/pd_qsys/fabric\n
-
Open the bpf.qsys file.
-
N6001
qsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qpf\n
-
N6000
qsys-edit bpf.qsys --quartus-project=$OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top.qpf\n
-
Find the bpf_hello_fim_slv instance:
-
Compile the Hello FIM design:
-
Return to the OFS root directory.
cd $OFS_ROOTDIR\n
-
Call the build_top.sh script for the target board.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -p n6001 work_n6001_hello_fim\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -p n6000 work_n6000_hello_fim\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#413-walkthrough-modify-and-run-unit-tests-for-a-fim-that-has-a-new-module","title":"4.1.3 Walkthrough: Modify and run unit tests for a FIM that has a new module","text":"Perform the following steps to modify the unit test files to support a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify $OFS_ROOTDIR/sim/unit_test/dfh_walker/test_csr_defs.sv
-
Add HELLO_FIM_IDX entry to t_dfh_idx enumeration.
...\ntypedef enum {\nFME_DFH_IDX,\nTHERM_MNGM_DFH_IDX,\nGLBL_PERF_DFH_IDX,\nGLBL_ERROR_DFH_IDX,\nQSFP0_DFH_IDX,\nQSFP1_DFH_IDX,\nHSSI_DFH_IDX,\nEMIF_DFH_IDX,\nHELLO_FIM_DFH_IDX, // New\nPMCI_DFH_IDX,\nST2MM_DFH_IDX,\nVUART_DFH_IDX,\nPG_PR_DFH_IDX,\nPG_PORT_DFH_IDX,\nPG_USER_CLK_DFH_IDX,\nPG_REMOTE_STP_DFH_IDX,\nAFU_ERR_DFH_IDX,\nMAX_DFH_IDX\n} t_dfh_idx;\n...\n
-
Add HELLO_FIM_DFH to get_dfh_names function.
...\nfunction automatic dfh_name[MAX_DFH_IDX-1:0] get_dfh_names();\n...\ndfh_names[PMCI_DFH_IDX] = \"PMCI_DFH\";\ndfh_names[HELLO_FIM_DFH_IDX] = \"HELLO_FIM_DFH\"; // New\ndfh_names[ST2MM_DFH_IDX] = \"ST2MM_DFH\";\n...\nreturn dfh_names;\n...\n
-
Add expected DFH value for Hello FIM to the get_dfh_values function.
...\nfunction automatic [MAX_DFH_IDX-1:0][63:0] get_dfh_values();\n...\ndfh_values[PMCI_DFH_IDX] = 64'h3_00000_xxxxxx_1012;\ndfh_values[PMCI_DFH_IDX][39:16] = fabric_width_pkg::bpf_pmci_slv_next_dfh_offset;\ndfh_values[HELLO_FIM_DFH_IDX] = 64'h3_00000_xxxxxx_0100; // New\ndfh_values[HELLO_FIM_DFH_IDX][39:16] = fabric_width_pkg::bpf_hello_fim_slv_next_dfh_offset; // New\ndfh_values[ST2MM_DFH_IDX] = 64'h3_00000_xxxxxx_0014;\ndfh_values[ST2MM_DFH_IDX][39:16] = fabric_width_pkg::apf_st2mm_slv_next_dfh_offset;\n...\nreturn dfh_values;\n...\n
-
Generate simulation files:
-
Navigate to the sim directory.
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n
-
Run the gen_sim_files.sh script.
-
N6001
./gen_sim_files.sh n6001\n
-
N6000
./gen_sim_files.sh n6000\n
-
Run DFH Walker Simulation
cd $OFS_ROOTDIR/ofs-common/scripts/common/sim\n\nsh run_sim.sh TEST=dfh_walker\n
-
Verify that the test passes, and that the output shows the Hello FIM in the DFH sequence.
********************************************\n Running TEST(0) : test_dfh_walking\n********************************************\n...\n\nREAD64: address=0x00015000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000010001009\n\nEMIF_DFH\n Address (0x15000)\nDFH value (0x3000000010001009)\nREAD64: address=0x00016000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x30000000a0000100\n\nHELLO_FIM_DFH\n Address (0x16000)\nDFH value (0x30000000a0000100)\nREAD64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0\n** Sending TLP packets **\n ** Waiting for ack **\n READDATA: 0x3000000200001012\n\nPMCI_DFH\n Address (0x20000)\nDFH value (0x3000000200001012)\n...\n\nTest status: OK\n\n********************\n Test summary\n********************\n test_dfh_walking (id=0) - pass\nTest passed!\nAssertion count: 0\n$finish called from file \"/home/ofs-agx7-pcie-attach/sim/unit_test/scripts/../../bfm/rp_bfm_simple/tester.sv\", line 210.\n$finish at simulation time 356791250000\nV C S S i m u l a t i o n R e p o r t\nTime: 356791250000 fs\nCPU Time: 61.560 seconds; Data structure size: 47.4Mb\nTue Aug 15 16:29:45 2023\nrun_sim.sh: USER_DEFINED_SIM_OPTIONS +vcs -l ./transcript\nrun_sim.sh: run_sim.sh DONE!\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#414-walkthrough-modify-and-run-uvm-tests-for-a-fim-that-has-a-new-module","title":"4.1.4 Walkthrough: Modify and run UVM tests for a FIM that has a new module","text":"Perform the following steps to modify the UVM simulation files to support the Hello FIM design.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design in order to simulate.
Steps:
-
Modify $OFS_ROOTDIR/verification/tests/sequences/dfh_walking_seq.svh
-
Modify the dfh_offset_array to insert the Hello FIM at dfh_offset_array[8].
dfh_offset_array = new[17];\ndfh_offset_array[0] = tb_cfg0.PF0_BAR0; // FME_DFH 0x8000_0000\ndfh_offset_array[1] = dfh_offset_array[0] + 64'h0_1000; // THERM_MNGM_DFH 0x8000_1000\ndfh_offset_array[2] = dfh_offset_array[1] + 64'h0_2000; // GLBL_PERF_DFH 0x8000_3000\ndfh_offset_array[3] = dfh_offset_array[2] + 64'h0_1000; // GLBL_ERROR_DFH 0x8000_4000\ndfh_offset_array[4] = dfh_offset_array[3] + 64'h0_E000; // QSFP0_DFH 0x8001_2000\ndfh_offset_array[5] = dfh_offset_array[4] + 64'h0_1000; // QSFP1_DFH 0x8001_3000\ndfh_offset_array[6] = dfh_offset_array[5] + 64'h0_1000; // HSSI_DFH 0x8001_4000\ndfh_offset_array[7] = dfh_offset_array[6] + 64'h0_1000; // EMIF_DFH 0x8001_5000\ndfh_offset_array[8] = dfh_offset_array[7] + 64'h0_1000; // HELLO_FIM_DFH 0x8001_6000\ndfh_offset_array[9] = dfh_offset_array[8] + 64'h0_A000; // PMCI_DFH 0x8002_0000\ndfh_offset_array[10] = dfh_offset_array[9] + 64'h2_0000; // ST2MM_DFH 0x8004_0000\ndfh_offset_array[11] = dfh_offset_array[10] + 64'h2_0000; // VUART_DFH 0x8006_0000\ndfh_offset_array[12] = dfh_offset_array[11] + 64'h1_0000; // PG_PR_DFH_IDX 0x8007_0000\ndfh_offset_array[13] = dfh_offset_array[12] + 64'h0_1000; // PG_PORT_DFH_IDX 0x8007_1000\ndfh_offset_array[14] = dfh_offset_array[13] + 64'h0_1000; // PG_USER_CLK_DFH_IDX 0x8007_2000\ndfh_offset_array[15] = dfh_offset_array[14] + 64'h0_1000; // PG_REMOTE_STP_DFH_IDX 0x8007_3000\ndfh_offset_array[16] = dfh_offset_array[15] + 64'h0_D000; // PG_AFU_ERR_DFH_IDX 0x8008_0000\n
-
Modify `$OFS_ROOTDIR/verification/tests/sequences/mmio_seq.svh``
-
Add test code related to the Hello FIM. This code will verify the scratchpad register at 0x16030 and read only the register at 0x16038.
// HELLO_FIM_Scratchpad 64 bit access\n`uvm_info(get_name(), $psprintf(\"////Accessing PF0 HELLO_FIM_Scratchpad Register %0h+'h16030////\", tb_cfg0.PF0_BAR0), UVM_LOW)\nassert(std::randomize(wdata));\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h30;\nmmio_write64(.addr_(addr), .data_(wdata));\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\naddr = tb_cfg0.PF0_BAR0+'h1_6000+'h38;\nwdata = 64'h6626_0701_5000_0034;\nmmio_read64 (.addr_(addr), .data_(rdata));\nif(wdata !== rdata)\n`uvm_error(get_name(), $psprintf(\"Data mismatch 64! Addr = %0h, Exp = %0h, Act = %0h\", addr, wdata, rdata))\nelse\n`uvm_info(get_name(), $psprintf(\"Data match 64! addr = %0h, data = %0h\", addr, rdata), UVM_LOW)\n
Note: uvm_info and uvm_error statements will put a message into log file.
-
Modify $OFS_ROOTDIR/verification/scripts/Makefile_VCS.mk
-
Add INCLUDE_HELLO_FIM define option to enable Hello FIM on UVM
VLOG_OPT += +define+INCLUDE_HELLO_FIM\n
-
Re-generate the UVM files
-
Navigate to the verification scripts directory
cd $VERDIR/scripts\n
-
Clean the output of previous builds
gmake -f Makefile_VCS.mk clean\n
-
Compile the IP files
-
N6001
gmake -f Makefile_VCS.mk cmplib_adp\n
-
N6000
gmake -f Makefile_VCS.mk cmplib_adp n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when compiling IP files for UVM simulation. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Build the RTL and Test Benches
-
N6001
gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1\n
-
N6000
gmake -f Makefile_VCS.mk build_adp DUMP=1 DEBUG=1 n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when building the RTL and Test Benches for UVM simulation. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Run the UVM DFH Walker Simulation
-
N6001
gmake -f Makefile_VCS.mk run TESTNAME=dfh_walking_test DUMP=1 DEBUG=1\n
-
N6000
gmake -f Makefile_VCS.mk run TESTNAME=dfh_walking_test DUMP=1 DEBUG=1 n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when running UVM simulations. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Verify the DFH Walker test results
-
The output logs are stored in the $VERDIR/sim/dfh_walking_test directory. The main files to note are described in Table 5-3:
Table 5-3 UVM Output Logs
File Name Description runsim.log A log file of UVM trans.log A log file of transactions on PCIe bus inter.vpd A waveform for VCS -
Run the following command to quickly verify that the Hello FIM module was successfully accessed. In the example below, the message DFH offset Match! Exp = 80016000 Act = 80016000 shows that the Hello FIM module was successfully accessed.
cd $VERDIR/sim/dfh_walking_test\ncat runsim.log | grep \"DFH offset\"\n
Expected output:
UVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 111950000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp = 80000000 Act = 80000000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 112586000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80001000 Act = 80001000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113222000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80003000 Act = 80003000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 113858000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80004000 Act = 80004000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 114494000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80012000 Act = 80012000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115147000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80013000 Act = 80013000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 115801000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80014000 Act = 80014000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 116628000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80015000 Act = 80015000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117283000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80016000 Act = 80016000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 117928000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80080000 Act = 80080000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 118594000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80100000 Act = 80100000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119248000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80130000 Act = 80130000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 119854000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80131000 Act = 80131000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 120460000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80132000 Act = 80132000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121065000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80133000 Act = 80133000\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/dfh_walking_seq.svh(73) @ 121672000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] DFH offset Match! Exp= 80140000 Act = 80140000\n
-
Run the UVM MMIO Simulation
-
N6001
gmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1 DEBUG=1\n
-
N6000
gmake -f Makefile_VCS.mk run TESTNAME=mmio_test DUMP=1 DEBUG=1 n6000_100G=1 DISABLE_EMIF=1\n
Note: The default N6000 shell has the Memory Sub-System removed, so the DISABLE_EMIF flag must be set when running UVM simulations. If you add the Memory Sub-System back to the shell design, you must run this command without the DISABLE_EMIF flag.
-
Verify the MMIO test results. Run the following commands to show the result of the scratchpad register and Hello FIM ID register. You can see the \"Data match\" message indicating that the registers are successfuly verified.
cd $VERDIR/sim/mmio_test\ncat runsim.log | grep \"Data\" | grep 1603\n
Expected output:
UVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/mmio_seq.svh(68) @ 115466000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016030, data = 880312f9558c00e1\nUVM_INFO /home/ofs-agx7-pcie-attach/verification/tests/sequences/mmio_seq.svh(76) @ 116112000000: uvm_test_top.tb_env0.v_sequencer@@m_seq [m_seq] Data match 64! addr = 80016038, data = 6626070150000034\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#415-walkthrough-hardware-test-a-fim-that-has-a-new-module","title":"4.1.5 Walkthrough: Hardware test a FIM that has a new module","text":"Perform the following steps to program and hardware test a FIM that has had a new module added to it.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has been generated with a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for generating a Hello FIM design.
Steps:
-
[OPTIONAL] In the work directory where the FIM was compiled, determine the PR Interface ID of your design. You can use this value at the end of the walkthrough to verify that the design has been configured to the FPGA.
-
Navigate to the syn_top directory of the target board in the work directory.
cd $OFS_ROOTDIR/<work_directory>/syn/board/<target_board>/syn_top/\n
As an example:
cd $OFS_ROOTDIR/work_n6001/syn/board/n6001/syn_top/\n
-
View the contents of fme-ifc-id.txt.
cat fme-ifc-id.txt\n
Example output:
a791757d-38a6-5159-a7fc-e1a61157a07b\n
-
Switch to your deployment environment.
-
Program the FPGA with the Hello FIM image. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Run fpgainfo to determine the PCIe B:D.F of your board, and to verify the PR Interface ID matches the ID you found in Step 1.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Initialize opae.io
sudo opae.io init -d <B:D.F>\n
For example:
sudo opae.io init -d 98:00.0\n
-
Run DFH walker. Note the value read back from offset 0x16000 indicates the DFH ID is 0x100 which matches the Hello FIM module.
sudo opae.io walk -d <B:D.F>\n
For example:
sudo opae.io walk -d 98:00.0\n
Example output:
...\noffset: 0x15000, value: 0x3000000010001009\n dfh: id = 0x9, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x16000, value: 0x30000000a0000100\n dfh: id = 0x100, rev = 0x0, next = 0xa000, eol = 0x0, reserved = 0x0, feature_type = 0x3\noffset: 0x20000, value: 0x3000000200001012\n dfh: id = 0x12, rev = 0x1, next = 0x20000, eol = 0x0, reserved = 0x0, feature_type = 0x3\n...\n
-
Read all of the registers in the Hello FIM module
-
Read the DFH Register
opae.io -d 98:00.0 -r 0 peek 0x16000\n
Example Output:
0x30000006a0000100\n
-
Read the Scratchpad Register
opae.io -d 98:00.0 -r 0 peek 0x16030\n
Example Output:
0x0\n
-
Read the ID Register
opae.io -d 98:00.0 -r 0 peek 0x16038\n
Example Output:
0x6626070150000034\n
-
Verify the scratchpad register at 0x16030 by writing and reading back from it.
-
Write to Scratchpad register
opae.io -d 0000:15:00.0 -r 0 poke 0x16030 0x123456789abcdef\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0x123456789abcdef\n
-
Write to Scratchpad register
opae.io -d 15:00.0 -r 0 poke 0x16030 0xfedcba9876543210\n
-
Read from Scratchpad register
opae.io -d 15:00.0 -r 0 peek 0x16030\n
Expected output:
0xfedcba9876543210\n
-
Release the opae.io tool
opae.io release -d 15:00.0\n
-
Confirm the driver has been set back to dfl-pci
opae.io ls\n
Example output:
[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#416-walkthrough-debug-the-fim-with-signal-tap","title":"4.1.6 Walkthrough: Debug the FIM with Signal Tap","text":"The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware that the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough uses a FIM design that has had a Hello FIM module added to it. Refer to the Add a new module to the OFS FIM section for step-by-step instructions for creating a Hello FIM design. You do not need to compile the design.
Perform the following steps in your development environment:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Synthesize the design using the -e build script option. You may skip this step if you are using a pre-synthesized design.
- Navigate to the OFS root directory
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script for the target board with the -e option.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -e n6001 work_hello_fim_with_stp\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -e n6000 work_hello_fim_with_stp\n
-
Open the design in Quartus. The Compilation Dashboard should show that the Analysis & Elaboration step has completed.
-
N6001
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/n6001/syn_top/ofs_top.qpf\n
-
N6000
quartus $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/n6000/syn_top/ofs_top.qpf\n
Note: If a full compile has been done in the work directory you are using, you may need to switch from the ofs_pr_afu view to the ofs_top view.
-
Open Tools -> Signal Tap Logic Analyzer
-
Select the Default template and click Create
-
Assign the clock for sampling the Signal Tap instrumented signals of interest. Note that the clock selected should correspond to the signals you want to view for best trace fidelity. Different clocks can be used, however, there maybe issues with trace inaccuracy due to sampling time differences. This example instruments the hello_fim_top module previously intetegrated into the FIM. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the block of interest and open the design instance for review.
-
In the Signal Configuration -> Clock box of the Signal Tap Logic Analyzer window, click the \"...\" button
-
In the Node Finder tool that opens, type clock into the Named field, and select hello_fim_top_inst in the Look in field, then click Search. Select the hello_fim_top_inst|clk signal from the Matching Nodes box and click the \">\" button to move it to the Nodes Found box. Click OK to close the Node Finder dialog.
-
Update the sample depth and other Signal Tap settings as needed for your debugging criteria.
-
In the Signal Tap GUI add the nodes to be instrumented by double-clicking on the \"Double-click to add nodes\" legend.
-
This brings up the Node Finder to add the signals to be traced. In this example we will monitor the memory mapped interface to the Hello FIM.
-
In the Named field, enter csr_lite_if*. In the Look In field, select hello_fim_top_inst.
-
Select the nets that appear in the Matching Nodes box. Click the > button to add them to the Nodes Found box.
-
Click Insert and close the Node Finder dialog.
-
To provide a unique name for your Signal Tap instance, select \"auto_signaltap_0\", right-click, and select Rename Instance (F2). Provide a descriptive name for your instance, for example, stp_for_hello_fim.
-
Save the newly created Signal Tap file, in the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/n6001/syn_top/ directory, and give it the same name as the instance. Ensure that the Add file to current project checkbox is ticked.
-
In the dialog that pops up, click \"Yes\" to add the newly created Signal Tap file to the project settings files.
This will automatically add the following lines to $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/<board_target>/syn_top/ofs_top.qsf:
set_global_assignment -name ENABLE_SIGNALTAP ON\nset_global_assignment -name USE_SIGNALTAP_FILE stp_for_hello_fim.stp\nset_global_assignment -name SIGNALTAP_FILE stp_for_hello_fim.stp\n
-
Close all Quartus GUIs.
-
Compile the project with the Signal Tap file added to the project. Use the -k switch to perform the compilation using the files in a specified working directory and not the original ones from the cloned repository.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -p -k n6001 work_hello_fim_with_stp\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -p -k n6000 work_hello_fim_with_stp\n
-
Ensure that the compile completes successfully and meets timing.
As an example:
***********************************\n***\n*** OFS_PROJECT: ofs-agx7-pcie-attach\n*** OFS_BOARD: n6001\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 6\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
Set up a JTAG connection to the n6001 or n6000. Refer to Set up JTAG section for step-by-step instructions.
-
Copy the ofs_top.sof and stp_for_hello_fim.stp files to the machine which is connected to the n6001/n6000 via JTAG.
-
From the JTAG connected machine, program the $OFS_ROOTDIR/work_hello_fim_with_stp/syn/board/<board_target>/syn_top/output_files/ofs_top.sof image to the n6001 or n6000 FPGA. Refer to the Program the FPGA via JTAG section for step-by-step programming instructions.
-
Open the Quartus Signal Tap GUI
$QUARTUS_ROOTDIR/bin/quartus_stpw\n
-
In the Signal Tap Logic Analyzer window, select File -> Open, and choose the stp_for_hello_fim.stp file.
-
In the right pane of the Signal Tap GUI, in the Hardware: selection box select the cable for the n6001/n6000. In the Device: selection box select the Agilex device.
-
If the Agilex Device is not displayed in the Device: list, click the 'Scan Chain' button to re-scan the JTAG device chain.
-
Create the trigger conditions. In this example, we will capture data on a rising edge of the Read Address Valid signal.
-
Start analysis by selecting the 'stp_for_hello_fim' instance and pressing 'F5' or clicking the Run Analysis icon in the toolbar. You should see a green message indicating the Acquisition is in progress. Then, move to the Data Tab to observe the signals captured.
-
To generate traffic in the csr_lite_if signals of the Hello FIM module, walk the DFH list or peek/poke the Hello FIM registers.
opae.io init -d 0000:98:00.0\nopae.io walk -d 0000:98:00.0\nopae.io release -d 0000:98:00.0\n
The signals should be captured on the rising edge of arvalid in this example. Zoom in to get a better view of the signals.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#42-preparing-fim-for-afu-development","title":"4.2 Preparing FIM for AFU Development","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#421-changing-the-afu-in-the-fim-port-gasket","title":"4.2.1 Changing the AFU in the FIM Port Gasket","text":"In the FIM build, the standard AFUs in the port gasket can be replaced with PIM-based AFUs, wrapped by the ofs_plat_afu() top-level module. Before invoking build_top.sh, set the environment variable AFU_WITH_PIM to the path of a sources text file -- the same sources file from examples such as sources.txt in hello_world:
export AFU_WITH_PIM=<path to>/tutorial/afu_types/01_pim_ifc/hello_world/hw/rtl/axi/sources.txt\n
When set during the build_top.sh setup stage, the FIM build is configured to load the specified AFU. PR will continue to work, if enabled. The only difference with AFU_WITH_PIM is the change to the AFUs present at power-on. Keep in mind that the default exercisers were chosen to attach to all devices and force reasonable routing decisions during the FIM build across the PR boundary. AFU_WITH_PIM is best used for FIMs that disable PR.
Configuration of the user clock from an AFU's JSON file is ignored in a FIM build.
The AFU_WITH_PIM setting matters only during the setup stage of build_top.sh. It is not consumed by later stages.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#422-compiling-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2 Compiling the FIM in preparation for designing your AFU","text":"To save area, the default Host Excercisers in the FIM can be replaced by a \"he_null\" blocks. There are a few things to note:
- \"he_null\" is a minimal block with registers that responds to PCIe MMIO request. MMIO responses are required to keep PCIe alive (end points enabled in PCIe-SS needs service downstream requests).
- If an exerciser with other I/O connections such as \"he_mem\" or \"he_hssi\" is replaced, then then those I/O ports are simply tied off.
- The options supported are
null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. Any combination and order can be enabled at the same time. - Finer grain control is provided for you to, for example, turn off only the exercisers in the Static Region in order to save area.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4221-walkthrough-compile-the-fim-in-preparation-for-designing-your-afu","title":"4.2.2.1 Walkthrough: Compile the FIM in preparation for designing your AFU","text":"Perform the following steps to compile a FIM for where the exercisers are removed and replaced with an he_null module while keeping the PF/VF multiplexor connections.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM with the HE_NULL compile options
-
Navigate to the OFS root directory
cd $OFS_ROOTDIR\n
-
Run the build_top.sh script with the null host exerciser options.
-
N6001
./ofs-common/scripts/common/syn/build_top.sh -p n6001:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_n6001\n
-
N6000
./ofs-common/scripts/common/syn/build_top.sh -p n6000:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_n6000\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#43-partial-reconfiguration-region","title":"4.3 Partial Reconfiguration Region","text":"To take advantage of the available resources in the Agilex\u00ae 7 FPGA for an AFU design, you can adjust the size of the AFU PR partition. An example reason for the changing the size of PR region is if you add more logic to the FIM region, then you may need to adjust the PR region to fit the additional logic into the static region. Similarly, if you reduce logic in the FIM region, then you can adjust the PR region to provide more logic resources for the AFU.
After the compilation of the FIM, the resources usage broken down by partitions as reported in the following two files
$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/<TARGET_BOARD>/syn_top/output_files/ofs_top.fit.place.rpt\n$OFS_ROOTDIR/<WORDK_DIRECTORY>/syn/board/<TARGET_BOARD>/syn_top/output_files/ofs_top.fit.rpt\n
An example report of the resources usage by partitions defined for the FIM is shown below.
In this case, the default size for the afu_top|port_gasket|pr_slot|afu_main PR partition is large enough to hold the logic of the default AFU, which is mainly occupied by the Host Exercisers. However, larger designs might require additional resources.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#431-walkthrough-resize-the-partial-reconfiguration-region","title":"4.3.1 Walkthrough: Resize the Partial Reconfiguration Region","text":"Perform the following steps to customize the resources allocated to the AFU in the PR regions:
-
The OFS flow provides the TCL file $OFS_ROOTDIR/syn/board/<TARGET_BOARD>/setup/pr_assignments.tcl which defines the PR partition where the AFU is allocated.
Example contents of this file for the n6000 are shown as follows:
#####################################################\n# Main PR Partition -- green_region\n#####################################################\nset_instance_assignment -name PARTITION green_region -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name CORE_ONLY_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name RESERVE_PLACE_REGION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name PARTIAL_RECONFIGURATION_PARTITION ON -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n\nset_instance_assignment -name PLACE_REGION \"X76 Y30 X300 Y195\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\nset_instance_assignment -name ROUTE_REGION \"X0 Y0 X344 Y212\" -to afu_top|pg_afu.port_gasket|pr_slot|afu_main\n
-
Use Quartus Chip Planner to identify the locations of the resources available within the Agilex\u00ae 7 FPGA chip for placement and routing your AFU. You need to identify a pair of coordinates, the origin (X0, Y0) and top right corner (X1, Y1) of the new or existing rectangles to modify as shown in the following image of the n6000 Chip Planner view.
The coordinates of the top right corner of the lock regions are computed indirectly based on the Width and Height, as follows.
X1 = X0 + Width \nY1 = Y0 + Height\n
-
Make changes to the partial reconfiguraton region in the $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments.tcl file. You can modify the size and location of existing lock regions or add new ones and assign them to the AFU PR partition.
-
Build your FIM and create the PR relocatable build tree using the build_top.sh script. Refer to the Compile OFS FIM section for step-by-step compilation instructions.
-
Analyze the resource utilization report per partition produced after recompiling the FIM.
-
Perform further modification to the PR regions until the results are satisfactory. Make sure timing constraints are met.
For more information on how to optimize the floor plan of your Partial Reconfiguration design refer to the following online documentation.
- Analyzing and Optimizing the Design Floorplan
- Partial Reconfiguration Design Flow - Step 3 - Floorplan the Design
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#44-pcie-configuration","title":"4.4 PCIe Configuration","text":"The PCIe sub-system IP and PF/VF MUX can be modified either using the OFSS flow or the IP Presets flow. The OFSS flow supports a subset of all available PCIe Sub-system settings, while the IP Preset flow can make any available PCIe Sub-system settings change. With PCIe-SS modifcations related to the PFs and VFs, the PF/VF MUX logic is automatically configured based on the PCIe-SS configuration when using OFSS.
As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the PCIe Subsystem Intel FPGA IP which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. A walkthrough is provided later in this section.
The sections below describe each modification.
- PCIe Configuration Using OFSS
- PCIe Sub-System configuration Using IP Presets
- [PCIe Sub-System Target IP]
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#441-pfvf-mux-configuration","title":"4.4.1 PF/VF MUX Configuration","text":"The default PF/VF MUX configuration for OFS PCIe Attach FIM for the n6001 can support up to 8 PFs and 2000 VFs distributed accross all PFs.
For reference FIM configurations, you must have at least 1 PF with 1VF, or 2PFs. This is because the PR region cannot be left unconnected. PFs must be consecutive. The PFVF Limitations table describes the supported number of PFs and VFs.
Table: PF/VF Limitations
Parameter Value Min # of PFs 1 (if at least 1 VF present) | 2 (if no VFs present) Max # of PFs 8 Min # of VFs 1 on PF0 (if only 1 PF present) | 0 (if 2PFs present) Max # of VFs 2000 distributed across all PFs Note that as the number of VFs goes up, timing closure can become more difficult.
The scripts provided in ${OFS_ROOTDIR}/ofs-common/tools/ofss_config allows you to easily reconfigure the number of PFs and VFs, bar addresses, vendor/device ID values and more. The PCIe Subsystem IP parameters that can be modified can be seen by reviewing ${OFS_ROOTDIR}/ofs-common/tools/ofss_config/ip_params/pcie_ss_component_parameters.py
New PF or VF instances will automatically have a null_afu module instantiated and connected to the new PF or VF.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#442-pcie-ss-configuration-registers","title":"4.4.2 PCIe-SS Configuration Registers","text":"The PCIe-SS configuration registers contain the Vendor, Device and Subsystem Vendor ID registers which are used in PCIe add-in cards to uniquely identify the card for assignment to software drivers. OFS has these registers set with Intel values for out of the box usage. If you are using OFS for a PCIe add in card that your company manufactures, then update the PCIe Subsytem Subsystem ID and Vendor ID registers as described below and change OPAE provided software code to properly operate with your company's register values.
The Vendor ID is assigned to your company by the PCI-SIG (Special Interest Group). The PCI-SIG is the only body officially licensed to give out IDs. You must be a member of PCI-SIG to request your own ID. Information about joining PCI-SIG is available here: PCI-SIG. You select the Subsystem Device ID for your PCIe add in card.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#443-pcie-configuration-using-ofss","title":"4.4.3 PCIe Configuration Using OFSS","text":"The general flow for using OFSS to modify the PCIe Sub-system and PF/VF MUX is as follows:
- Create or modify a PCIe OFSS file with the desired PCIe configuration.
- Call this PCIe OFSS file when running the FIM build script.
The PCIe IP OFSS File Options table lists all of the configuration options supported by the OFSS flow. Any other customizations to the PCIe sub-system must be done with the IP Presets Flow.
Table: PCIe IP OFSS File Options
Section Parameter Options Default Description [ip] type pcie N/A Specifies that this OFSS file configures the PCIe-SS [settings] output_name pcie_ss N/A Specifies the output name of the PCIe-SS IP preset String N/A OPTIONAL - Specifies the name of a PCIe-SS IP presets file to use when building the FIM. When used, a presets file will take priority over any other parameters set in this OFSS file. [pf*] num_vfs Integer 0 Specifies the number of Virtual Functions in the current PF bar0_address_width Integer 12 bar4_address_width Integer 14 vf_bar0_address_width Integer 12 ats_cap_enable 0 | 1 0 vf_ats_cap_enable 0 | 1 0 prs_ext_cap_enable 0 | 1 0 pasid_cap_enable 0 | 1 0 pci_type0_vendor_id 32'h Value 0x00008086 pci_type0_device_id 32'h Value 0x0000bcce revision_id 32'h Value 0x00000001 class_code 32'h Value 0x00120000 subsys_vendor_id 32'h Value 0x00008086 subsys_dev_id 32'h Value 0x00001771 sriov_vf_device_id 32'h Value 0x0000bccf exvf_subsysid 32'h Value 0x00001771"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4431-walkthrough-modify-the-pcie-sub-system-and-pfvf-mux-configuration-using-ofss","title":"4.4.3.1 Walkthrough: Modify the PCIe Sub-System and PF/VF MUX Configuration Using OFSS","text":"Perform the following steps to modify the PF/VF MUX configuration.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
View the default PCIe OFSS file contents. For example, the n6001 default PCIe OFSS configuration is defined in the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n
-
Create a new PCIe OFSS file from the existing pcie_host.ofss file
cp $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_pfvf_mod.ofss\n
-
Configure the new pcie_pfvf_mod.ofss with your desired settings. In this example we will add PF5 with 2 VFs.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\n\n[pf0]\nnum_vfs = 3\nbar0_address_width = 20\nvf_bar0_address_width = 20\n[pf1]\n[pf2]\nbar0_address_width = 18\n[pf3]\n[pf4]\n[pf5]\nnum_vfs = 2\n
-
Compile the FIM with the new PCIe OFSS file. In this example we are targeting the n6001 board.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_pfvf_mod.ofss n6001 work_n6001_pfvf_mod\n
-
Copy the resulting .bin user 1 image to your deployment environmenment.
-
Switch to your deployment environment.
-
Program the .bin image to the n6001 FPGA. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Verify the number of VFs on the newly added PF8. In this example, we defined 2 VFs on PF5 in Step 5.
sudo lspci -vvv -s 98:00.5 | grep VF\n
Example output:
Initial VFs: 2, Total VFs: 2, Number of VFs: 0, Function Dependency Link: 05\nVF offset: 4, stride: 1, Device ID: bccf\nVF Migration: offset: 00000000, BIR: 0\n
-
Verify communication with the newly added PF5. New PF/VF are seamlessly connected to their own CSR stub, which can be read at DFH Offset 0x0. You can bind to the function and perform opae.io peek commands to read from the stub CSR. Similarly, perform opae.io poke commands to write into the stub CSRs. Use this mechanism to verify that the new PF/VF Mux configuration allows to write and read back values from the stub CSRs.
The GUID for every new PF/VF CSR stub is the same.
* NULL_GUID_L = 64'haa31f54a3e403501\n* NULL_GUID_H = 64'h3e7b60a0df2d4850\n
1. Initialize the driver on PF5
```bash\nsudo opae.io init -d 98:00.5\n```\n
2. Read the GUID for the PF5 CSR stub.
```bash\nsudo opae.io -d 98:00.5 -r 0 peek 0x8\nsudo opae.io -d 98:00.5 -r 0 peek 0x10\n```\n\nExample output:\n\n```bash\n0xaa31f54a3e403501\n0x3e7b60a0df2d4850\n```\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#444-pcie-sub-system-configuration-using-ip-presets","title":"4.4.4 PCIe Sub-System configuration Using IP Presets","text":"The general flow for using IP Presets to modify he PCIe Sub-system is as follows:
- [OPTIONAL] Create a work directory using OFSS files if you wish to use OFSS configuration settings as a starting point.
- Open the PCIe-SS IP and make desired modifications.
- Create an IP Presets file.
- Create an PCIe OFSS file that uses the IP Presets file.
- Build the FIM with the PCIe OFSS file from Step 4.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4441-walkthrough-modify-pcie-sub-system-configuration-using-ip-presets","title":"4.4.4.1 Walkthrough: Modify PCIe Sub-System Configuration Using IP Presets","text":"Perform the following steps to use an IP preset file to configure the PCIe Sub-system and PF/VF MUX. In this example, we will change the Revision ID on PF0. While this modification can be done with the OFSS flow, this walkthrough is intended to show the procedure for making any PCIe configuration change using IP presets. This walkthrough targets the n6001, but similar steps can be applied to the n6000.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment to test the design. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script using your desired OFSS configration to create a working directory for the target board. In this example we will target the n6001.
./ofs-common/scripts/common/syn/build_top.sh --stage setup n6001 work_n6001\n
-
Open the PCIe-SS in the work directory using Quartus Parameter Editor. If you performed Step 3, open the PCIe-SS IP from the work directory; otherwise, open the PCIe-SS IP from the source files.
qsys-edit $OFS_ROOTDIR/work_n6001/ipss/pcie/qip/pcie_ss.ip\n
-
Modify the settings as desired. In this example we are changing Revision ID to 0x2. In the IP Parameter Editor, scroll down and expand the PCIe Interfaces Ports Settings -> Port 0 -> PCIe0 Device Identification Registers -> PCIe0 PF0 IDs tab and make these changes.
-
Once you are satisfied with your modifcations, create a new IP Preset file.
-
click New... in the Presets window.
-
In the New Preset window, set a unique Name for the preset; for example, n6001-rev2.
-
Click the ... button to set the save location for the IP Preset file to $OFS_ROOTDIR/ipss/pcie/presets. Set the File Name to match the name selected in Step 9. Click OK.
-
In the New Preset window, click Save. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Create a new PCIe OFSS file in the $OFS_ROOTDIR/tools/ofss_config/pcie directory. For example:
touch $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_mod_preset.ofss\n
Insert the following into the OFSS file to specify the IP Preset file created in Step 6.
[ip]\ntype = pcie\n\n[settings]\noutput_name = pcie_ss\npreset = n6001-rev2\n
-
Compile the design with the modified new pcie_host_mod_preset.ofss file.
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_mod_preset.ofss n6001 work_n6001_pcie_mod\n
-
Copy the resulting $OFS_ROOTDIR/work_n6001_pcie_mod/syn/board/n6001/syn_top/output_files/ofs_top_hps.sof image to your deployment environmenment for JTAG programming, or copy a bin file (e.g. ofs_top_page1_unsigned_user1.bin) for RSU programming.
Note: OPAE FPGA management commands require recognition of the FPGA PCIe Device ID for control. If there is a problem between OPAE management recognition of FPGA PCIe values, then control of the card will be lost. For this reason, you are strongly encouraged to program the FPGA via JTAG to load the test FPGA image. If there is a problem with the SOF image working with your host software that is updated for the new PCIe settings, then you can load a known good SOF file to recover. Once you sure that both the software and FPGA work properly, you can load the FPGA into FPGA flash using the OPAE command fpgasupdate.
-
Switch to your deployment environment.
-
Program the image to the n6001 FPGA. Refer to the Program the FPGA via JTAG Section for step-by-step JTAG programming instructions, or the Program the FPGA via RSU Section for step-by-step RSU programming instructions.
-
Determing the PCIe B:D.F of your board. You may use the OPAE command fpgainfo fme to determine this.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : bdea3c8cdf144c7c227b416cac1358f7\nUser1 Image Info : bbe54f1804c88bbd5a555b072c9f5d80\nUser2 Image Info : bdea3c8cdf144c7c227b416cac1358f7\n
-
Use lspci to verify that the PCIe changes have been implemented.
lspci -nvmms 98:00.0\n
Example output:
Slot: 98:00.0\nClass: 1200\nVendor: 8086\nDevice: bcce\nSVendor: 8086\nSDevice: 1771\nPhySlot: 1\nRev: 02\nNUMANode: 1\nIOMMUGroup: 180\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#445-pcie-sub-system-target-ip","title":"**4.4.5 PCIe Sub-System Target IP **","text":"As of the OFS 2024.2 release, the PCIe-SS uses the AXI Streaming Intel FPGA IP for PCI Express by default, which does not support Data Mover Mode. OFS releases prior to OFS 2024.2 used the Intel FPGA IP Subsystem for PCI Express which does support Data Mover Mode. If Data Mover Mode is required, you must change the target IP in the PCIe OFSS file. You must also reduce the IOPLL frequency to a maximum of 470 MHz.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#4451-walkthrough-change-the-pcie-sub-system-target-ip","title":"4.4.5.1 Walkthrough: Change the PCIe Sub-System Target IP","text":"Perform the following steps to change the target PCIe IP from AXI Streaming Intel FPGA IP for PCI Express to Intel FPGA IP Subsystem for PCI Express.
Pre-requisites:
- This walkthrough requires a development environment to build the FIM. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Add the following line to the [settings] section of the PCIe OFSS file you are using. In this example, it will be added to the $OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host.ofss file, which is the default PCIe OFSS file used for n6001.
ip_component = pcie_ss\n
Note: To change back to using the AXI Streaming Intel FPGA IP for PCI Express, simply remove this line.
-
Build the FIM using the 470 MHz IOPLL and the PCIe OFSS file that was modified in step 3.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/iopll/iopll_470MHz.ofss,tools/ofss_config/pcie/pcie_host.ofss n6001 work_n6001\n
-
When the build completes, the output files can be found in $OFS_ROOTDIR/work_n6001/syn/board/n6001/output_files.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#45-minimal-fim","title":"4.5 Minimal FIM","text":"In a minimal FIM, the exercisers and Ethernet subsystem are removed from the design, the PF/VF configuration is reduced to either 2PF/0VF or 1PF/1VF, and the AFU PR area is expanded to make use of the available area previously used by the removed components.
There are two pre-provided PCIe configurations that can be used to create minimal FIM:
- 2PF: this PCIe configuration has two physical functions, with the APF/BPF on PF0, and the AFU PR region on PF1.
- 1PF/1VF: This PCIe configuration has a single physical function, with the APF/BPF on PF0, and the AFU PR region on PF0-VF0.
The FIM source repository contains OFSS file that can be used to configure the PCIe IP for the minimal FIM.
$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_2pf.ofss$OFS_ROOTDIR/tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#451-walkthrough-create-a-minimal-fim","title":"4.5.1 Walkthrough: Create a Minimal FIM","text":"Perform the following steps to create a Minimal FIM. A minimal FIM is one that has the host exercisers and ethernet subsystem removed. This frees up resources that can be used.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment for hardware testing. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile)) for instructions on setting up a deployment environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
[N6001 ONLY] The OFS FIM repo provides a PR assignments TCL file which optimizes the PR region for the minimal FIM. Copy the minimal PR assignments TCL file into the pr_assignments.tcl file location for use in the FIM build process.
-
Rename the current pr_assignments.tcl file to pr_assignments_base.tcl for future use
mv $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments.tcl $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments_base.tcl\n
-
Copy the pr_assignments_slim.tcl file to pr_assignments.tcl to be used in the current build
cp $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments_slim.tcl $OFS_ROOTDIR/syn/board/n6001/setup/pr_assignments.tcl\n
-
Compile the FIM with Null HEs compile option, the No HSSI compile option, and the desired PCIe PF/VF configuration.
cd $OFS_ROOTDIR\n
-
2PF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2pf.ofss n6001:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_n6001_minimal_fim\n
-
1PF/1VF Minimal FIM
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss n6001:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_n6001_minimal_fim\n
-
Review the $OFS_ROOTDIR/work_n6001_minimal_fim/syn/board/n6001/syn_top/output_files/ofs_top.fit.rpt utilization report to see the utilization statistics for the Minimal fim. Refer to Appendix A: Resource Utilization Tables Table A-4 for the expected utilization for this Minimal FIM.
-
Copy the resulting $OFS_ROOTDIR/work_n6001_minimal_fim/syn/board/n6001/syn_top/output_files/ofs_top_page1_unsigned_user1.bin image to your deployment environmenment.
-
Switch to your deployment environment, if different than your development environment.
-
Program the .bin image to the n6001 FPGA. Refer to the Program the FPGA via RSU Section for step-by-step programming instructions.
-
Verify the minimal FIM design on the board.
-
For 2PF Minimal FIM:
- Use
opae.io ls to verify that there are two PFs
sudo opae.io ls\n
Example output:
[0000:98:00.1] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
-
Use fpgainfo port to verify the ports.
sudo fpgainfo port\n
Example output:
//****** PORT ******//\nInterface : DFL\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nInterface : UIO\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 00000000-0000-0000-0000-000000000000\n
-
Bind the VFIO driver on PF1.
sudo opae.io init -d 98:00.1\n
Example output:
Unbinding (0x8086,0xbcce) at 0000:98:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:98:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:98:00.1 is 15\n
-
Use fpgainfo port to verify the ports. There should now be a port entry for PF1. The Accelerator GUID for PF1 should be as shown below, which is the GUID for the HE-MEM.
//****** PORT ******//\n Interface : DFL\n Object Id : 0xEE00000\n PCIe s:b:d.f : 0000:98:00.0\n Vendor Id : 0x8086\n Device Id : 0xBCCE\n SubVendor Id : 0x8086\n SubDevice Id : 0x1771\n Socket Id : 0x00\n //****** PORT ******//\n Interface : VFIO\n Object Id : 0x2098000000000000\n PCIe s:b:d.f : 0000:98:00.1\n Vendor Id : 0x8086\n Device Id : 0xBCCE\n SubVendor Id : 0x8086\n SubDevice Id : 0x1771\n Socket Id : 0x01\n Accelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n //****** PORT ******//\n Interface : UIO\n Object Id : 0xED00000\n PCIe s:b:d.f : 0000:98:00.0\n Vendor Id : 0x8086\n Device Id : 0xBCCE\n SubVendor Id : 0x8086\n SubDevice Id : 0x1771\n Socket Id : 0x01\n Accelerator GUID : 00000000-0000-0000-0000-000000000000\n
-
For 1PF/1VF Minimal FIM:
- Use
opae.io ls to verify that there is one PF.
sudo opae.io ls\n
Example output:
[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
-
Use lspci to verify that there is 1 VF on PF0.
sudo lspci -vvv -s 98:00.0 | grep VF\n
Example output:
Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00\nVF offset: 1, stride: 1, Device ID: bcce\nVF Migration: offset: 00000000, BIR: 0\n
-
Use fpgainfo port to verify the ports.
sudo fpgainfo port\n
Example output:
//****** PORT ******//\nInterface : DFL\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nInterface : UIO\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 00000000-0000-0000-0000-000000000000\n
-
Initialize one virtual function
sudo pci_device 0000:98:00.0 vf 1\n
-
Use opae.io ls to verify the VF has been initialized.
sudo opae.io ls\n
Example output:
[0000:98:00.0] (0x8086:0xbcce 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n[0000:98:00.1] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: dfl-pci)\n
-
Bind the VFIO driver on VF0
sudo opae.io init -d 0000:98:00.1\n
-
Use fpgainfo port to verify the ports. There should now be a port entry for VF0. The Accelerator GUID for VF0 should be as shown below, which is the GUID for the HE-MEM.
sudo fpgainfo port\n
Example output:
//****** PORT ******//\nInterface : DFL\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nInterface : VFIO\nObject Id : 0x2098000000000000\nPCIe s:b:d.f : 0000:98:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nInterface : UIO\nObject Id : 0xED00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 00000000-0000-0000-0000-000000000000\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#46-migrate-to-a-different-agilex-device-number","title":"4.6 Migrate to a Different Agilex Device Number","text":"The following instructions enable a user to change the device part number of the Intel\u00ae FPGA SmartNIC N6001-PL. Please be aware that this release works with Agilex\u00ae 7 FPGA devices that have P tile for PCIe and E tile for Ethernet. Other tiles will take further work.
You may wish to change the device part number for the following reasons
- Migrate to same device package but with a different density
- Migrate to a different package and with a different or same density
The default device for the Intel\u00ae FPGA SmartNIC N6001-PL is AGFB014R24A2E2V
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#461-walkthrough-migrate-to-a-different-agilex-device-number","title":"4.6.1 Walkthrough: Migrate to a Different Agilex Device Number","text":"Perform the following steps to migrate your design to target a different Agilex device using the OFSS build flow. In this example we will change the device from the default AGFB014R24A2E2V to a new device AGFB022R25A2E2V. This walkthrough will target the n6001, but similar steps can be used for the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Modify the part field in the $OFS_ROOTDIR/tools/ofss_config/n6001_base.ofss file to use AGFB022R25A2E2V. This is only necessary if you are using the OFSS flow.
[ip]\ntype = ofs\n\n[settings]\nplatform = n6001\nfamily = agilex\nfim = base_x16\npart = AGFB022R25A2E2V\ndevice_id = 6001\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFB022R25A2E2V\n
-
Modify the DEVICE field in the $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_pr_afu.qsf file.
############################################################################################\n# FPGA Device\n############################################################################################\nset_global_assignment -name FAMILY Agilex\nset_global_assignment -name DEVICE AGFB022R25A2E2V\n
-
Modify the DEVICE field in te $OFS_ROOTDIR/ipss/pmci/pmci_ss.qsf file.
set_global_assignment -name DEVICE AGFB022R25A2E2V\n
-
If you are changing to a device with a different package, you must change the pin assignments in the location files. If you would like Quartus to attempt to pin out the design automatically, you may remove all pin assignments instead. Typically you will be required to set certain pins manually in order to guide Quartus for a successful compile (e.g. transceiver reference clocks).
-
Comment out all of the pin constraints in the following files and attempt to let Quartus pin out the design as much as possibe:
$OFS_ROOTDIR/syn/board/n6001/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/hps_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/pmci_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl
-
Identify the pins you wish to assign prior to compiling. In this example, we will re-pin some of the reference clocks to help guide the fitter. Refer to the Pin-Out Files for Altera FPGAs for the pin list of your device. In this example, the Migration Re-Pin Mapping table below shows the pins we will re-pin in the constraints files.
Table: Migration Re-Pin Mapping
Pin Name FIM Signal Name AGF 014 R24A Pin # AGF 022 R25A Pin # REFCLK_GXER9A_CH0p cr3_cpri_reflclk_clk[0] PIN_AT13 PIN_CE18 REFCLK_GXER9A_CH0n \"cr3_cpri_reflclk_clk[0](n)\" PIN_AP13 PIN_CA18 REFCLK_GXER9A_CH1p cr3_cpri_refclk_clk[1] PIN_AR14 PIN_CC19 REFCLK_GXER9A_CH1n \"cr3_cpri_refclk_clk[1](n)\" PIN_AN14 PIN_BW19 REFCLK_GXER9A_CH2p cr3_cpri_refclk_clk[2] PIN_AJ12 PIN_BL17 REFCLK_GXER9A_CH2n \"cr3_cpri_refclk_clk[2](n)\" PIN_AH11 PIN_BJ15 REFCLK_GXER9A_CH3p qsfp_ref_clk PIN_AK13 PIN_BN18 REFCLK_GXER9A_CH3n \"qsfp_ref_clk(n)\" PIN_AH13 PIN_BJ18 REFCLK_GXER9A_CH4p cr3_cpri_reflclk_clk_184_32m PIN_AJ14 PIN_BL19 REFCLK_GXER9A_CH4n \"cr3_cpri_reflclk_clk_184_32m(n)\" PIN_AL14 PIN_BR19 REFCLK_GXER9A_CH5p cr3_cpri_reflclk_clk_153_6m PIN_AR16 PIN_CC21 REFCLK_GXER9A_CH5n \"cr3_cpri_reflclk_clk_153_6m(n)\" PIN_AN16 PIN_BW21 REFCLK_GXPL10A_CH0n \"PCIE_REFCLK0(n)\" PIN_AH49 PIN_DD56 REFCLK_GXPL10A_CH0p PCIE_REFCLK0 PIN_AJ48 PIN_DF57 REFCLK_GXPL10A_CH2n \"PCIE_REFCLK1(n)\" PIN_AD49 PIN_CT56 REFCLK_GXPL10A_CH2p PCIE_REFCLK1 PIN_AE48 PIN_CV57 -
Re-pin the pins identified in the previous step in the $OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl file for the new pinout.
set_location_assignment PIN_BN18 -to qsfp_ref_clk\nset_location_assignment PIN_BJ18 -to \"qsfp_ref_clk(n)\"\nset_location_assignment PIN_CC19 -to cr3_cpri_refclk_clk[1]\nset_location_assignment PIN_BW19 -to \"cr3_cpri_refclk_clk[1](n)\"\nset_location_assignment PIN_BL17 -to cr3_cpri_refclk_clk[2]\nset_location_assignment PIN_BJ15 -to \"cr3_cpri_refclk_clk[2](n)\"\nset_location_assignment PIN_CE18 -to cr3_cpri_reflclk_clk[0]\nset_location_assignment PIN_CA18 -to \"cr3_cpri_reflclk_clk[0](n)\"\nset_location_assignment PIN_BL19 -to cr3_cpri_reflclk_clk_184_32m\nset_location_assignment PIN_BR19 -to \"cr3_cpri_reflclk_clk_184_32m(n)\"\nset_location_assignment PIN_CC21 -to cr3_cpri_reflclk_clk_153_6m\nset_location_assignment PIN_BW21 -to \"cr3_cpri_reflclk_clk_153_6m(n)\"\nset_location_assignment PIN_DD56 -to \"PCIE_REFCLK0(n)\"\nset_location_assignment PIN_DF57 -to PCIE_REFCLK0\nset_location_assignment PIN_CT56 -to \"PCIE_REFCLK1(n)\"\nset_location_assignment PIN_CV57 -to PCIE_REFCLK1\n
-
Un-comment the instance assignemnts for the transceiver reference clocks defined in $OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl.
set_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=156250000\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to qsfp_ref_clk\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=TRUE\" -to qsfp_ref_clk\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=184320000\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_refclk_clk[1]\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=153600000\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_refclk_clk[2]\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=disable_3p3v_tol\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=245760000\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_reflclk_clk[0]\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=enable_3p3v_tol\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=184320000\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_reflclk_clk_184_32m\nset_instance_assignment -name IO_STANDARD \"DIFFERENTIAL LVPECL\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_termination=enable_term\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_enable_3p3v=enable_3p3v_tol\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_disable_hysteresis=enable_hyst\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_input_freq=153600000\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_powerdown_mode=false\" -to cr3_cpri_reflclk_clk_153_6m\nset_instance_assignment -name HSSI_PARAMETER \"refclk_divider_use_as_bti_clock=FALSE\" -to cr3_cpri_reflclk_clk_153_6m\n
-
Compile a flat design. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh n6001:flat work_n6001\n
-
Verify that the build completes successfuly. If the compilation fails with errors relating to the pinout, review the error messages and modify the pinout accordingly. If there are timing violations, try building with a different seed. Refer to the Change the Compilation Seed section for instructions on changing the build seed.
***********************************\n***\n*** OFS_PROJECT: n6001\n*** OFS_BOARD: n6001\n*** Q_PROJECT: ofs_top\n*** Q_REVISION: ofs_top\n*** SEED: 3\n*** Build Complete\n*** Timing Passed!\n***\n***********************************\n
-
When you are satisfied with the pinout, preserve it by hard-coding the desired pinout back to the followig files:
$OFS_ROOTDIR/syn/board/n6001/setup/emif_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/hps_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/pmci_loc.tcl$OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl ```
-
When you are ready to re-incorporate PR into the design, modify the PR region to be compatible with the new device. Refer to the Resize the Partial Reconfiguration Region section for instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#47-modify-the-memory-sub-system","title":"4.7 Modify the Memory Sub-System","text":"OFS allows modifications on the Memory Sub-System in the FIM. This section provides an example walkthrough for modifiying the Memory-SS.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#471-walkthrough-modify-the-memory-sub-system-using-ip-presets-with-ofss","title":"4.7.1 Walkthrough: Modify the Memory Sub-System Using IP Presets With OFSS","text":"This walkthrough will go through the flow of modifying the Memory-SS in the OFS FIM. In this example, we will enable ECC on memory channels 0-3. Note that routes for the ECC pins on Channels 0 and 1 are not physiclly present on standard n6001/n6000 board hardware; the purpose of this walkthrough is only to show an example of how to make modifications to the IP. This walkthrough targets the n6001 board, but similar steps can be used for the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Memory Subsystem mem_ss.ip in IP Parameter Editor
qsys-edit $OFS_ROOTDIR/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
The Memory Subsystem IP will open in IP Parameter Editor. Click Dive Into Packaged Subsystem.
-
The Platform Designer mem_ss view opens. All of the EMIFs are shown in the Filter window.
-
Click each EMIF 0 through 3 and perform the following actions.
-
In the Parameters window, click the Memory tab and change the DQ width to 40.
-
In the Parameters window, click the Controller tab.
-
Scroll down and check the box for Enable Error Detection and Correction Logic with ECC.
-
Once Step 6 has been done for each EMIF 0-3, click File -> Save. Close the Platform Designer window.
-
In the IP Parameter Editor Presets window, click New to create an IP Presets file.
-
In the New Preset window, set the Name for the preset. In this case we will name it n6001.
-
Click the ... button to select the location for the Preset file.
-
In the Save As window, change the save location to $OFS_ROOTDIR/ipss/mem/qip/presets and change the File Name to n6001.qprs. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close the IP Parameter Editor. You do not need to generate or save the IP.
-
Edit the $OFS_ROOTDIR/syn/board/n6001/setup/emif_loc.tcl file to add pin assignments for the new signals supporting ECC on Channels 0-3. Note that routes for the ECC pins on Channels 0 and 1 are not physiclly present on a standard n6001 board.
# CH0 DQS4 (ECC)\nset_location_assignment PIN_CG48 -to ddr4_mem[0].dbi_n[4]\nset_location_assignment PIN_CF47 -to ddr4_mem[0].dqs_n[4]\nset_location_assignment PIN_CH47 -to ddr4_mem[0].dqs[4]\nset_location_assignment PIN_CE50 -to ddr4_mem[0].dq[32]\nset_location_assignment PIN_CG50 -to ddr4_mem[0].dq[33]\nset_location_assignment PIN_CF49 -to ddr4_mem[0].dq[34]\nset_location_assignment PIN_CH49 -to ddr4_mem[0].dq[35]\nset_location_assignment PIN_CE46 -to ddr4_mem[0].dq[36]\nset_location_assignment PIN_CG46 -to ddr4_mem[0].dq[37]\nset_location_assignment PIN_CF45 -to ddr4_mem[0].dq[38]\nset_location_assignment PIN_CH45 -to ddr4_mem[0].dq[39]\n# CH1 DQS4 (ECC)\nset_location_assignment PIN_DC34 -to ddr4_mem[1].dbi_n[4]\nset_location_assignment PIN_CY33 -to ddr4_mem[1].dqs_n[4]\nset_location_assignment PIN_DB33 -to ddr4_mem[1].dqs[4]\nset_location_assignment PIN_DA36 -to ddr4_mem[1].dq[32]\nset_location_assignment PIN_DC36 -to ddr4_mem[1].dq[33]\nset_location_assignment PIN_CY35 -to ddr4_mem[1].dq[34]\nset_location_assignment PIN_DB35 -to ddr4_mem[1].dq[35]\nset_location_assignment PIN_DA32 -to ddr4_mem[1].dq[36]\nset_location_assignment PIN_DC32 -to ddr4_mem[1].dq[37]\nset_location_assignment PIN_CY31 -to ddr4_mem[1].dq[38]\nset_location_assignment PIN_DB31 -to ddr4_mem[1].dq[39]\n# CH2 DQS4 (ECC)\nset_location_assignment PIN_G36 -to ddr4_mem[2].dbi_n[4]\nset_location_assignment PIN_H35 -to ddr4_mem[2].dqs_n[4]\nset_location_assignment PIN_F35 -to ddr4_mem[2].dqs[4]\nset_location_assignment PIN_G38 -to ddr4_mem[2].dq[32]\nset_location_assignment PIN_J38 -to ddr4_mem[2].dq[33]\nset_location_assignment PIN_H33 -to ddr4_mem[2].dq[34]\nset_location_assignment PIN_J34 -to ddr4_mem[2].dq[35]\nset_location_assignment PIN_F33 -to ddr4_mem[2].dq[36]\nset_location_assignment PIN_H37 -to ddr4_mem[2].dq[37]\nset_location_assignment PIN_F37 -to ddr4_mem[2].dq[38]\nset_location_assignment PIN_G34 -to ddr4_mem[2].dq[39]\n# CH3 DQS4 (ECC)\nset_location_assignment PIN_L50 -to ddr4_mem[3].dbi_n[4]\nset_location_assignment PIN_P49 -to ddr4_mem[3].dqs_n[4]\nset_location_assignment PIN_M49 -to ddr4_mem[3].dqs[4]\nset_location_assignment PIN_M51 -to ddr4_mem[3].dq[32]\nset_location_assignment PIN_N48 -to ddr4_mem[3].dq[33]\nset_location_assignment PIN_M47 -to ddr4_mem[3].dq[34]\nset_location_assignment PIN_L48 -to ddr4_mem[3].dq[35]\nset_location_assignment PIN_P47 -to ddr4_mem[3].dq[36]\nset_location_assignment PIN_P51 -to ddr4_mem[3].dq[37]\nset_location_assignment PIN_N52 -to ddr4_mem[3].dq[38]\nset_location_assignment PIN_L52 -to ddr4_mem[3].dq[39]\n
-
Compile the design. In this example we are targeting the n6001.
./ofs-common/scripts/common/syn/build_top.sh -p n6001 work_n6001_mem_ecc_preset\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#472-walkthrough-add-or-remove-the-memory-sub-system","title":"4.7.2 Walkthrough: Add or remove the Memory Sub-System","text":"The Memory Sub-System can be added or removed to the FIM design using the INCLUDE_DDR4 macro defined in the ofs_top.qsf file. The Memory-SS is enabled by default in the n6001, and is disabled by default in the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit the ofs_top.qsf file to either enable or disable the INCLUDE_DDR4 macro. Comment out this macro assignemnt if you wish to remove the Memory-SS.
Note: The default Memory-SS has connections to the HPS. When enabling the Memory-SS, you must either ensure that the INCLUDE_HPS and INCLUDE_UART macros are also enabled, or you must remove the connections from the Memory-SS to the HPS. Refer to the Remove the HPS section for step-by-step instructions on removing the HPS from the design.
-
Compile the design. Refer to the Compile OFS FIM section for step-by-step instructions.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#48-modify-the-ethernet-sub-system","title":"4.8 Modify the Ethernet Sub-System","text":"This section describes the flows for modifying the Ethernet Sub-System. There are three flows you may use to make modifications.
- Modify the Ethernet Sub-System with OFSS supported changes only. These modifications are supported natively by the build script, and are made at run-time of the build script. This flow is useful for users who only need to leverage natively supported HSSI OFSS settings.
- Modify the Ethernet Sub-System with OFSS supported changes, then make additional custom modifications not covered by OFSS. These modifications will be captured in a presets file which can be used in future compiles. This flow is useful for users who whish to leverage pre-made HSSI OFSS settings, but make additional modifications not natively supported by HSSI OFSS.
- Modify the Ethernet Sub-System without HSSI OFSS. These modification will be made directly in the source files.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#481-walkthrough-modify-the-ethernet-sub-system-channels-with-pre-made-hssi-ofss","title":"4.8.1 Walkthrough: Modify the Ethernet Sub-System Channels With Pre-Made HSSI OFSS","text":"This walkthrough describes how to use OFSS to configure the Ethernet-SS. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This walkthrough is useful for users who only need to leverage the pre-made, natively supported HSSI OFSS settings.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Compile the FIM using the desired HSSI configuration.
cd $OFS_ROOTDIR\n
-
N6001 - 2x4x25GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_8x25.ofss n6001 work_n6001\n
-
N6001 - 2x4x10GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_8x10.ofss.ofss n6001 work_n6001\n
-
N6001 - 2x1x100GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_2x100.ofss n6001 work_n6001\n
-
N6000 - 4x1x100GbE
./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/hssi/hssi_4x100.ofss n6000 work_n6000\n
-
The resulting FIM will contain the Ethernet-SS configuration specified in Step 3. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#482-walkthrough-add-channels-to-the-ethernet-sub-system-channels-with-custom-hssi-ofss","title":"4.8.2 Walkthrough: Add Channels to the Ethernet Sub-System Channels With Custom HSSI OFSS","text":"This walkthrough describes how to create an use a custom OFSS file to add channels to the Ethernet-SS and compile a design with a 3x4x10GbE Ethernet-SS configuration. This walkthrough is useful for users who wish to leverage the natively supported HSSI OFSS settings.
Note: The n6000 uses 16 channels in the HSSI-SS by default: 8 to the QSFPs, and 8 to the E810 device. Thus, this walkthrough applies only to the n6001.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a new HSSI OFSS file $OFS_ROOTDIR/tools/ofss_config/hssi/hssi_12x10.ofss with the following contents.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 12\ndata_rate = 10GbE\n
-
Identify the which channels will be added. You may use the E-Tile Channel Placement Tool to aid in your design. In this example we will add the 4 new 10GbE channels to Channels 8-11.
-
Based on your channel selection, identify which pins will be used. Refer to the Pin-Out Files for Altera FPGAs determine the required pins for your device. In this example we are targeting the AGFB014R24A2E2V device. Set the pin assignments in the $OFS_ROOTDIR/syn/board/n6001/setup/top_loc.tcl file.
set_location_assignment PIN_AV7 -to qsfp_serial[2].rx_p[0]\nset_location_assignment PIN_AW10 -to qsfp_serial[2].rx_p[1]\nset_location_assignment PIN_BB7 -to qsfp_serial[2].rx_p[2]\nset_location_assignment PIN_BC10 -to qsfp_serial[2].rx_p[3]\nset_location_assignment PIN_AV1 -to qsfp_serial[2].tx_p[0]\nset_location_assignment PIN_AW4 -to qsfp_serial[2].tx_p[1]\nset_location_assignment PIN_BB1 -to qsfp_serial[2].tx_p[2]\nset_location_assignment PIN_BC4 -to qsfp_serial[2].tx_p[3]\n
-
Change the number of QSFP ports from 2 to 3 in the $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv file.
localparam NUM_QSFP_PORTS_USED = 3; // Number of QSFP cages on board used by target hssi configuration\n
-
Edit $OFS_ROOTDIR/ofs-common/src/fpga_family/agilex/hssi_ss/hssi_wrapper.sv so that the QSFP LED signals use NUM_QSFP_PORTS_USED defined in the previous step.
// Speed and activity LEDS\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_green, // Link up in Nx25G or 2x56G or 1x100G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_speed_yellow, // Link up in Nx10G speed\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_green, // Link up and activity seen\noutput logic [NUM_QSFP_PORTS_USED-1:0] o_qsfp_activity_red // LOS, TX Fault etc\n
-
Compile the design using the OFSS file created previously. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you determine the correct pinout for your design.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_12x10.ofss n6001:flat work_n6001_12x10\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#483-walkthrough-modify-the-ethernet-sub-system-with-pre-made-hssi-ofss-plus-additional-modifications","title":"4.8.3 Walkthrough: Modify the Ethernet Sub-System With Pre-Made HSSI OFSS Plus Additional Modifications","text":"This walkthrough describes how to use OFSS to first modify the Ethernet-SS, then make additional modifications on top. Refer to section HSSI IP OFSS File for detailed information about modifications supported by Ethernet-SS OFSS files. This flow is useful for users who whish to leverage pre-made OFSS settings, but make additional modifications not natively supported by OFSS. This walkthorugh targets the n6001, however, similiar steps can be performed on the n6000.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Run the setup stage of the build script with the desired HSSI OFSS file to create a work directory whith the desired HSSI configuration. For example, to create a work directory for n6001 with HSSI configuration 8x25 GbE.
cd $OFS_ROOTDIR\n./ofs-common/scripts/common/syn/build_top.sh --stage setup --ofss tools/ofss_config/hssi/hssi_8x25.ofss n6001 work_n6001\n
-
Open the Ethernet-SS IP in Quartus Parameter Editor. The IP settings will match te configuration of the OFSS file defined in Step 3. Make any additional modifications in the Parameter Editor.
qsys-edit $OFS_ROOTDIR/work_n6001/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the New... button in the Presets pane of IP Parameter Editor.
-
In the New Preset window, create a unique Name. In this example the name is n6001-hssi-presets.
-
Click the ... button to select where to save the preset file. Give it a name, and save it to $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/presets
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close out of all Quartus GUIs. You do not need to save or compile the IP.
-
Create a new HSSI OFSS file in the $OFS_ROOTDIR/tools/ofss_config/hssi directory named hssi_preset_n6001.ofss with the following contents. Note that the num_channels and data_rate settings will be overwritten by the contents of the preset file. The preset setting must match the name you selected in Step 7.
[ip]\ntype = hssi\n\n[settings]\noutput_name = hssi_ss\nnum_channels = 8\ndata_rate = 25GbE\npreset = n6001-hssi-presets\n
-
Compile the design using the new HSSI OFSS file. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_preset_n6001.ofss n6001:flat work_n6001_hssi_preset\n
-
The resulting FIM will contain the Ethernet-SS configuration specified by the presets file. The Ethernet-SS IP in the resulting work directory shows the parameter settings that are used.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#484-walkthrough-modify-the-ethernet-sub-system-without-hssi-ofss","title":"4.8.4 Walkthrough: Modify the Ethernet Sub-System Without HSSI OFSS","text":"This walkthrough describes how to modify the Ethernet-SS wihout using OFSS. This flow will edit the Ethernet-SS IP source directly. This walkthrough is useful for users who wish to make all Ethernet-SS modifications manually, without leveraging HSSI OFSS.
Pre-Requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Open the Ethernet-SS IP in Quartus Parameter Editor. Make your modifications in the Parameter Editor.
qsys-edit $OFS_ROOTDIR/ipss/hssi/qip/hssi_ss/hssi_ss.ip\n
-
Once you are satisfied with your changes, click the Generate HDL. Save the design if prompted.
-
Compile the design with the nodefault option to ensure that the source files are used during compilation. You may specify OFSS files for any component other than the HSSI if desired.
-
If you are not using any other OFSS files in your compilation flow (i.e. only the source files configuration will be used), use the following command to compile. It is recommended to compile a flat design before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss nodefault n6001:flat work_n6001\n
- If you are using OFSS files for other IP in the design, list them after
nodefault, but ensure that an HSSI OFSS file is not specified. It is recommended to compile a flat design first before incorporating a PR region in the design. This reduces design complexity while you modify the FIM.
./ofs-common/scripts/common/syn/build_top.sh --ofss nodefault, <pcie_ofss_file>,<memory_ofss_file>,<iopll_ofss_file>,<base_ofss_file> n6001:flat work_n6001\n
-
The resulting FIM will contain the Ethernet-SS configuration contained in the hssi_ss.ip source file.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#49-modifying-the-hps","title":"4.9 Modifying the HPS","text":"This section describes ways to modify the HPS.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#491-walkthrough-remove-the-hps","title":"4.9.1 Walkthrough: Remove the HPS","text":"Perform the following steps to remove the HPS from the FIM design.
Note: The n6000 default FIM has the HPS and Memory-SS disabled by default. This walkthrough targets the n6001.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Create a Memory Sub-system IP presets file with the connection to the HPS removed.
-
Open the the Memory Sub-System IP
qsys-edit $OFS_ROOTDIR/ipss/mem/qip/mem_ss/mem_ss.ip\n
-
In the IP Parameter Editor window that opens, remove the entries corresponding to the HPS (row #4) in the Memory Interfaces and Application Interfaces sections. To do this, click the row to be removed, then click the minus (-) button.
-
In the Presets pane, click New... to create a new IP presets file.
-
In the New Preset window set the preset name to n6001.
-
Click the ... button to select the save location of the IP presets file. In the Save As window, set the Look In field to the memory IP presets directory $OFS_ROOTDIR/ipss/mem/qip/presets. Set the File Name field to overwrite the existing mem_presets.qprs file. Click OK.
-
Click Save in the New Preset window. Click No when prompted to add the file to the IP search path.
-
Close IP Parameter Editor without saving or generating HDL.
-
Edit $OFS_ROOTDIR/syn/board/n6001/syn_top/ofs_top.qsf to comment out the INCLUDE_HPS and INCLUDE_UART macro definitions.
#set_global_assignment -name VERILOG_MACRO \"INCLUDE_HPS\"\n#set_global_assignment -name VERILOG_MACRO \"INCLUDE_UART\"\n
-
Build the FIM.
./ofs-common/scripts/common/syn/build_top.sh -p n6001 work_n6001_no_hps\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#491-walkthrough-add-the-hps","title":"4.9.1 Walkthrough: Add the HPS","text":"Note: The n6001 default FIM has the HPS and Memory-SS enabled by default. This walkthrough targets the n6000.
Pre-requisites:
- This walkthrough requires a development environment. Refer to the Set Up Development Environment Section for instructions on setting up a development environment.
Steps:
-
Clone the OFS PCIe Attach FIM repository (or use an existing cloned repository). Refer to the Clone FIM Repository section for step-by-step instructions.
-
Set development environment variables. Refer to the Set Development Environment Variables section for step-by-step instructions.
-
Edit $OFS_ROOTDIR/syn/board/n6000/syn_top/ofs_top.qsf to un-comment the INCLUDE_DDR4, INCLUDE_HPS, and INCLUDE_UART macro definitions.
set_global_assignment -name VERILOG_MACRO \"INCLUDE_DDR4\"\n\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_HPS\"\nset_global_assignment -name VERILOG_MACRO \"INCLUDE_UART\"\n
-
Build the FIM.
./ofs-common/scripts/common/syn/build_top.sh -p work_n6000_w_hps\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#5-fpga-configuration","title":"5. FPGA Configuration","text":"Configuring the Agilex FPGA on the n6001 and n6001 can be done by Remote System Update (RSU) using OPAE commands, or by programming a SOF image to the FPGA via JTAG using Quartus Programer.
Programming via RSU will program the flash device on the board for non-volatile image updates. Programming via JTAG will configure the FPGA for volatile image updates.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#51-walkthrough-set-up-jtag","title":"5.1 Walkthrough: Set up JTAG","text":"Perform the following steps to set up a JTAG connection to the Intel\u00ae FPGA SmartNIC N6001-PL/Intel\u00ae FPGA SmartNIC N6001-PL.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
- This walkthrough requires an Intel FPGA Download Cable II.
Steps:
-
Set the board switches to dynamically select either the Agilex\u00ae 7 FPGA or MAX\u00ae 10 device on the JTAG chain.
- Set SW1.1=ON as shown in the next image. The switches are located at the back of the Intel\u00ae FPGA SmartNIC N6001-PL.
-
The Intel\u00ae FPGA SmartNIC N6001-PL has a 10 pin JTAG header on the top side of the board. Connect an Intel FPGA Download II Cable to the JTAG header of the Intel\u00ae FPGA SmartNIC N6001-PL as shown in picture below. This picture shows the Intel\u00ae FPGA SmartNIC N6001-PL card installed in the middle bay, top slot of a SuperMicro\u00ae SYS-220HE-FTNR server where the lower slot does not have card installed allowing the Intel\u00ae FPGA Download II cables to pass through removed the slot access.
Note: If using the Intel FGPA download Cable on Linux, add the udev rule as described in Intel FPGA Download Cable Driver for Linux.
-
Set the JTAG chain to select the Agilex\u00ae 7 FPGA as the target by writing to the JTAG enable register in the BMC (Register 0x378). This is done via PMCI registers 0x2040C and 0x20400.
Note: The commands below are targeted to a board with PCIe B:D.F of 98:00.0. Use the correct PCIe B:D.F of your card.
sudo opae.io init -d 0000:98:00.0 $USER\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x2040c 0x100000000\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:98:00.0\n
Note: To later re-direct the JTAG back to the MAX 10 device, execute the following commands.
sudo opae.io init -d 0000:b1:00.0 $USER\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x2040c 0x000000000\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:b1:00.0\n
Optionally, rather than dynamically commanding Agilex\u00ae 7 FPGA/MAX10 selection with the PMCI register settings, you can fix the selection with the following switch settings shown in the table below:
SW1.1 SW2 JTAG Target OFF OFF Agilex\u00ae 7 FPGA OFF ON MAX\u00ae 10 FPGA ON X Agilex\u00ae 7 FPGA if BMC register 0x378=0x1 ON X MAX\u00ae 10 FPGA if BMC register 0x378=0x0 -
Use the jtagconfig tool to check that the JTAG chain contains the AGFB014R24A2E2V device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#52-walkthrough-program-the-fpga-via-jtag","title":"5.2 Walkthrough: Program the FPGA via JTAG","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL/Intel\u00ae FPGA SmartNIC N6000-PL with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the n6001. Refer to the Set up JTAG section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae FPGA SmartNIC N6001-PL is connected via JTAG. The version of the programmer must be the same as the Quartus version used to build the FIM.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the n6001, if different than your deployment machine.
-
Open the Quartus programmer GUI
Note: the Quartus programmer version must be the same as the version of Quartus used to build the design.
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the n6001.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB014R24A2E2V device. The JTAG chain should show the divice.
-
Right click the AGFB014R24A2E2V row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGFB014R24A2E2V row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI. You can answer 'No' if a dialog pops up asking to save the 'Chain.cdf' file
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#53-remote-system-update","title":"5.3 Remote System Update","text":"The OPAE fpgasupdate tool can be used to update the Max10 Board Management Controller (BMC) image and firmware (FW), root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate tool only accepts images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then you must also sign the image using the correct keys. Refer to the Security User Guide: Open FPGA Stack for information on created signed images and on programming and managing the root entry hash.
The Intel\u00ae FPGA SmartNIC N6001-PL ships with a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL on all cards. The platform ships with a single FIM image that can be programmed into either user1 or user2, depending in the image selected.
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#531-walkthrough-program-the-fpga-via-rsu","title":"5.3.1 Walkthrough: Program the FPGA via RSU","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL with a BIN image via RSU.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for instructions on setting up a deployment environment.
- This walkthrough requires a
BIN image which will be programmed to the Agilex FPGA. Refer to the Compile OFS FIM Section for step-by-step instructions for generating a BIN image. The image used for programming must be formatted with PACsign before programming. This is done automatically by the build script.
Steps:
-
Start in your deployment environment.
-
Determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Use the OPAE fpgasupdate command to program a PACsign signed image to flash. The flash slot which will be programmed is determined by the PACsign header.
sudo fpgasupdate <IMAGE> <PCIe B:D.F>\n
-
Example: update User Image 1 in flash
sudo fpgasupdate ofs_top_page1_unsigned_user1.bin 98:00.0\n
-
Example: update User Image 2 in flash
sudo fpgasupdate ofs_top_page2_unsigned_user2.bin 98:00.0\n
-
Example: update Factory Image in flash
sudo fpgasupdate ofs_top_page0_unsigned_factory.bin 98:00.0\n
-
Use the OPAE rsu command to reconfigure the FPGA with the new image. You may select which image to configure from (User 1, User 2, Factory).
sudo rsu fpga --page <PAGE> <PCIe B:D.F>\n
-
Example: configure FPGA with User 1 Image
sudo rsu fpga --page user1 98:00.0\n
-
Example: configure FPGA with User 2 Image
sudo rsu fpga --page user2 98:00.0\n
-
Example: configure FPGA with Factory Image
sudo rsu fpga --page factory 98:00.0\n
"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#appendix","title":"Appendix","text":""},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#appendix-a-resource-utilization-tables","title":"Appendix A: Resource Utilization Tables","text":"Table A-1 Default Out-of-Tree FIM Resource Utilization
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 207715.9 42.63 689 9.69 PCIE_RST_CTRL.rst_ctrl 476.7 0.1 0 0.0 afu_top 145973.9 29.96 331 4.66 auto_fab_0 2839.6 0.58 12 0.17 bpf 1286.6 0.26 0 0.0 fme_top 615.7 0.13 6 0.08 hps_ss 0.0 0.0 0 0.0 hssi_wrapper 22143.5 4.55 131 1.84 local_mem_wrapper 8456.9 1.74 60 0.84 pcie_wrapper 18449.2 3.79 114 1.6 pmci_wrapper 6225.3 1.28 27 0.38 qsfp_0 622.5 0.13 4 0.06 qsfp_1 622.7 0.13 4 0.06 sys_pll 0.5 0.0 0 0.0 Table A-2 Minimal FIM Resource Utilization
Compilation Hierarchy Node ALMs needed ALM Utilization % M20Ks M20K Utilization % 90688.0 18.61 410 5.77 PCIE_RST_CTRL.rst_ctrl 384.0 0.08 0 0.0 afu_top 49696.4 10.2 197 2.77 auto_fab_0 1742.0 0.36 8 0.11 bpf 1380.2 0.28 0 0.0 fme_top 667.2 0.14 6 0.08 hps_ss 0.0 0.0 0 0.0 hssi_dummy_csr 688.1 0.14 0 0.0 local_mem_wrapper 9020.6 1.85 60 0.84 pcie_wrapper 18899.2 3.88 112 1.58 pmci_wrapper 6838.5 1.4 27 0.38 qsfp0_dummy_csr 688.5 0.14 0 0.0 qsfp1_dummy_csr 682.4 0.14 0 0.0 sys_pll 0.4 0.0 0 0.0"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#appendix-b-glossary","title":"Appendix B: Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/","title":"Hard Processor System Software Developer Guide: OFS for Intel Agilex FPGAs Targeting Intel\u00ae N6000/1-PL FPGA SmartNIC Platform","text":"Quartus Prime Pro Version: 23.1
Last updated: Last updated: November 19, 2024
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#glossary","title":"Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel Max10 or Intel Cyclone10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implemented on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Management Component Transport Protocol MCTP A standardized model for communication with management controllers. Defines the transport protocol carrying PLDM messages through the BMC. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE-SDK The OPAE-SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Level Data Model PLDM A specification for reporting telemetry data to the host, such as board temperature, voltage, and current. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#file-types","title":"File Types","text":"Extension Description ITS File (*.its) The image source file which describes the contents of the image and defines various properties used during boot. Actual contents to be included in the image (kernel, ramdisk, etc.) are specified in the image source file as paths to the appropriate data files. ITB File (*.itb) Produced as output from mkimage, using an image source file. Contains all the referenced data (kernel, ramdisk, SSBL, etc.) and other information needed by UBoot to handle the image properly. This image is transferred to the target and booted. DTB File (*.dtb) The Device Tree Blob is loaded into memory by U-Boot during the boot process, and a pointer to it is shared with the kernel. This file describes the system's hardware layout to the kernel. FIT Image (*.fit) Format used for uImage payloads developed by U-Boot. On aarch64 the kernel must be in image format and needs a device tree to boot. SPL (*.spl) The Secondary Program Loader is a small binary which is embedded in a FIM SOF and loaded into board DDR4 RAM when the FPGA is initially configured. This file is responsible for loading U-Boot into system RAM."},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#10-introduction","title":"1.0 Introduction","text":"The Open FPGA Stack (OFS) is a modular collection of hardware platform components, open source upstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS Provides a framework of FPGA synthesizable code, a simulation environment and synthesis/simulation scripts. The updated OFS architecture for Intel Agilex FPGA devices improves upon the modularity, configurability and scalability of the first release of the OFS architecture while maintaining compatibility with the original design. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem
- HSSI Subsystem
- Memory Subsystem
- Hard Processor System (HPS)
- Reset Controller
- FPGA Management Engine (FME)
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- SPI Interface to BMC controller
The Intel\u00ae N6000-PL and N6001-PL FPGA SmartNIC Platforms are acceleration cards that use the OFS infrastructure. The key difference between these two platforms is:
- Intel\u00ae N6000-PL SmartNIC Platform has a bifurcated PCIe bus with Gen4x8 interfacing to the the Intel Agilex FPGA and Gen4x8 interfacing to an Intel E810 SmartNIC. This platform is targeted specifically for VRAN, UPF and vCSR applications. The FPGA designs targeting these vertical market applications were generated using the OFS infrastructure.
- Intel\u00ae N6001-PL SmartNIC Platform has a Gen4x16 interface directly to the Intel Agilex FPGa and is not populated with an Intel E810 SmartNIC. This platform is the reference platform for the OFS reference designs for Intel Agilex FPGA.
Note: throughout this document \"Intel N6000/1-PL FPGA SmartNIC Platform\" denotes both cards. This document describes the software package that runs on the Hard Processor System (HPS) which is a key component within both platforms.
The Intel N6000/1-PL FPGA SmartNIC Platform has a customized build script that can be used to both set up a development environment and build the essential pieces of the HPS software image. This script, meta-bake.py, has its own dedicated Section 3.1 Building U-Boot which can be used to quickly get started with the HPS build flow. It is recommended you use this build script to construct the first and second stage bootloader files, as it will handle all of the setup and patching required to build out your complete Yocto image. You can familiarize yourself with the contents of this package in its public GitHub repository located at https://github.com/OPAE/meta-opae-fpga/tree/main/tools/meta-bake. All other information included for individual components is included for learning purposes only and is not meant as a recipe for image creation.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#11-reference-documents","title":"1.1 Reference Documents","text":"This document pulls much of its information from related Agilex FPGA documentation hosted on intel.com. Reading these references is not required for initial platform bring up, but will serve to further your knowledge of the FPGA SoC boot and configuration process.
Table 1. Reference Documents
Document Title Intel\u00ae Agilex\u2122 Hard Processor System Technical Reference Manual Intel\u00ae Agilex\u2122 SoC FPGA Boot User Guide Intel\u00ae Agilex\u2122 Configuration User Guide"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#12-reference-images","title":"1.2 Reference Images","text":"Intel has provided a set of two pre-compiled ITB images that can be used for exploration and evaluation of the HPS bring-up flow. These images contain the complete SSBL package specific to the board and can be copied to the N6000/1-PL SmartNIC Platform with an HPS enabled FIM loaded. Refer to Section 4.1 Example Boot for an example on how to use the built-in copy engine IP in tandem with the host-side cpeng software to transfer an SSBL.
The package is found on the official OFS 2024.2-1 Release on GitHub. Two ITB artifacts are included at the bottom of the page under Assets - one with the Vendor Authorized Boot (VAB) certificate included, and one without. Which you choose to load depends on whether the currently loaded FIM requires VAB authentication. Section 4.3 Example Boot contains instructions on the boot flow using these files for platform bring up.
The default username for these two images is root and the password is empty. A good place to start after loading the ITB is to set up SSH for file transfers and the remote console, as seen in 8.0 Connecting remotely to the HPS using ssh.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#20-architecture-overview","title":"2.0 Architecture Overview","text":"The OFS architecture is classified into:
- 1. Host Interface Adapters (PCIe)
- 2. Low Performance Peripherals
- 2.1. Slow speed peripherals (example: JTAG, I2C, SMBus, and so on)
- 2.2. Management peripherals (example: FPGA FME)
- 3. High Performance Peripherals
- 3.1. Memory peripherals
- 3.2. Acceleration Function Units (AFUs)
- 3.3. HPS Peripheral
- 4. Fabrics
- 4.1. Peripheral Fabric (multi drop)
- 4.2. AFU Streaming fabric (point to point)
The HPS is connected to the AFU and implements all the board specific flows that customers require to begin the application development using the HPS such as host communication, firmware load and update, integration with OFS, and memory. The HPS implements a basic Hello World software application and is intended as a starting point for customers to begin development with HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#21-hps-peripherals","title":"2.1 HPS Peripherals","text":"Figure 1 Intel Agilex FPGA HPS Peripherals
The Intel Agilex\u2122 SoC integrates a full-featured Arm\u00ae Cortex-A53\u00ae MPCore Processor.
The Cortex-A53 MPCore supports high-performance applications and provides the capability for secure processing and virtualization.
Each CPU in the processor has the following features:
- Support for 32-bit and 64-bit instruction sets.
- To pipeline with symmetric dual issue of most instructions.
- Arm NEON\u00ae Single Instruction Multiple Data (SIMD) co-processor with a Floating-Point Unit (FPU)
- Single and double-precision IEEE-754 floating point math support
- Integer and polynomial math support.
- Symmetric Multiprocessing (SMP) and Asymmetric Multiprocessing (AMP) modes.
- Armv8 Cryptography Extension.
- Level 1 (L1) cache:
- 32 KB two-way set associative instruction cache.
- Single Error Detect (SED) and parity checking support for L1 instruction cache.
- 32 KB four-way set associative data cache.
- Error checking and correction (ECC), Single Error Correct, Double Error Detect (SECDED) protection for L1 data cache.
- Memory Management Unit (MMU) that communicates with the System MMU (SMMU).
- Generic timer.
- Governor module that controls clock and reset.
- Debug modules:
- Performance Monitor Unit.
- Embedded Trace Macrocell (ETMv4).
- Arm CoreSight\u00ae cross trigger interface, the four CPUs share a 1 MB L2 cache with ECC, SECDED protection.
A Snoop Control Unit (SCU) maintains coherency between the CPUs and communicates with the system Cache Coherency Unit (CCU). At a system level, the Cortex-A53 MPCore interfaces to a Generic Interrupt Controller (GIC), CCU, and System Memory Management Unit (SMMU).
Beyond the Arm Cortex-A53 MPCore Processor, the HPS integrates a variety of useful peripherals for use in your design, such as Ethernet, USB, Memory Controller, on-chip RAM, SPI, UART and more. Refer to the Intel\u00ae Agilex\u2122 Hard Processor System Technical Reference Manual for more information.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#22-zarlink-device","title":"2.2 Zarlink Device","text":"The Microchip\u00ae Zarlink device ZL30793 is used for time synchronization. It acts as the protocol engine that drives IEEE 1588-2008 PTP protocol. The Zarlink device is connected to the HPS side and the programming interface is SPI. The FPGA bitstream containing the HPS has the First Stage Bootloader (FSBL) only. This enable commands to be given from a terminal program connected through UART.
The software in HPS can access the Clock generator through SPI to enable write and read operations controlled by the terminal program. It can also read the status of the hold over and Loss of Lock signals and control the LED.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#23-copy-engine","title":"2.3 Copy Engine","text":"A host with OPAE SDK and Linux DFL installed will provide the hps OPAE command with related options to transfer images from host to the HPS image. The module in the OFS FIM and HPS software that performs this transfer is called the Copy Engine (CPE), which is included by default within the HPS image.
Refer to the Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL) for platform and software installation instructions.
The CPE software is patched into Linux on the HPS in Yocto through the meta-intel-fpga-refdes layer. This service is daemonized and requires systemd in order to operate. This service will communicate with the HPS IP integrated in the FIM in order to coordinate and monitor file transfers from the host CPE software to DDR connected the HPS. The CPE HPS-side software takes advantage of the built-in I/O lightweight kernel module to communicate with the FIM's HPS IP. It can restart the transfer if the initial transfer of the image is not successful. The CPE can also serve as reference on how to integrate your own systemd service in the Linux build running on the HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#24-boot-flow","title":"2.4 Boot Flow","text":"The boot flow for the Agilex OFS design for the Intel N6000/1-PL FPGA SmartNIC Platform is an FPGA-first boot flow, meaning the Intel Agilex Secure Device Manager (SDM) configures the FPGA first and then boots the HPS. Alternatively, you can boot the HPS first and then configure the FPGA core as part of the Second-Stage Boot Loader (SSBL) or after the Operating System (OS) boots. HPS-first boot is not covered in this document, but for more information please refer to the Intel\u00ae Agilex\u2122 SoC FPGA Boot User Guide.
For the FPGA-first boot flow supported by the Intel Agilex OFS FIM, the First Stage Bootloader is part of FPGA bitstream. The available Secure Device Manager (SDM) in the FPGA initially configures the FPGA core and periphery in this mode. The first stage bootloader is produced as a part of a U-Boot build and cna be manually inserted into a Quartus generated SOF file as shown in step 7 of Section 9.2 Configuring the HPS.
After completion, the HPS boots. All the I/O, including the HPS-allocated I/O, are configured, and brought out of tri-state. If the HPS is not booted:
- The HPS is held in reset
- HPS-dedicated I/O are held in reset
- HPS-allocated I/O are driven with reset values from the HPS.
- If the FPGA is configured before the HPS boots, then the boot flow looks as shown in the example figure below.
Figure 2. Typical FPGA First Configuration Steps
The flow includes the Time from Power-on-Reset (TPOR) to boot completion (TBoot_Complete).
Table 2. FPGA Configuration First Stages
Time Boot Stage Device State TPOR to T1 POR Power-on reset T1 to T2 SDM: Boot ROM 1. SDM samples the MSEL pins to determine the configuration scheme and boot source. 2. SDM establishes the device security level based on eFuse values. 3. SDM initializes the device by reading the configuration firmware (initial part of the bitstream) from the boot source. 4. SDM authenticates and decrypts the configuration firmware (this process occurs as necessary throughout the configuration). 5. SDM starts executing the configuration firmware. T2 to T3 SDM: Configuration Firmware 1. SDM I/O are enabled. 2. SDM configures the FPGA I/O and core (full configuration) and enables the rest of your configured SDM I/O. 3. SDM loads the FSBL from the bitstream into HPS on-chip RAM. 4. SDM enables HPS SDRAM I/O and optionally enables HPS debug. 5. FPGA is in user mode. 6. HPS is released from reset. CPU1-CPU3 are in a wait-for-interrupt (WFI) state. T3 to T4 First-Stage Boot Loader (FSBL) 1. HPS verifies the FPGA is in user mode. 2. The FSBL initializes the HPS, including the SDRAM. 3. The user application through the host must request the copy engine using the OPAE command hps to transfer the itb image (SSBL +Linux) to the HPS DRAM. 4. HPS peripheral I/O pin mux and buffers are configured. Clocks, resets, and bridges are also configured. 5. HPS I/O peripherals are available. T4 to T5 Second-Stage Boot Loader (SSBL) 1. HPS bootstrap completes. 2. OS is loaded into SDRAM. T5 to TBoot\\Complete Operating System (OS) The OS boots and applications are scheduled for runtime launch. When using the Pre-boot Execution Environment (PXE) boot mechanism, you must use an option ROM. OFS FIM does not have PXE boot implemented in the HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#25-authorization","title":"2.5 Authorization","text":"The HPS FSBL is part of the static region (SR) FPGA bitstream. Intel provides the capability to sign the FPGA bitstream binaries so that they can be authenticated when remotely updated and when configuring the FPGA. Signing of the SR bitstream is a two-stage process where you must sign with:
1. `quartus_sign` tool\n2. OPAE `PACSign` tool\n
Signing with PACSign ensures the security of the BMC RSU update process to the flash, and requires a compatible binary file. Quartus signing provides ensures security when the bitstream is configured through the SDM into the Intel Agilex FPGA device using Vendor Authorized Boot.
Vendor Authorized Bootloader (VAB) considers the SDM as a trusted root entity such that when firmware is authenticated and booted and is running on the SDM with dedicated crypto HW IP blocks, it is considered a trusted entity. As such it is trusted to perform the authentication and authorization steps for subsequent bitstreams.
Each subsequent loaded object after the SDM boot firmware does not need to re-implement the authentication and authorization functions. The authentication and authorization functions are centralized. Arm Trusted Firmware (ATF) is used to make a trusted execution environment (TEE) in the HPS. The source code for both Arm Trusted firmware and the First Stage Boot Loader (FSBL) is provided in the GitHub.
The SSBL + Linux is a part of an itb file and may also be signed with Quartus_sign and PACSign for authentication. This process is demonstrated in Section 9.2 Configuring the HPS.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#30-environment-setup","title":"3.0 Environment Setup","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#31-building-u-boot","title":"3.1 Building U-Boot","text":"When creating a development environment using meta-bake.py both U-Boot and the patches required to work with the Intel N6000/1-PL FPGA SmartNIC Platform are located at /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig. To review the required patches applied to U-Boot, navigate to /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/git/patches. From there, using git commands such as git status and git branch will show changes to the build environment.
Currently the meta-bake build flow requires a specific environment and software dependencies. Refer to section 6.1 Running meta-bake.py for more information. Download the script from meta-opae-fpga.
Invoke the meta-bake.py build script to build your entire image, including U-Boot.
$ cd /meta-opae-fpga/tools/meta-bake\n$ ./meta-bake.py --conf n6000/layers.yaml builddir\n
This build process is highly system dependent and can take upwards of 1 hour to complete. Make sure you have at least 200 GB of free space on the system before running a build.
To build U-Boot manually after execution of meta-bake.py navigate to /meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-srcfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig and run make. After running meta-bake.py, you can rebuild U-Boot to incorporate any changes you have made. Navigate to the U-Boot directory at /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig and run the following commands to rebuild.
$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\n$ tar xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\n$ rm gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\n$ export CROSS_COMPILE=`pwd`/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-\n$ export ARCH=arm64\n$ make -j `nproc`\n
This recompile will result in a new ITB SSBL which may be loaded on an Intel FPGA SmartNIC N6000/1 platform. Several components of the ITB image are present under the U-Boot directory but are not rebuilt as a part of this flow. These files will need to be replaced before rebuilding U-Boot for changes to take affect.
U-Boot comes with its own dumpimage tool, which can be used to identify an image and extract and identify its contents. This tool is built by default under /u-boot-socfpga/tools, and in the meta-bake.py environment setup under /meta-opae-fpga/tools/meta-bake/build/agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig/tools. This tool can also be used to extract specific components of the ITB file.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#32-yocto","title":"3.2 Yocto","text":"Yocto is an open source toolkit used to create Linux distributions and commonly used for creating Linux images and bootloaders for embedded environments. A Yocto build environment is made up of one or more layers, with each layer consisting of recipes that get processed to build/install components in a layer. The workhorse of a Yocto build is the program called bitbake. This program processes the recipes to compile and build packages and images for the target platform. For SoC platforms, like the HPS, the ARM cross compiler is required.
The build script used for the Agilex SoC GSRD, create-linux-distro-release, is a bash script that automates the build of Linux images of different types (gsrd, nand, etc.) that are compatible with a target FPGA platform (agilex, stratix10, etc.). This script has been ported to Python 3 and modified to build an environment for the Intel FPGA SmartNIC N6000/1 platform, named meta-bake.py. This script pulls in the necessary patches and additional changes needed to support the platform.
In general, meta-bake.py pulls Yocto layers/recipes from public repositories, configures a Yocto build environment, and builds an image for a supported FPGA platform. The Yocto layer is always the first to be built, and includes the bitbake utility. The following table lists remote repositories hosting Yocto meta data source used by meta-bake.py and create-linux-distro as well as source code used for building binaries that make up the Linux image (kernel and rootfs).
Note: Not all repositories can be viewed in a web browser. All can be cloned using git.
Repository Description https://git.yoctoproject.org/git/poky Base build tools and meta data layers https://git.openembedded.org/meta-openembedded Layers for OE-core packages https://git.yoctoproject.org/git/meta-intel-fpga Meta data layers for Intel FPGA SoC platforms https://github.com/altera-opensource/meta-intel-fpga-refdes BSP layer for Intel SoC FPGA GSRD https://github.com/altera-opensource/linux-socfpga Linux kernel source repository for socfpga https://github.com/altera-opensource/u-boot-socfpga U-Boot bootloader source repository for socfpga https://github.com/altera-opensource/arm-trusted-firmware Source for ATF Recipes in the meta-intel-fpga-refdes layer mostly inherit from and extend recipes in other layers. The following table lists the new or modified recipes (in meta-intel-fpga-refdes) necessary to support an Intel FPGA SmartNIC N6000/1 HPS boot image.
Component Recipe Description Linux Kernel recipes-kernel/linux/linux-socfpga-lts_5.10.bbappend Recipe to append the GSRC SoC FPGA device tree to the Yocto build U-Boot recipes-bsp/u-boot/u-boot-socfpga_v2021.07.bbappend Recipe to fetch and build socfpga U-Boot. Modified to support N6000/1 in U-Boot. This also creates a shell script, *mkuboot-fit.sh. copy-engine recipes-bsp/copy-engine/copy-engine-0.1.bb New recipe to build copy-engine daemon in rootfs. N6000/1 Image recipes-images/poky/n6000-image-minimal.bb New recipe to create the N6000/1 image with copy-engine and linuxptp packages installed. mkuboot-fit.sh is meant to be called after a Yocto build to create the U-Boot FIT image for N6000/1, and is called automatically by meta-bake.py. This is a workaround for the Yocto build order which builds the bootloader (U-Boot) before building the Linux image rootfs. Because the rootfs is part of the U-Boot FIT image, the rootfs must be built before building the bootloader. The result of calling this script is copying the rootfs (as a .cpio file) to the U-Boot source file tree and calling make in the U-Boot build tree. When called again with the rootfs present, the resulting image will contain the rootfs. This is performed automatically as a part of the meta-bake.py build flow.
See here for more information regarding Yocto. Several reference designs found in rocketboards.org use Yocto for building the Linux image and/or bootloader. For the N6000/1 image and boot flow, the Yocto build script for the Agilex SoC Golden System Reference Design has been adapted to automate building the boot loader, Linux Image, and filesystem needed to support N6000/1 devices.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#321-customizing-the-yocto-image","title":"3.2.1 Customizing the Yocto Image","text":"The following is a list of customizations made for building Yocto to run on the Intel FPGA SmartNIC N6000/1-PL platform.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3211-extending-the-u-boot-recipe","title":"3.2.1.1 Extending the U-Boot recipe","text":"A recipe extension file (recipes-bsb/u-boot/u-boot-socfpga_v2021.07.bbappend) has been added to the meta-intel-fpga-refdes layer which accomplishes the following:
- Adds patches using Yocto's patching mechanism
- Introduces a new U-Boot config, socfpga_agilex_n6000_defconfig, and associates it with a keyword,
agilex-n6000, that can be referenced in Yocto configuration files. These patches are necessary until those changes are merged into the public u-boot-socfpga repository. This config works for both Smartnic Platforms. - Creates mkuboot-fit.sh script file with variables for U-Boot source and build directories that will get expanded to the actual paths that Yocto uses for fetching/building U-Boot. Along with this recipe file, relevant patch files have been added. Once the changes are in the U-Boot repository, the patches and any references to them must be removed.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3212-patching-the-linux-kernel","title":"3.2.1.2 Patching The Linux Kernel","text":"The kernel extension recipe, meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts_5.10.bbappend, in the meta-intel-fpga-refdes layer, has been modified to add a patch file using Yocto's patching mechanism. This patch file adds the device tree for N6000/1 and is only necessary until this change is merged into the public linux-socfpga repository.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3213-adding-custom-user-space-software","title":"3.2.1.3 Adding Custom User Space Software","text":"A new recipe, meta-intel-fpga-refdes/recipes-bsp/copy-engine-0.1.bb and relevant source files, have been added to the meta-intel-fpga-refdes layer. This recipe includes instructions for building the copy-engine program as well as installing it as a systemd service. Yocto will build this into an RPM package that gets installed into any image that includes it in the IMAGE_INSTALL variable. This recipe may be used as a guide for installing additional user space software.
You may also create a new Hello World application and add it to the Yocto build as shown below.
- Generate a BSD 3-Clause License and create an MD5 hash of it.
Copyright (c) 2023, User's Name All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Company Name nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
After license creation you will need to create an MD5 Hash of the clause. On Linux you can either pipe the raw text into echo \"text\" | md5sum, or create a new file and point to it md5sum licensefile.txt.
- Create a BB recipe file, following the same directory structure as other Yocto recipes.
$ cd /meta-opae-fpga/tools/meta-bake/build/meta-intel-fpga-refdes\n$ mkdir -p recipe-example/helloworld && cd recipe-example/helloworld\n
Create recipe file helloworld.bb in directory helloworld.
SUMMARY = \"Example hello world\" DESCRIPTION = \"helloworld in HPS\" AUTHOR = \"Your Name <your.email@address.com>\" LICENSE = \"BSD-3-Clause\" LIC_FILES_CHKSUM = \"file://${COMMON_LICENSE_DIR}/BSD-3-Clause;md5=<Your MD5 Hash>\" inherit systemd pkgconfig SRC_URI = \"file://helloworld.c\" S = \"${WORKDIR}\" do_compile() { ${CC} ${CFLAGS} ${LDFLAGS} helloworld.c -o helloworld }\ndo_install() { install -d ${D}${bindir} install -m 0755 helloworld ${D}${bindir} }
- Create source file
helloworld.c in the same helloworld directory.
#include <stdio.h> void main void() { Printf(\u201c\\nHello World\\n\u201d) }
- Re-run
./meta-bake.py --conf n6000/layers.yaml <Build Directory>. This will a new programmable SSBL that contains your Hello World program. Program the resulting ITB file as shown in Section 4.3 Example Boot and verify the application has been included in your build.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3214-adding-kernel-driver-software","title":"3.2.1.4 Adding Kernel Driver Software","text":"New recipes for custom kenel modules can be created at /build/meta-intel-fpga-refdes/recipes-kernel/linux/, and instructed to include custom module code. These can be patched in, included as a part of a new branch, or included by default if upstreamed. For more information visit the YoctoProject's Linux Kernel Development Manual. An example file from N6000/1 that can be used as an example is /build/meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts_5.10.bbappend.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3215-creating-an-image-type","title":"3.2.1.5 Creating an Image Type","text":"A new recipe, meta-intel-fpga-refdes/recipes-images/poky/n6000-image-minimal.bb, has been added that includes directives to install the copy-engine package (built in this layer) as well as the linuxptp package (available in other layers). In addition to including these packages, this recipe includes a rootfs post processing command that removes the Linux kernel image files from the rootfs. This is done because the Linux kernel is part of the U-Boot FIT image and therefore not used from the rootfs. Removing this redundant file reduces the final U-Boot FIT image by about 30Kb. This recipe may be modified or used as a guide to add additional user space software.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#3216-testing-and-debugging","title":"3.2.1.6 Testing and Debugging","text":"As mentioned previously, the script will erase source files every time it is executed. This means that any changes made locally will be lost when the script is run again after making these changes. The example below shows how to test local changes without executing the script again.
$ cd build\n$ source poky/oe-init-build-env agilex-gsrd-rootfs/\n$ bitbake n6000-image-minimal\n$ ./agilex-n6000-rootfs/tmp/deploy/images/agilex/mkuboot-fit.sh\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#40-booting-the-hps","title":"4.0 Booting the HPS","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#41-ofs-fim-boot-overview","title":"4.1 OFS FIM Boot Overview","text":"This implementation of an FPGA First boot flow requires that the FSBL poll on a given register before continuing to boot the HPS. Once this register indicates it is (copy engine) ready, the FSBL loads a monolithic U-Boot FIT image at a given offset 0x02000000.
This image is made up of the following components:
- U-Boot bootloader also referred to as second stage bootloader
- Linux Kernel image
- Root filesystem (rootfs) consisting of kernel modules as well as user space software.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#42-booting-ofs-fim-for-intel-agilex-fpga","title":"4.2 Booting OFS FIM for Intel Agilex FPGA","text":"As mentioned before, the Intel N6000/1-PL FPGA SmartNIC boot flow is an FPGA-first boot flow which requires that the Intel Agilex FPGA to be configured with the necessary components (SPL/FSBL, copyengine) in order for the HPS to boot.
SD/eMMC is not supported for FSBL for HPS first
- First Stage Bootloader (FSBL): u-boot-spl-dtb.hex is embedded into FPGA image
- Monolithic FIT Image Downloaded from Host, u-boot.itb contains the following components
1) Second Stage Bootloader (SSBL): U-Boot + U-Boot Device Tree
2) Linux kernel, Image, + Linux Device Tree
3) Linux RAM based Root File System
- First Stage Bootloader (FSBL) is U-boot-spl
- U-boot-spl is built when U-Boot is built
- Artifact is u-boot-spl-dtb.hex
- The user has to check into build location : ofs-n6001/syn/setup/vab_sw/u-boot-spl-dtb.hex
- Then run the command
- quartus/pfg -c -o hps/path=u-boot-spl-dtb.hex orig.sof orig/fsbl.sof
Things to consider while developing your ITB image:
- The size of the u-boot.itb matters.\n- FIT is downloaded to [0x2000000](https://github.com/altera-opensource/u-boot-socfpga/blob/541b6afcb183ddb350ad367c9b63cc6db94c1f6e/configs/socfpga_agilex_n6010_defconfig#L4)\n- Linux Device Tree and RootFS are unpacked to high memory\n- Linux is unpacked to an address specified in the FIT, [0xb600000](https://github.com/altera-opensource/u-boot-socfpga/blob/541b6afcb183ddb350ad367c9b63cc6db94c1f6e/arch/arm/dts/socfpga_agilex_n6010-u-boot.dtsi#L4)\n- If size of u-boot.itb is greater than 0xb600000 \u2013 0x2000000, then FIT will be corrupted mid extraction, resulting in unpredictable kernel crashes.\n
This example assumes the following preconditions have been met prior to booting HPS:
1) A SOF file synthesized with the SPL (u-boot-spl-dtb.hex).
2) Copy engine IP with relevant registers accessible to host and HPS.
Once the host FPGA boots with the required bitstream, the SPL in the HPS begins polling a register in the copy engine. One way to get an indication that the HPS is ready to continue past the SPL is to use a terminal emulator on a host with a serial cable connected to the FPGA's UART port. To transfer the U-Boot FIT image, use the hps cpeng subcommand from the host. Note, the hps program can be installed as part of installing the OPAE SDK and Linux DFL suite of packages.
hps command details are located in Section 5.0 HPS Command Usage.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#43-example-boot","title":"4.3 Example Boot","text":"This example assumes the following preconditions have been met prior to booting HPS:
- A SOF file synthesized with the SPL (u-boot-spl-dtb.hex).
- Copy engine IP with relevant registers accessible to host and HPS.
Once the host FPGA boots with the required bitstream, the SPL in the HPS will begin polling a register in the copy engine. One way to get an indication that the HPS is ready to continue past the SPL is to use a terminal emulator on a host with a serial cable connected to the FPGA's UART port.
To transfer the U-Boot FIT image, use the hps program with cpeng subcommand from the host. Note, the hps program is installed as part of installing the OPAE SDK suite of packages. See here for information on running the hps program. The following example assumes your N6000/1 board is at PCIe BDF 0000:b1:00.0.
# Bind vfio-pci driver to Copy Engine PCIe Physical Function\n$ sudo opae.io init -d b1:00.4 root\n# Load the HPS SSBL\n$ hps cpeng -f u-boot.itb\n[2021-09-25 01:59:25.538] [cpeng] [info] starting copy of file:u-boot.itb, size: 116725656, chunk size: 4096\n[2021-09-25 01:59:29.935] [cpeng] [info] last chunk 1944, aligned 2048\n[2021-09-25 01:59:29.935] [cpeng] [info] transferred file in 28498 chunk(s)\n[2021-09-25 01:59:29.935] [cpeng] [info] waiting for ssbl verify...\n[2021-09-25 01:59:33.848] [cpeng] [info] ssbl verified\n[2021-09-25 01:59:33.848] [cpeng] [info] waiting for kernel verify...\n[2021-09-25 01:59:39.626] [cpeng] [info] kernel verified\n
This will transfer the U-Boot FIT image via the copy engine IP to the HPS DDR and then signal completion of the transfer to the copy engine. Once the copy engine completes the actual transfer, it will write to the register the HPS SPL is polling on allowing the SPL to load the U-Boot bootloader which will in turn boot into the Linux image embedded in the U-Boot FIT image. If a terminal emulator is connected to the UART as described above, a user can observe U-Boot and Linux running on the HPS.
- Validate the HPS SSBL has been loaded by checking for its heartbeat.
$ hps heartbeat\n[2021-09-25 01:59:42.722] [heartbeat] [info] heartbeat value: 0x30015\n[2021-09-25 01:59:43.722] [heartbeat] [info] heartbeat value: 0x40015\n[2021-09-25 01:59:44.722] [heartbeat] [info] heartbeat value: 0x50015\n[2021-09-25 01:59:45.723] [heartbeat] [info] heartbeat value: 0x60015\n[2021-09-25 01:59:46.723] [heartbeat] [info] heartbeat value: 0x70015\n
- Login to HPS as user, root, with no password over serial connection. This process is covered in 8.0 Connecting remotely to the HPS using
ssh.
agilex login: root\nroot@agilex:~# ls\nroot@agilex:~# ls /\nbin dev home lib mnt root sbin sys usr\nboot etc init media proc run srv tmp var\nroot@agilex:~#\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#50-hps-command-usage","title":"5.0 HPS Command Usage","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#51-synopsis","title":"5.1 Synopsis","text":"hps OPTIONS SUBCOMMAND SUBCOMMAND\\_OPTIONS
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#52-description","title":"5.2 Description","text":"hps is an application to aid in the development, deployment, and debugging of an HPS (hard processor system) on an Intel Agilex device using OFS. The current version of the hps program assumes an AFU (accelerator functional unit) is configured into the FPGA that can be discovered/accessed through an OPAE library and used for communicating with the HPS. When invoked with one of its subcommands, hps will enumerate the AFU for that subcommand before executing it.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#53-options","title":"5.3 Options","text":"-h,--help\n\nPrint this help message and exit\n\n-p,--pci-address address\n\nUse address in the filter for locating the AFU where address must be in\n\nthe following format: [domain]\\bus\\:\\device\\.\\function\\\n\n-l,--log-level \\level\\\n\nstdout logging level. Must be one of:\n\n{trace,debug,info,warning,error,critical,off}\n\nDefault is info.\n\n-s,--shared\n\nopen in shared mode, default is off\n\n-t,--timeout timeout\n\nProgram timeout in milliseconds. Default is 60000 ms."},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#54-subcommands","title":"5.4 Subcommands","text":""},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#541-cpeng","title":"5.4.1 cpeng","text":"The copy engine command is used to program copy engine AFU registers to copy an image file from the host into the FPGA DDR. When the HPS boots, the first stage boot loader loads an image from a specific offset in DDR that will be used to transition into the second stage boot loader and subsequently boot into the embedded Linux that is also part of this image.
cpeng options description -h,--help Print this help message and exit -f,--filename filename Path to image file to copy. Default is u-boot.itb -d,--destination offset DDR Offset. Default is 0x2000000. -t,--timeout cpeng timeout Timeout of cpeng command in microseconds. Default is 1 sec (1000000 usec). -r,--data-request-limit size Can be 64, 128, 512, or 1024 and represents the PCIe request size in bytes that the copy engine IP will use. This is encoded to 0, 1, 2, or 3 and written to the copy engine DATA\\REQUEST\\LIMIT register. Default is 512. -c,--chunk size Split the copy into chunks of size size. 0 indicates no chunks. Chunk sizes must be aligned with data request limit. Default is 4096. --soft-reset Issue a soft reset only. --skip-ssbl-verify Do not wait for ssbl verify. --skip-kernel-verify Do now wait for kernel verify."},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#542-heartbeat","title":"5.4.2 heartbeat","text":"This subcommand reads the value in the HPS2HOST register to check for a hearbeat. This compares the value to previous value read and determines the HPS is alive if the value is incrementing. This relies on the hps running the hello-cpeng program in heartbeat mode which will increment the upper 16 bits in the HPS2HOST register. Please see a typical sequence of using the rsu and hps commands as below for a device with BDF 15:00:0
rsu fpga -p user1 15:00.0\nsudo opae.io release -d 15:00.0\nsudo opae.io init -d 15:00.4 root:root\nhps cpeng -f u-boot-userkey-vab.itb\ntimeout 5 hps heartbeat\nsudo opae.io release -d 15:00.0\nhps cpeng -f u-boot-userkey-vab.itb\n
The above command will transfer the U-Boot FIT image via the copy engine IP to the HPS DDR and then signal completion of the transfer to the copy engine. After the copy engine completes the actual transfer, it writes to the register the HPS SPL is polling on allowing the SPL to load the U-Boot bootloader which in turn boots into the Linux image embedded in the U-Boot FIT image. If a terminal emulator is connected to the UART as described above, a user can observe U-Boot and Linux running on the HPS.
First FSBL is loaded and executed by FPGA configuration. Then Board/Server gets powered on. FPGA Configuration is done via JTAG followed by a reboot
The FSBL will send the following output the serial port:
U-Boot SPL 2021.07-00312-g32c0556437 (Sep 17 2021 - 08:42:45 -0700)\nReset state: Cold\nMPU 1200000 kHz\nL4 Main 400000 kHz\nL4 sys free 100000 kHz\nL4 MP 200000 kHz\nL4 SP 100000 kHz\nSDMMC 50000 kHz\nDDR: Warning: DRAM size from device tree (1024 MiB)\nmismatch with hardware (2048 MiB).\nDDR: 1024 MiB\nSDRAM-ECC: Initialized success with 239 ms\nwaiting for host to copy image\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#60-meta-bakepy","title":"6.0 meta-bake.py","text":"A script called meta-bake.py has been added to allow for more control of configuration/customization of recipes and their dependencies. This script separates the data from the logic by requiring that data be expressed in a yaml configuration file. This file contains the following confiration data:
- machine - The FPGA/SoC platform to build for, choices are agilex, stratix10, arria10, cyclone5
- image - The image type to build, choices are gsrd, nand, pcie, pr, qsqpi, sgmii, tse, n6000
- target - The build target to build. This is typically a Yocto image to build.
- fit - Make a monolothic FIT image after the Yocto build. This will use U-Boot source and binaries as well as the rootfs made for the image.
- repos - A list of repositories to pull for Yocto recipes. This information is made up of:
- name - The project name (this is also the directory where source is clone to)
- url - The URL to pull the source from
- branch - The branch to checkout
- add_layers - Can be either True or a list of sub-directories to add as layers in bblayers.conf
- patch - Path to a file to use to patch the source code
- keep - When set to true, this will leave the source tree untouched on subsequent runs
- upstream_versions - Dependencies/versions to use for either Linux kernel, U-Boot, and/or ATF. This information is made up of:
- name - Project name
- version - version to configure recipes that use it
- branch - branch to use, will use git hash in recipe
- url - URL to pull the source from
- disabled - when set to True, this project will be ignored
- local - Used to configure local.conf used by Yocto/bitbake build. This information is made up of:
- remove - List of keys to remove from local.conf
- values - Dictionary of key/value pairs to use to insert into local.conf. Any existing key/value pairs will be overwritten.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#61-running-meta-bakepy","title":"6.1 Running meta-bake.py","text":"First, download the script from meta-opae-fpga.
To create a U-Boot fit and spl image for N6000/1 platforms, you must first meet these setup conditions:
- Host PC with Ubuntu 20.04 LTS
- ARM cross compiler
- set CROSS_COMPILE environment variable to: /bin/aarch64-linux-gnu- - e.x. export CROSS_COMPILE=aarch64-linux-gnu-
- set ARCH to: arm64 - e.x. export ARCH=arm64
- At least 100 Gb of disk space
- Tested on OFS 3.0.0
This script does not require any bare-metal accesses to perform its build and can be run from a VM with no alterations. Ensure that the Ubuntu 20.04 Guest VM you create has enough to space to perform the entire build (recommend at least 200 GiB total space), as does the drive it is stored on. You will need to configure proxy access in the VM if required for your environment. You can use any VM technology; having ssh enabled in the VM will allow you to more easily transfer the completed build files back to the host system but is not required.
Package dependencies to build Yocto on each supported OS can be found on the Yocto Quick Start page.
wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\ntar xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\nrm gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz\nexport CROSS_COMPILE=`pwd`/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-\nexport ARCH=arm64\n./meta-bake.py build\n
After running this build, the images you need to boot the HPS are located under build/agilex-n6000-images. Follow the steps in Section 4.3 Example Boot to finish bringing up your board.
This script will do the following:
- Parse layers.yaml for configuration to use for build
- Download recipe repositories (including poky) listed in
repos secion of layers.yaml - Apply refdes-n6000 patch to meta-intel-fpga-refdes source tree
- Configure Yocto build in build directory
- Source build/poky/oe-init-build-env passing in agilex-n6000-rootfs. This will initialize conf files.
- Configure build/agilex-n6000-rootfs/conf/local.conf using values in
local section of layers.yaml * Note: IMAGE_FSTYPES is configured to include cpio - Configure build/agilex-n6000-rootfs/conf/bblayers.conf using layer specification in
repos section of layers.yaml - Run Yocto build for target listed in layers.conf
- Call
bitbake n6000-image-minimal - Get environment variables to locate rootfs cpio file as well as U-Boot source and build directories
- Copy rootfs created by Yocto build for U-Boot
- Copy rootfs cpio file (n6000-image-minimal-agilex*.rootfs.cpio) to U-Boot build directory for selected configuration (socfpga_agilex_n6000_defconfig)
- Call U-Boot build in directory for selected configuration
- Copy FIT image (u-boot.itb) to images directory, build/agilex-n6000-images
- Many important images are copied to build/agilex-n6000-images, which may be useful if using VAB
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#62-required-changes","title":"6.2 Required Changes","text":"The patch file applied on top of the meta-intel-fpga-refdes repository introduces patches to:
- Add patch files so that Yocto can modify Linux kernel to add configuration for creating a device tree binary (DTB) compatible with N6000/1
- Add patch files so that Yocto can modify the bootloader in U-Boot to support booting with the assistance of the copy engine IP
- Modify rootfs to include copy-engine daemon as well as other packages that can be useful for debug
These changes may eventually be merged into upstream repositories for linux-socfpga, u-boot-socfpga, and meta-intel-fpga-refdes. Once all changes make their way into the repositories for the aforementioned projects, it will no longer be necessary to apply patches.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#63-manual-build","title":"6.3 Manual Build","text":"One may use meta-bake.py to only pull down required repositories and configure a Yocto build environment by using the --skip-build command line argument. To initiate a build after this, source poky/oe-init-build-env passing in a directory as the only argument. This will set up the user's environment to be able to run bitbake. To build the Yocto image, run bitbake n6000-image-minimal. This will build all the components necessary to build a FIT image. Once the build is complete, U-Boot make system may be used to make the FIT. The U-Boot build directory for the selected configuration can be found in the Yocto build environment directory at:
$ cd tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig\n
Once in this directory, ensure that the necessary files are present in here in order to assemble the FIT image (u-boot.itb) $ cp ../../../../../../deploy/images/agilex/n6000-image-minimal-agilex.cpio rootfs.cpio\n$ ls Image linux.dtb rootfs.cpio\nImage linux.dtb rootfs.cpio\n$ make\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#64-manual-vab-signing","title":"6.4 Manual VAB Signing","text":" - By default,
meta-bake.py will sign and certify the proper files for use with VAB. Below is an example on how to perform the manual VAB Signing Process.
First, use gzip to compress the follwoing four files before signing: bl31.bin, linux.dtb, rootfs.cpio.gz, and Image. They need to be compressed before you begin the VAB signing process or the image will fail to load through the CPE later on.
gzip <filename>\n
Make sure Quartus already installed and its tools added to environment. Example PATH=$PATH:/home/intelFPGA\\pro/21.3/quartus/bin/
$ cd HPS_VAB\n$ quartus_sign --family=agilex --operation=make_private_pem --curve=secp384r1 --no_passphrase userkey_root_private.pem\n$ quartus_sign --family=agilex --operation=make_public_pem userkey_root_private.pem userkey_root_public.pem\n$ quartus_sign --family=agilex --operation=make_rootuserkey_root_public.pem userkey_root_public.qky\n$ chmod +x fcs_prepare\n$ ./fcs_prepare --hps_cert bl31.bin -v\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_cert_bl31.bin.ccert\n\n# ATF Sign\n$ ./fcs_prepare --finish signed_cert_bl31.bin.ccert --imagefile bl31.bin\n$ mv hps_image_signed.vab signed-bl31.bin\n$ rm unsigned_cert.ccert\n\n# u-boot-nodtb\n$ ./fcs_prepare --hps_cert u-boot-nodtb.bin -v\n\n#signed_u-boot-nodtb.bin.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_u-boot-nodtb.bin.ccert\n\n# u-boot-nodtb.bin Sign\n$ ./fcs_prepare --finish signed_u-boot-nodtb.bin.ccert --imagefile u-boot-nodtb.bin\n$ mv hps_image_signed.vab signed-u-boot-nodtb.bin\n$ rm unsigned\\_cert.ccert\n\n# u-boot.dtb\n$ ./fcs_prepare --hps_cert u-boot.dtb -v\n\n#signed_u-boot.dtb.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_u-boot.dtb.ccert\n\n# u-boot.dtb Sign\n$ ./fcs_prepare --finish signed_u-boot.dtb.ccert --imagefile u-boot.dtb\n$ mv hps_image_signed.vab signed-u-boot.dtb\n$ rm unsigned_cert.ccert\n\n# Image\n$ ./fcs_prepare --hps/cert Image -v\n\n#signed_Image.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_Image.ccert\n\n# Image Sign\n$ ./fcs_prepare --finish signed_Image.ccert --imagefile Image\n$ mv hps_image_signed.vab signed-Image\n$ rm unsigned_cert.ccert\n\n# linux.dtb\n$ ./fcs_prepare --hps_cert linux.dtb -v\n\n#signed_linux.dtb.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_linux.dtb.ccert\n\n# linux.dtb Sign\n$ ./fcs_prepare --finish signed_linux.dtb.ccert --imagefile linux.dtb\n$ mv hps_image_signed.vab signed-linux.dtb\n$ rm unsigned_cert.ccert\n\n# rootfs.cpio\n$ ./fcs_prepare --hps_cert rootfs.cpio -v\n\n#signed_rootfs.cpio.ccert\n$ quartus_sign --family=agilex --operation=SIGN --qky=userkey_root_public.qky --pem=userkey_root_private.pem unsigned_cert.ccert signed_rootfs.cpio.ccert\n\n# rootfs.cpio\n$ ./fcs_prepare --finish signed_rootfs.cpio.ccert --imagefile rootfs.cpio\n$ mv hps_image_signed.vab signed-rootfs.cpio\n$ rm unsigned_cert.ccert\n
Copy the following files to u-boot-socfpga folder:
#Copy the image back to uboot folder\n$ cp signed-bl31.bin ../u-boot-socfpga/\n$ cp signed-u-boot-nodtb.bin ../u-boot-socfpga/\n$ cp signed-u-boot.dtb ../u-boot-socfpga/\n$ cp signed-Image ../u-boot-socfpga/\n$ cp signed-linux.dtb ../u-boot-socfpga/\n$ cp signed-root\n$ fs.cpio ../u-boot-socfpga/\n
Recompile the U-Boot $ git clone https://github.com/altera-opensource/u-boot-socfpga\n$ cd u-boot-socfpga\n$ export CROSS\\COMPILE=aarch64-none-linux-gnu-; export ARCH=arm\n$ make socfpga/agilex/n6000/vab/defconfig\n$ make -j 24\n$ cd ..\n
Figure 3.1 N6000/1 Configs
If you not see the defconfig desired, please checkout the correct branch version. Example config shown above is socfpga_v2021.10.
If the memory device tree it mismatches with your hardware (figure below), change the memory device tree at u-boot-socfpga/arch/arm/dts/socfpga_agilex_n6000-u-boot.dtsi
To make it 2GB, change as
memory {\n\n\\* 2GB \\*\n\nreg = <0 0x00000000 0 0x40000000>,<0 0x00000000 0 0x40000000>;\n\n};\n
Figure 3.2 Device tree mismatches example Refer to 6. Host Side Startup
$ sudo opae.io init -d 4b:00.4 root:root\n$ hps cpeng -f u-boot.itb\n$ timeout 5 hps heartbeat\n
The error happen (Figure below) when the Images do not sign with VAB. Figure 3.3 VAB certificate error example
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#70-debugging","title":"7.0 Debugging","text":"Debugging the HPS from the host side is standard HPS debugging. The primary debug tool is UART on the HPS and Arm DS-5 debugger.
A UART connection can be enabled on the board using the following procedure:
-
Connect the HPS Debug Card and HPS UART to the Intel N6000/1-PL FPGA SmartNIC Platform board
-
Open Putty with the following setting
Port:COM4\n\n Baudrate:115200\n\n Data bits : 8\n\n Stop bits : 1\n\n Parity : None\n\n Flow Control : None\n
-
Reboot the Intel N6000/1-PL FPGA SmartNIC Platform board by typing reboot in the shell. You will be able to see the HPS UART traffic in the putty. If any issues are encountered in this step, check the HPS UART connection and the UART driver.
-
Check the PCI bdf ( lspci | grep acc ) or fpgainfo fme at the shell prompt.
-
Run the rsu and fpga\\reconfig scripts with respective arguments to print the logs.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#80-connecting-remotely-to-the-hps-using-ssh","title":"8.0 Connecting remotely to the HPS using ssh","text":"The HPS running on the Intel FPGA N6000/1-PL SmartNIC Platform can be remotely accessed to via the utility ssh, allowing the user to run commands and copy files. SSH must be run over a Point-To-Point Protocol daemon that has been included in the HPS software (as a part of the meta-openembedded layer, in the recipes-daemons/ippool recipe). In this example, the HPS is set up as a PPP Server, and the host OS is set up as a PPP Client. Serial communication between the host and HPS is accomplished via HPS UART1, which communicates through the FIM to the Soft UART on the FPGA, who in turn communicates with the host over PCIe.
The following steps assume the SSBL has not yet been loaded onto the HPS. If it has, a cold boot will reset the system.
- The HPS Copy Engine Module is available for access on PF 4 via the PF/VF Mux on the FPGA. This port needs to be bound to driver
vfio-pci (the following example assumes PCIE BDF 0000:b1:00.0). Substitute your device's BDF address and desired user/group access permissions into the following command.
$ sudo opae.io init -d 0000:b1:00.4 <USER>[:<GROUP>]\nUnbinding (0x8086,0xbcce) at 0000:b1:00.4 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.4 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.4 is 190\nAssigning /dev/vfio/190 to DCPsupport\nChanging permissions for /dev/vfio/190 to rw-rw----\n
- When an HPS enabled SOF or BIN with the FSBL is loaded onto the FPGA, a message will be displayed on the host OS (seen via
dmesg) after boot once the serial port has been registered with the dfl-uart driver. The UART driver is included as a part of the linux-dfl driver package. An example output from dmesg is shown below (search dmesg using dmesg | grep dfl-uart):
[ 7.343014] dfl-uart dfl_dev.7: serial8250_register_8250_port 2\n
The device file that corresponds with serial UART port 2 is /dev/ttyS2 (format is /dev/ttyS<port number>). A serial communication program can be used to view the HPS boot in realtime, then log in and run commands when boot has completed. Minicom is the program that will be used in this example, although others will work. Install Minicom using DNF sudo dnf install minicom.
- Minicom requires configuration changes before it can listen to the serial device. Using the built-in menu accessed by
sudo minicom -s, ensure the information under \"Serial port setup\" matches the following, where the serial device corresponds with the serial port discussed previously:
+-----------------------------------------------------------------------+\n | A - Serial Device : /dev/ttyS2 |\n| |\n| C - Callin Program : |\n| D - Callout Program : |\n| E - Bps/Par/Bits : 115200 8N1 |\n| F - Hardware Flow Control : Yes |\n| G - Software Flow Control : No |\n| |\n| Change which setting? |\n+-----------------------------------------------------------------------+\n
-
Save and exit the configuration menu. Run Minicom using the command sudo minicom and keep the terminal open and connected.
-
Load the SSBL onto the HPS using a second terminal. This requires a built ITB image.
$ hps cpeng -f u-boot.itb\n
- You should see the HPS boot sequence continue through your Minicom terminal. Once boot has completed, log in using the user
root with an empty password.
...\n...\n...\n[ OK ] Finished Load/Save Random Seed.\n[ OK ] Finished OpenSSH Key Generation.\n\nPoky (Yocto Project Reference Distro) 3.3.6 agilex ttyS0\n\nagilex login: root\nroot@agilex:~#\n
- Configure the running Yocto image on the HPS as a PPP server. Run the following command through Minicom on the HPS (connects address 192.168.250.2 on the HPS to 192.168.250.1 on the host):
root@agilex:~# pppd noauth passive 192.168.250.1:192.168.250.2\n[ 410.465450] PPP generic driver version 2.4.2\n...\n
- Exit the Minicom program running on the host using
^A X. Execute the following command on the host to establish a PPP connection as the client (if not installed on the host, run sudo dnf install ppp):
$ sudo pppd ttyS2 115200 crtscts lock local noauth passive debug\n
- A new network interface device registered to ppp should be visible.
$ ip -4 addr\n...\n8: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 3\ninet 192.168.250.2 peer 192.168.250.1/32 scope global ppp0\n valid_lft forever preferred_lft forever\n
With both the client and server communicating, ssh and scp can be used to run commands and transfer files using IPv4 address 192.168.250.1 on the host. An example operation run on the host OS is shown below:
[user@localhost ]: scp file_package.tar.gz root@192.168.250.1\n
Note: If you are developing software for the HPS and altering system settings, it is possible for ssh to prohibit a connection due to a false man-in-the-middle attack warning. The flag <ssh/scp> -o StrictHostKeyChecking=no can be used to ignore the warning.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#90-example-design-enabling-emmc-i2c-and-uart-in-platform-design","title":"9.0 Example Design - Enabling eMMC, I2C and UART in Platform Design","text":"The following section will walk through the process by which eMMC, I2C, and UART can be added to the FIM and the HPS image. The goal of this section is to allow the HPS to configure eMMC memory on boot and uses WNC's FPGA SmartNIC Series as a reference.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#91-configuring-the-fim","title":"9.1 Configuring the FIM","text":" - Configure eMMC, I2C, and UAET in Platform Designer. ACtual pin assignments are determined by the WNC board schematic. In Quartus, navigate to the HPS Processor Subsystem Intel Agilex FPGA IP -> Pin Mux and Peripherals -> Advanced -> Advanced IP Placement.
Check your pin assigments for the eMMC, UART and I2C in the Pin Planner. If these assignments are not present, then they can be found at the following link. Based on the changes shown above, the UART pins are removed on HPS IO3 and IO4 what are mapped on AG6 and AB1.
- Click Apply Selections->Generate HDL
- Check for instantiation in
top.sv. Click Generate -> Show Instatiation Template.
The following image demonstrates eMMC and I2C properly instatiated in top.sv.
- Add the following to the hps_ss modules in
top.sv.
- Compile the design.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#92-configuring-the-hps","title":"9.2 Configuring the HPS","text":" - Enable mmc and DesignWare Memory Card interface flags in U-Boot (.config). After building U-Boot with
meta-bake.py, this file is located at /agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga//build/socfpga_agilex_n6000_defconfig/.config. CONFIG_CMD_MMC=y //Enable mmc command tool in uboot CONFIG_MMC_DW=y // support DesignWare Memory Card Interface\n
- Enable and configure I2C and eMMC in U-Boot (socfpga_agilex_n6000.dts). After building U-Boot with
meta-bake.py, this file is located at /agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/git/arch/arm/dts/socfpga_agilex_n6000.dts. - Enable and configure I2C and eMMC in the Linux device tree (socfpga_agilex_n6000.dts). After building U-Boot with
meta-bake.py, this file is located at /agilex-n6000-rootfs/tmp/work-shared/agilex/kernel-source/arch/arm64/boot/dts/intel/socfpga_agilex_n6000.dts. - Compile Linux by navigating to the directory agilex-n6000-rootfs/tmp/work/agilex-poky-linux/linux-socfpga-lts/5.10.60-lts+gitAUTOINC+c35d63f9c7-r0/linux-agilex-standard-build and running the following:
$ make -j `nproc` Image dtbs
- Add the software utilities
util-linux-mkfs e2fsprogs.mke2fs e2fsprogs in to the Linux RootFS. Thes utilities will be used to create a filsystem (ext4, FAT32, etc.) and partition the eMMC. Make the following changes in meta-intel-fpga-refdes/recipes-images/poky/n6000-image-minimal.bb.
-
As an output from the Linux compilation from step 4 you will produce the files Image and socfpga_agilex_n6000.dtb. Transfer both over to the socfpga_agilex_n6000_defconfig directory. Rename socfpga_agilex_n6000.dtb to linux.dtb. Compile U-Boot by running make in directory agilex-n6000-rootfs/tmp/work/agilex-poky-linux/u-boot-socfpga/1_v2021.07+gitAUTOINC+24e26ba4a0-r0/build/socfpga_agilex_n6000_defconfig/. This compilation will produce both spl/u-boot-spl-dtb.hex and u-boot.itb.
-
Combines these files with your SOF FSBL bootloader (created in Section 9.1 Configuring the FIM).
$ quartus_pfg -c ofs_top.sof ofs_top_hps.sof -o hps_path=u-boot-spl-dtb.hex\n$ quartus_pfg -c ofs_top_hps_pof_flash.pfg //to pof file and flash to qspi\n
Program the FSBL enabled SOF onto the FPGA and warm reboot the server for changes to take affect. After reboot has completed use the following commands to program the SSBL.
$ sudo opae.io release -d d8:00.0\n$ sudo opae.io init -d d8:00.4 root:root\n$ hps cpeng -f u-boot.itb\n
During HPS boot you should see the following message if the eMMC has been properly configured.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#93-emmc-testing-from-the-hps","title":"9.3 eMMC Testing from the HPS","text":"The following memory test was run from the U-Boot shell.
- Erase eMMC using a start block offset 0x400.
$ mmc erase 0x400 20\n
Write test data (at 0x00A00000) into the eMMC offset 0x400.
$ mmc write 0x00A00000 0x400 10\n
Read test data (at 0x00A00040) back from eMMC offset 0x400.
$ mmc read 0x00A00040 0x400 10\n
Data comparison at memory offset 0x00A00000 and 0x00A00040. Data should match.
$ cmp.l 0x00A00000 0x00A00040 10.\n
The following memory test was run from Linux running on the HPS.
- Display the eMMC and its partitions.
$ fdisk -l\n
- Create a primary partition on the eMMC.
- Verify the partition has been created.
- Format the ext3 filesystem in the partition you just created (p1).
- Create the directory
mydata in /mnt. Mount the eMMC p1 partition to /mnt/mydata and verify the filsystem mount was successful.
$ mkdir -p /mnt/mydata\n$ mount /dev/mmcblk0p1 /mnt/mydata\n$ df\n
- Create a new text file and write some data in it - \"Hello World!\". After the device has been written to run
sync, unmount eMMC p1 partition and verify the unmount was successful.
$ sync\n$ umount /dev/mmcblk0p1\n$ df\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#layersyaml-reference","title":"layers.yaml Reference","text":"machine: agilex\nimage: n6000\ntarget: n6000-image-minimal\nfit: true\nuboot-dtb: [u-boot-nodtb.bin, u-boot.dtb]\nlinux-binary: [bl31.bin, linux.dtb, rootfs.cpio, Image]\nroot-public-qky: userkey_root_public.qky\nroot-private-pem: userkey_root_private.pem\nroot-public-pem: userkey_root_public.pem\nrepos:\n- name: poky\nurl: https://git.yoctoproject.org/git/poky.git\nbranch: hardknott\n- name: meta-intel-fpga\nurl: https://git.yoctoproject.org/git/meta-intel-fpga.git\nbranch: hardknott\nadd_layers: true\n- name: meta-intel-fpga-refdes\nurl: https://github.com/altera-opensource/meta-intel-fpga-refdes.git\n#url__: https://github.com/intel-innersource/os.linux.yocto.reference-design.meta-intel-fpga-refdes.git\n#url: git@github.com:intel-innersource/os.linux.yocto.reference-design.meta-intel-fpga-refdes.git\n#branch: rrojo/n6000\nbranch: hardknott\npatch: refdes-n6000-ppp.patch\nkeep: true\nadd_layers: true\n- name: meta-openembedded\nurl: https://github.com/openembedded/meta-openembedded.git\nbranch: hardknott\nadd_layers:\n- meta-oe\n- meta-networking\n- meta-python\n- name: fcs_prepare\nurl: https://github.com/altera-opensource/fcs_apps.git\nbranch: fcs_prepare\ningredients:\nlinux:\nname: linux-socfpga\nversion: '5.10.100'\nbranch: socfpga-5.10.100-lts\nurl: https://github.com/altera-opensource/linux-socfpga.git\nuboot:\nname: u-boot-socfpga\nversion: '2021.07'\nbranch: socfpga_v2021.07\nurl: https://github.com/altera-opensource/u-boot-socfpga.git\natf:\ndisabled: true\nversion: '2.4.1'\nbranch: socfpga_v2.4.1\nurl: https://github.com/altera-opensource/arm-trusted-firmware.git\nlocal:\nremove:\n- MACHINE\n- UBOOT_CONFIG\n- IMAGE\n- SRC_URI\nvalues:\nMACHINE: $machine\nDL_DIR: $build_dir/downloads\nDISTRO_FEATURES_append: \" systemd\"\nVIRTUAL-RUNTIME_init_manager: systemd\nIMAGE_TYPE: $image\nIMAGE_FSTYPES: \"+=cpio tar.gz\"\nPREFERRED_PROVIDER_virtual/kernel: linux-socfpga-lts\nPREFERRED_VERSION_linux-socfpga-lts: 5.10%\nUBOOT_CONFIG: agilex-n6000\nPREFERRED_PROVIDER_virtual/bootloader: u-boot-socfpga\nPREFERRED_VERSION_u-boot-socfpga: v2021.07%\n
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#faqs","title":"FAQs","text":"Below are the Frequently Asked Questions:
-
How will you get the software stack for HPS (FSBL, U-Boot, Kernel)? Or will there be a package available to them on Git, Intel RDC?
Answer : HPS software has been around for quite a long time. Support for the OFS and the N6000-PL FPGA SmartNIC Platform will be upstreamed and available from rocketboards.com, just like any other HPS based project.
-
What are the recommended steps for building the binaries and where will those be located?
Answer: There are many documents on building the binaries at rocketboards.com. Any reference binaries can be stored at rocketboards.com as well.
-
What are the host side commands used to put the binaries to Copy Engine and from there to HPS?
Answer: There is a single command, hps to download the single binary through the Copy Engine to the HPS.
-
What are the host side commands used to reset the HPS from Host side?
Answer: This functionality is planned to be added to the hps command.
-
What is the procedure used to debug the HPS from Host side?
Answer: Debugging the HPS from the host side is standard HPS debugging. The primary debug tool is UART on the HPS and the Arm DS debugger.
-
Do we have performance metrics about HPS, like whether any bench marking information available with any sample application is available?
Answer: Any performance metrics on the HPS would be available on rocketboards.com.
-
What is the PXE boot flow and what is required to enable the same?
Answer: On some configurations, HPS is treated as a fully-fledged SoC and can PXE boot itself. But you must add this functionality.
"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#further-links","title":"Further Links","text":"Description Link OFS Getting Started User Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach FPGAs https://ofs.github.io/hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/ Intel\u00ae FPGA Interface Manager Developer Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach FPGAs https://ofs.github.io/hw/n6001/dev_guides/fim_dev/ug_dev_fim_ofs_n6001/ Open FPGA Stack Technical Reference Manual for Intel Agilex FPGA PCIe Attach https://ofs.github.io/hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/ Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs https://ofs.github.io/hw/N6001/dev_guides/afu_dev/ug_dev_afu_n6001/ UVM Simulation User Guide: OFS for Intel\u00ae Agilex\u00ae PCIe Attach FPGAs https://ofs.github.io/hw/n6001/user_guides/ug_sim_ofs_n6001/ug_sim_ofs_n6001/ FPGA Device Feature List (DFL) Framework Overview https://github.com/OFS/linux-dfl/blob/fpga-ofs-dev/Documentation/fpga/dfl.rst#fpga-device-feature-list-dfl-framework-overview ofs-platform-afu-bbb https://github.com/OFS/ofs-platform-afu-bbb Connecting an AFU to a Platform using PIM https://github.com/OFS/ofs-platform-afu-bbb/blob/master/plat_if_develop/ofs_plat_if/docs/PIM_AFU_interface.md example AFUs https://github.com/OFS/examples-afu.git PIM Tutorial https://github.com/OFS/examples-afu/tree/main/tutorial Non-PIM AFU Development https://github.com/OFS/examples-afu/tree/main/tutorial Intel FPGA IP Subsystem for PCI Express IP User Guide https://github.com/OFS/ofs.github.io/docs/hw/common/user_guides/ug_qs_pcie_ss.pdf Memory Subsystem Intel FPGA IP User Guide https://github.com/OFS/ofs.github.io/docs/hw/common/user_guides/ug_qs_mem_ss.pdf OPAE.io https://opae.github.io/latest/docs/fpga_tools/opae.io/opae.io.html OPAE GitHub https://github.com/OFS/opae-sdk README_ofs_n6001_eval.txt https://github.com/OFS/ofs-n6001/blob/release/ofs-2023.1/eval_scripts/README_ofs_n6001_eval.txt FIM MMIO Regions https://ofs.github.io/hw/d5005/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#mmio_regions evaluation script https://github.com/OFS/ofs-n6001/tree/release/ofs-2023.1/eval_scripts OFS https://github.com/OFS OFS GitHub page https://ofs.github.io DFL Wiki https://github.com/OPAE/linux-dfl/wiki"},{"location":"hw/n6001/dev_guides/hps_dev/hps_developer_ug/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/n6001/doc_modules/Glossary/","title":"Glossary","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel Max10 or Intel Cyclone10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implemented on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Management Component Transport Protocol MCTP A standardized model for communication with management controllers. Defines the transport protocol carrying PLDM messages through the BMC. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE-SDK The OPAE-SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Level Data Model PLDM A specification for reporting telemetry data to the host, such as board temperature, voltage, and current. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/doc_modules/Notices_%26_Disclaimers/","title":"Notices & Disclaimers","text":""},{"location":"hw/n6001/doc_modules/Notices_%26_Disclaimers/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"hw/n6001/doc_modules/n6001_wt_program_fpga_via_jtag/","title":"N6001 wt program fpga via jtag","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL with a SOF image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)] for instructions on setting up a deployment environment.
- This walkthrough requires a
SOF image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a SOF image.
- This walkthrough requires a JTAG connection to the n6001. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae FPGA SmartNIC N6001-PL is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
If the card is already programmed with an OFS enabled design, determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Remove the card from PCIe prior to programming. This will disable AER for the PCIe root port to prevent a surprise link-down event during programming.
sudo pci_device b1:00.0 unplug\n
-
Switch to the machine with JTAG connection to the n6001, if different than your deployment machine.
-
Open the Quartus programmer GUI
quartus_pgmw\n
-
Click Hardware Setup to open the Hardware Setup dialog window.
-
In the Currently selected hardware field select the n6001.
-
In the Hardware frequency field enter 16000000 Hz
-
Click Close
-
In the Quartus Prime Programmer window, click Auto Detect.
-
If prompted, select the AGFB014R24A2E2V device. The JTAG chain should show the divice.
-
Right click the AGFB014R24A2E2V row and selct Change File.
-
In the Select New Programming File window that opens, select the .sof image you wish to program and click Open.
-
Check the Program/Configure box for the AGFB014R24A2E2V row, then click Start. Wait for the Progress bar to show 100% (Success).
-
Close the Quartus Programmer GUI. You can answer 'No' if a dialog pops up asking to save the 'Chain.cdf' file
-
Switch to the deployment environment, if different than the JTAG connected machine.
-
Replug the board PCIe
sudo pci_device b1:00.0 plug\n
-
Run fpgainfo fme to verify communication with the board, and to check the PR Interface ID.
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
"},{"location":"hw/n6001/doc_modules/n6001_wt_program_fpga_via_rsu/","title":"N6001 wt program fpga via rsu","text":"This walkthrough describes the steps to program the Agilex FPGA on the Intel\u00ae FPGA SmartNIC N6001-PL with a BIN image via JTAG.
Pre-Requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)] for instructions on setting up a deployment environment.
- This walkthrough requires a
BIN image which will be programmed to the Agilex FPGA. Refer to the [Walkthrough: Compile OFS FIM] Section for step-by-step instructions for generating a BIN image. The image used for programming must be formatted with PACsign before programming. This is done automatically by the build script.
- This walkthrough requires a JTAG connection to the n6001. Refer to the [Walkthrough: Set up JTAG] section for step-by-step instructions.
- This walkthrough requires a Full Quartus Installation or Standalone Quartus Prime Programmer & Tools running on the machine where the Intel\u00ae FPGA SmartNIC N6001-PL is connected via JTAG.
Steps:
-
Start in your deployment environment.
-
Determine the PCIe B:D.F of the card using OPAE command fpgainfo fme. In this example, the PCIe B:D.F is B1:00.0.
fpgainfo fme\n
Example output:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.0\nBoard Management Controller Build version: 3.15.0\nPBA: B#FB2CG1@AGF14-A0P2\nMMID: 217000\nSN: Q171211700050\n//****** FME ******//\nInterface : DFL\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:98:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : 9035190d637c383453173deb5de25fdd\nUser1 Image Info : 893e691edfccfd0aecb1c332ad69551b\nUser2 Image Info : 8cd2ae8073e194525bcd682f50935fc7\n
-
Use the OPAE fpgasupdate command to program a PACsign signed image to flash. The flash slot which will be programmed is determined by the PACsign header.
sudo fpgasupdate <IMAGE> <PCIe B:D.F>\n
-
Example: update User Image 1 in flash
sudo fpgasupdate ofs_top_page1_unsigned_user1.bin 98:00.0\n
-
Example: update User Image 2 in flash
sudo fpgasupdate ofs_top_page2_unsigned_user2.bin 98:00.0\n
-
Example: update Factory Image in flash
sudo fpgasupdate ofs_top_page0_unsigned_factory.bin 98:00.0\n
-
Use the OPAE rsu command to reconfigure the FPGA with the new image. You may select which image to configure from (User 1, User 2, Factory).
sudo rsu fpga --page <PAGE> <PCIe B:D.F>\n
-
Example: configure FPGA with User 1 Image
sudo rsu fpga --page user1 98:00.0\n
-
Example: configure FPGA with User 2 Image
sudo rsu fpga --page user2 98:00.0\n
-
Example: configure FPGA with Factory Image
sudo rsu fpga --page factory 98:00.0\n
"},{"location":"hw/n6001/doc_modules/n6001_wt_set_up_jtag/","title":"N6001 wt set up jtag","text":"Perform the following steps to set up a JTAG connection to the Intel\u00ae FPGA SmartNIC N6001-PL.
Pre-requisites:
- This walkthrough requires an OFS Agilex PCIe Attach deployment environment. Refer to the [Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)] for instructions on setting up a deployment environment.
- This walkthrough requires a workstation with Quartus Prime Pro Version 24.1 tools installed, specifically the
jtagconfig tool.
- This walkthrough requires an [Intel FPGA Download Cable II].
Steps:
-
Set the board switches to dynamically select either the Agilex\u00ae 7 FPGA or MAX\u00ae 10 device on the JTAG chain.
- Set SW1.1=ON as shown in the next image. The switches are located at the back of the Intel\u00ae FPGA SmartNIC N6001-PL.
-
The Intel\u00ae FPGA SmartNIC N6001-PL has a 10 pin JTAG header on the top side of the board. Connect an Intel\u00ae FPGA Download II Cable to the JTAG header of the Intel\u00ae FPGA SmartNIC N6001-PL as shown in picture below. This picture shows the Intel\u00ae FPGA SmartNIC N6001-PL card installed in the middle bay, top slot of a SuperMicro\u00ae SYS-220HE-FTNR server where the lower slot does not have card installed allowing the Intel\u00ae Download II cables to pass through removed the slot access.
Note: If using the Intel FGPA download Cable on Linux, add the udev rule as described in [Intel FPGA Download Cable Driver for Linux].
-
Set the JTAG chain to select the ${{ env.DEVICE }} as the target by writing to the JTAG enable register in the BMC (Register 0x378). This is done via PMCI registers 0x2040C and 0x20400.
Note: The commands below are targeted to a board with PCIe B:D.F of 98:00.0. Use the correct PCIe B:D.F of your card.
sudo opae.io init -d 0000:98:00.0 $USER\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x2040c 0x100000000\nsudo opae.io -d 0000:98:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:98:00.0\n
Note: To later re-direct the JTAG back to the MAX 10 device, execute the following commands.
sudo opae.io init -d 0000:b1:00.0 $USER\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x2040c 0x000000000\nsudo opae.io -d 0000:b1:00.0 -r 0 poke 0x20400 0x37800000002\nsudo opae.io release -d 0000:b1:00.0\n
Optionally, rather than dynamically commanding Agilex\u00ae 7 FPGA/MAX10 selection with the PMCI register settings, you can fix the selection with the following switch settings shown in the table below:
SW1.1 SW2 JTAG Target OFF OFF Agilex\u00ae 7 FPGA OFF ON MAX\u00ae 10 FPGA ON X Agilex\u00ae 7 FPGA if BMC register 0x378=0x1 ON X MAX\u00ae 10 FPGA if BMC register 0x378=0x0 -
Use the jtagconfig tool to check that the JTAG chain contains the AGFB014R24A2E2V device.
<QUARTUS_INSTALL_DIR>/24.1/quartus/bin/jtagconfig\n
Example expected output:
TBD\n
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/","title":"Mnl fim ofs n6001","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#shell-technical-reference-manual-ofs-for-agilex-7-pcie-attach-fpgas","title":"Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1-overview","title":"1 Overview","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#11-about-this-document","title":"1.1 About this Document","text":"This document describes the hardware architecture for the PCIe attach reference FIMs of the Open FPGA Stack (OFS) targeting the Agilex\u00ae FPGA. After reviewing this document you should understand the features and functions of the components that comprise the FPGA Interface Manager (FIM), also known as the \"shell\". Different target boards have different default FIM configurations.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#12-glossary","title":"1.2 Glossary","text":"This table defines some of the common terms used when discussing OFS.
Table 1-1 Glossary Table
Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#13-introduction-to-open-fpga-stack","title":"1.3 Introduction to Open FPGA Stack","text":"The Open FPGA Stack (OFS) is a modular infrastructure of hardware platform components, open source upstreamed software, and broad ecosystem support that enables an efficient path to develop a custom FPGA platform. OFS provides a framework of FPGA synthesizable code, simulation environment and synthesis/simulation scripts. The key components of OFS include:
- Target development platforms such as Intel-branded Programmable Acceleration Cards (PACs), Acceleration Development Platforms (ADPs) and third-party platforms.
- Board Management Controller RTL and firmware that supports telemetry monitoring and capability for remote configuration updates.
- Source accessible, modular FPGA Interface manager (FIM) RTL with a UVM infrastructure unit tests that can be leveraged for your own custom FIM design. The FIM can be thought of as the FPGA shell that provides the I/O ring and timing closed management components for the FPGA.
- Basic building blocks for interconnect and PF/VF translation and arbitration; Platform Interface Manager (PIM) which provides Avalon\u00ae bus compliant interfaces.
- AFU examples both in the git repository and workload examples provided by 3rd party vendors.
- A OneAPI acceleration support package (oneapi-asp) that provides a bridge layer that is used by OneAPI runtime to communicate with the kernel.
- Unit level simulation test suite
- System level simulation through a unified verification methodology (UVM)
- OPAE software development kit (APIs, upstreamed Linux drivers and software tools)
- Support for other frameworks to be built on top of the OPAE such as DPDK
These components are available under the https://github.com/OFS site.
The OFS hardware repository supports hardware development and simulation. Repositories for OFS high level design support and board management controller RTL and firmware source code are also provided. These repositories can be found in the Altera Opensource Technology GitHub location.
Table 1-2 OFS Hardware Repositories
Repository Contains ofs-agx7-pcie-attach Contains FIM or shell RTL, automated compilation scripts, and unit tests and UVM framework. oneapi-asp Contains the hardware and software components you need to develop your own OneAPI board support package ofs-platform-afu-bbb Contains the files and scripts to build the platform interface manager. ofs-examples-afu Contains AFU examples you can use. ofs-bmc 1 Provides the OFS Board Management Controller RTL, firmware, scripts and collateral targeting the Intel\u00ae FPGA SmartNIC N6001-PL which can be leveraged for your own OFS design. 1 Access to BMC repositories requires entitlement access. To request access, please contact your local Altera sales representative.
Table 1-3 OFS Software Repositories
OPAE Git Repository Folder Contains opae-sdk Contains the files for building and installing OPAE SDK from source. linux-dfl Contains OFS Linux drivers that are being upstreamed to the Linux kernel. linux-dfl-backport Contains the backport version of the linux-dfl kernel driver for FPGA devices. opae-sim Contains the files for an AFU developer to build the Accelerator Functional Unit Simulation Environment (ASE) for workload development. Providing the hardware and software source code and supporting test frameworks in a GitHub repository allows you to easily customize your designs with the latest versions.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14-ofs-features","title":"1.4 OFS Features","text":"The OFS architecture within the FPGA comprises two partitions:
- FPGA Interface Manager (FIM)
- Accelerator Functional Unit (AFU)
The FIM or shell provides platform management functionality, clocks, resets and interface access to the host and peripheral features of the acceleration platform. The FIM architecture along with the supporting OPAE software supports features such as partial reconfiguration and virtualization. The FIM provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 datapath interface. The FIM resides in the static region of the FPGA.
The AFU partition is provided for custom acceleration workloads and may contain both static and partial reconfiguration regions.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#141-fpga-interface-manager-fim","title":"1.4.1 FPGA Interface Manager (FIM)","text":"The primary components of the FPGA Interface Manager, or shell, of each target board's default reference design are given in the following table:
Table: Default FIM Components
FIM Component SmartNIC N6001-PL SmartNIC N6000-PL F-Series Development Kit I-Series Development Kit PCIe Subsystem \u2714 \u2714 \u2714 \u2714 HSSI Subsystem \u2714 \u2714 \u2714 \u2714 Memory Subsystem \u2714 \u2716 [1] \u2714 \u2714 Hard Processor System \u2714 \u2714 \u2714 \u2716 [2] Reset Controller \u2714 \u2714 \u2714 \u2714 FPGA Management Engine \u2714 \u2714 \u2714 \u2714 AFU Peripheral Fabric for AFU accesses to other interface peripherals \u2714 \u2714 \u2714 \u2714 Board Peripheral Fabric for master to slave CSR accesses from Host or AFU \u2714 \u2714 \u2714 \u2714 Platform Management Controller Interface (PMCI) to the board management controller \u2714 \u2714 \u2716 [3] \u2716 [3] [1] The n6000 default shell design does not have the memory subsystem enabled. It can be enabled by following the instructions in [Section 4.7.2 of the PCIe Attach F-Series (P-Tile/E-Tile) Shell Developer Guide: Add or remove the Memory Sub-System [2] The HPS is not enabled in the FIM for the I-Series Development Kit [3] The F-Series Development Kit and I-Series Development Kit do not use the OFS BMC, and therefore do not use the PCMI.
The AFU Region provides design space for custom workloads and contains both static and partial reconfiguration regions. Partial reconfiguration allows you to update your specific logic blocks or entire workload while the rest of your static design is still in operation.
Note that the BMC RTL and firmware that works with this OFS design provided in a separate entitled repository. Please email ofs.github@intel.com if you would like to use our BMC code for your own design.
Figure: OFS FIM for Intel\u00ae FPGA SmartNIC N6001-PL Block Diagram
Figure: OFS FIM for Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) Block Diagram
Figure: OFS FIM for Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) Block Diagram
The table provides an overview of the OFS features targeting the Agilex\u00ae 7 FPGA. This reference FIM (shell) is a starting point for your custom FPGA design. With this initial starting point, you can add or subtract interfaces or ports to different Agilex devices.
Table 1-4 OFS FIM Features
Key Feature SmartNIC N6001-PL F-Series Development Kit I-Series Development Kit PCIe P-tile PCIe Gen4x16 F-tile PCIe Gen4x16 R-tile PCIe 1xGen5x16R-tile PCIe 2xGen5x8R-tile PCIe 1xGen4x16 Virtualization 5 physical functions/3 virtual functions with ability to expand 5 physical functions/3 virtual functions with ability to expand 5 physical functions/3 virtual functions with ability to expand Memory 5 DDR Channels:\u2022 One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each\u2022 Four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB 3 DDR Channels:\u2022 One HPS DDR4 bank, x40 (x32 Data and x8 ECC), 2400 MHz, 1GB each\u2022 Two Fabric DDR4 banks, x64 (no ECC), 2400 MHz, 8GB 4 Fabric DDR4 channels, x64 (no ECC), 2666 MHz, 8GB Ethernet \u2022 N6001-PL: 2x4x25GbE, 2x4x10GbE, or 2x100GbE \u2022 N6000-PL: 4x100GbE 2x4x25GbE 2x4x25GbE, 2x200GbE, 2x400GbE Hard Processor System 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. 64-bit quad core Arm\u00ae Cortex\u00ae-A53 MPCore with integrated peripherals. Not enabled Configuration and Board Manageability \u2022 FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration)\u2022 Platform Controller Management Interface (PMCI) Module for Board Management Controller \u2022 FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) \u2022 FPGA Management Engine that provides general control of common FPGA tasks (ex. error reporting, partial reconfiguration) Partial Reconfiguration Supported Supported Supported OneAPI OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime OneAPI Acceleration Support Package (ASP) provided with compiled FIM to support OneAPI Runtime Software Support \u2022 Linux DFL drivers targeting OFS FIMs\u2022 OPAE Software Development Kit\u2022 OPAE Tools \u2022 Linux DFL drivers targeting OFS FIMs\u2022 OPAE Software Development Kit\u2022 OPAE Tools \u2022 Linux DFL drivers targeting OFS FIMs\u2022 OPAE Software Development Kit\u2022 OPAE Tools Target Board \u2022 Intel\u00ae FPGA SmartNIC N6001-PL\u2022 Intel\u00ae 7 FPGA SmartNIC N6000-PL Agilex\u00ae 7 7 FPGA F-Series Development Kit (2x F-Tile) Agilex\u00ae 7 7 FPGA I-Series Development Kit (2xR-Tile, F-Tile)"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#subsystem-interfaces","title":"Subsystem Interfaces","text":"The PCIe, Memory and Ethernet interfaces in this design use a new flexible subsystem design that provides a standard Arm\u00ae AMBA\u00ae 4 AXI4 interface. To access these FPGA IP Subsystem documents. Please go to the links below: * AXI Streaming IP for PCI Express User Guide * Memory Subsystem Intel FPGA IP User Guide * Ethernet Subsystem Intel FPGA IP User Guide (public document)
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#hard-processor-system-hps","title":"Hard Processor System (HPS)","text":"The HPS SoC contains a 64-bit quad core ARM\u00ae Cortex\u00ae-A53 MPCore with a variety of integrated modules such as on-chip RAM, Ethernet, USB, UARTs and SPI controllers and memory controllers. For more information about the Agilex HPS, please refer to the Agilex Hard Processor System Technical Reference Manual.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FIM contains only one FME, regardless of the number of host interfaces to the FIM. The FME provides management features for the platform and controls reset and loading of the AFU into the partial reconfiguration region of the FPGA.
Any feature, such as a memory interface or global error control that you want to control through FME, must expose its capability to host software drivers. New features are exposed to the FME by adding a device feature header (DFH) register at the beginning of the feature's control status register (CSR) space. The FME CSR maps to physical function 0 (PF0) Base address register 0 (BAR0) so that software can access it through a single PCIe link. For more information about DFHs, refer to the FPGA Device Feature List Framework Overview.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#streaming-datapath","title":"Streaming Datapath","text":"The FIM implements an AXI4-Stream bus protocol for data transfer in the FIM. AXI4-Stream channels send data packets to and from the host channel IP without data abstraction. Memory-mapped I/O (MMIO) CSR accesses are routed to the ST2MM module, which converts the AXI4-Stream to an AXI4 memory-mapped protocol.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#virtualization","title":"Virtualization","text":"This design supports virtualization by making use of the virtualization functionality in the PCIe Hard IP and mapping packets to the appropriate physical or virtual function through a PF/VF multiplexer.
The reference FIM example enables 5 PFs and 3 VFs by default; however, you may extend your configuration to whatever the PCIe Hard IP can support or your application requires.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#142-afu","title":"1.4.2 AFU","text":"An AFU is an acceleration workload that interfaces with the FIM. The AFU boundary in this design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required for partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components:
- ST2MM module.
- Null Host exerciser (HE_NULL) stub.
- PCIe loopback host exerciser (HE-LB).
- HSSI host exerciser (HE-HSSI).
- Memory Host Exerciser (HE-MEM).
- Traffic Generator to memory (HE-MEM-TG). Port Gasket (PRG).
- HPS Copy Engine.
- Arm\u00ae AMBA\u00ae 4 AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
For this design the PF/VF Mux provides the following mappings (found in src/afu_top/mux/top_cfg_pkg.sv):
Table 1-5 PF/VF Mapping
Module PF/VF Mapping AXI4 Stream to Memory Mapped Module (ST2MM) PF0 Memory Host Exerciser (HE_MEM) PF0-VF0 HSSI Host Exerciser (HE_HSSI) PF0-VF1 Memory Traffic Generator (HE_MEM_TG) PF0-VF2 Null Host exerciser (HE_NULL) stub PF1-VF0 PCIe Loopback (HE_LB) PF2 Null Host exerciser (HE_NULL) stub PF3 HPS Copy Engine Module PF4 The figure below highlights the AFU portion of the OFS block diagram for the SmartNIC N6001-PL as an example.
Figure: SmartNIC N6001-PL AFU Diagram
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#143-platform-interface-manager","title":"1.4.3 Platform Interface Manager","text":"The PIM provides a way to abstract the AXI4-Stream interface to the AFU by providing a library of shims that convert the host channel native packet into other protocols such as AXI4 memory-mapped, Avalon\u00ae streaming (Avalon-ST) or Avalon\u00ae memory-mapped (Avalon-MM).
The FPGA or AFU developer implements these interface abstractions in the AFU region (afu_main) of the design.
For more information, refer to Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#144-platform-feature-discovery","title":"1.4.4 Platform Feature Discovery","text":"This reference design comes with specific FPGA drivers that are upstreamed to linux-dfl. These drivers abstract the hardware and operating system specific details of the platform to the host.
The FIM implements a device feature header (DFH) at the beginning of each host-discoverable feature in a linked list format that allows an FPGA platform driver running on the host to discover FME, port, and AFU features.
You must implement a 64-bit DFH Device Feature Header register at the beginning (first 8B aligned address) of the feature CSR space for a new feature to be discovered or enumerated by a driver.
During host discovery, the driver traverses the DFH of the first feature from the first address on PF0 BAR0. Based on the information in the DFH, a driver can determine the CSR address range of the feature and other associated details. The end of the DFH contains a \"next DFH offset\" field that points the driver to the DFH of the next feature.
The software must continue traversing the linked list until it sees the EOL (End-Of-List) bit set to 1 in the \"next DFH offset\" field it inspects. A 1 indicates this is the last feature in the feature set. The figure below gives a simple illustration of the feature discovery by traversing the DFH registers. This model is similar to how PCIe enumeration occurs.
Figure 1-4 Device Feature Header Linked List Traversal
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#145-ofs-reference-design","title":"1.4.5 OFS Reference Design","text":"OFS provides FIM designs you can use as a starting point for your own custom design. These designs target a specific programmable acceleration card or development kit and exercise key FPGA device interfaces.
FIM designs are released to [https://github.com/OFS/ofs-agx7-pcie-attach] for evaluation and use. The provided reference designs can target the following boards:
- Intel\u00ae FPGA SmartNIC N6001-PL
- Intel\u00ae FPGA SmartNIC N6000-PL
- Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile)
- Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#146-fim-simulation","title":"1.4.6 FIM Simulation","text":"OFS provides unit tests and a UVM environment for the FIM and a framework for new feature verification. UVM provides a modular, reusable, and scalable testbench structure by providing an API framework that can be deployed across multiple projects.
The FIM testbench is UVM compliant and integrates third-party verification IPs from Synopsys that require a license.
Verification components include:
- FIM monitor to detect correct design behavior
- FIM assertions for signal level integrity testing
- Arm\u00ae AMBA\u00ae 4 AXI4 scoreboards to check data integrity
- FIM coverage to collect functional data
The verification infrastructure can be found in the verification directory for evaluation and use. Please refer to the UVM Simulation User Guide: OFS for Agilex\u00ae 7 PCIe Attach for more information.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#2-ofs-high-level-architecture","title":"2 OFS High Level Architecture","text":"OFS provides distinct data paths that simplify the design and integration process for adding or removing interface modules:
- High Bandwidth data path for AFU-attached high-performance peripherals (HSSI, Memory, HPS, workload).
- Low Bandwidth data path for OFS management and slow peripheral components (JTAG, I2C, SMBus).
- AFU Peripheral Fabric (APF) to Board Peripheral Fabric (BPF) path to communicate with interface control and status registers (CSRs) and board components.
- Peer-to-peer datapath between AFU components.
- Peer-to-peer datapath between BPF components.
Depending on your design goals, you can present peripherals to software as:
- OFS managed peripherals with a device feature header that is part of a device feature list.
- Native driver managed peripherals that are exposed through an independent physical function or virtual function.
Figure 2-1 OFS Datapath Structure
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#3-pcie-subsystem","title":"3 PCIe Subsystem","text":"The FIM's PCIe Subsystem is a hierarchical design that targets the PCIe Hard IP and is configured to support Gen4/Gen5 speeds. The default FIM uses the AXI Streaming Intel FPGA IP for PCIe Express. The IP supports SR-IOV and is configured to provide 5 PFs and 3 VFs by default. Native PCIe TLP packets are sent through the PCIe using Arm\u00ae AMBA\u00ae 4 AXI4 Stream Protocol. Tag allocation and management for packets sent from the application to the host are done by the PF/VF Mux that is part of the AFU region.
Note that this default PCIe-SS does not support Arm\u00ae AMBA\u00ae 4 AXI4 Data Mover functional mode. If Data Mover mode is required, you must instead build the FIM using the Intel FPGA IP Subsystem for PCI Express by changing build settings prior to FIM compilation; refer to the Shell Developer Guides for instructions on making this change.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs
Figure 3-1 AXI Streaming Intel FPGA IP for PCIe Express Block Diagram
Some key features of the PCIe interface are:
Table: PCIe Subsystem OFS Default Configuration
Feature n6001 n6000 fseries-dk iseries-dk Mode PCIe Gen4x16 PCIe Gen4x16 PCIe Gen4x16 PCIe Gen5x16 Port Mode Native Endpoint Native Endpoint Native Endpoint Native Endpoint SR-IOV 5PFs, 3VFs 5PFs, 3VFs 5PFs, 3VFs 5PFs, 3VFs MSI-X Support Yes Yes Yes Yes Profile Basic Basic Basic Basic TLP Bypass No No No No Header Packing Scheme Simple Simple Simple HIP Native Mode Data Width 512-bit (64-byte) 512-bit (64-byte) 512-bit (64-byte) 1024-bit (128-byte) AXI-ST Clock Frequency 500 MHz 350 MHz 500 MHz 500 MHz Tags Supported 256 256 256 768 Reordering No reordering of requests, no completion reordering No reordering of requests, no completion reordering No reordering of requests, no completion reordering No reordering of requests, no completion reordering Maximum Payload Size 512 Bytes 512 Bytes 512 Bytes 512 Bytes Memory Requests Supported 1CL, 2CL, 4CL TBD TBD TBD MMIO transaction Size 4B, 8B TBD TBD TBD Control Shadow Interface Enabled Enabled Enabled Enabled Completion Timeout Interface Enabled Enabled Enabled Enabled The PCIe PF, VF and Base Address Register (BAR) configuration can be modified in the PCIe Subsystem Platform Designer GUI interface. The current implementation for the OFS FIM for Agilex FPGA is as follows:
Table 3-1 Function and BAR Table for OFS for Agilex FPGA
PF VF Feature BAR BAR Size PF0 - OFS Managed Peripherals (PCIe, Memory, Ethernet) BAR0 512 KB PF0 - AFU Peripherals BAR0 256 KB PF0 - Board Peripherals BAR0 256 KB PF0 - MSI-X BAR4 16 KB PF0 VF0 Memory Host Exerciser (HE-MEM) BAR0 256 KB PF0 VF1 HSSI Host Exerciser (HE-HSSI) BAR0 256 KB PF0 VF2 Memory Traffic Generator (HE-MEM-TG) BAR0 256 KB PF1 Null Host exerciser (HE_NULL) BAR0 4 KB PF2 PCIe Loopback (HE-LB) BAR0 256 KB PF3 Null Host exerciser (HE_NULL) BAR0 4 KB PF4 HPS Copy Engine BAR0 4 KB"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#31-pcie-subsystem-header-format","title":"3.1 PCIe Subsystem Header Format","text":"The first 32 bytes of the TLP from the PCIe subsystem denotes the PCIe header. The default PCIe-SS in the OFS FIM for Agilex FPGA only supports the Power User Mode Header. If using Data Mover Mode with the old IP, then the tuser_vendor[0] bit on the AXI4-Stream channel indicates the header format of the TLP; tuser_vendor[0] =0 indicates Power User Mode header and tuser_vendor[0] =1 indicates Data Mover Mode header.
For more detailed information about the PCIe Subsystem, see the PCIe Subsystem FPGA User Guide.
Table 3-2 PCIe Subsystem Header Format Support for OFS for Agilex FPGA
Direction Type Power User Data Mover Host to Endpoint MWr, MRd Yes No Host to Endpoint CPL/CPLd Yes Yes Host to Endpoint Msg Yes No Endpoint to Host MWr, MRd Yes Yes Endpoint to Host Intr Yes (MWr) Yes Endpoint to Host CPL/CPLd Yes Yes Endpoint to Host Msg Yes Yes"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#32-pcie-subsystem-interface-module","title":"3.2 PCIe Subsystem Interface Module","text":"The PCIe Subsystem Interface module (/ipss/pcie/rtl/pcie_ss_if.sv), provides the supporting interface between software and the PCIe subsystem.
The interface module provides the following:
- Device Feature Header Registers
- Control and Status Registers
- Indirect access to PCIe subsystem CSR registers through a CSR mailbox in the PCIe Subsystem Interface.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#33-pcie-request-cycles","title":"3.3 PCIe Request Cycles","text":"For Host read request cycles using the OFS FIM for Agilex FPGA:
- All requests in the RX direction will be MMIO.
- Requester ID from the request does get sent to the AFU. It is the AFU's responsibility to send back a completion to the host with the correct completer ID.
- Prefix is not supported.
- Memory Mapped (MM) Mode is not supported.
- Slot Number is 0.
- Base address is not sent to the AFU.
- Local Address field is not used.
For AFU/application request cycles using the OFS FIM for Agilex FPGA:
- All requests in the TX direction will be Memory Read/Write.
- The tag must be generated by the AFU/application.
- Prefix is not supported.
- MM Mode is not supported.
- Slot Number is 0 (non-0 only for switch)
- VF Active, VF number and PF number are obtained from Data Mover Header Packet.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#34-pcie-completion-cycles","title":"3.4 PCIe Completion Cycles","text":"For Host completion cycles using the OFS FIM for Agilex FPGA:
- All completions in the RX direction will be Data Completions.
- Prefix is not supported.
- MM Mode is not supported.
- Slot Number is 0.
- Data packet responses (for Memory Read requests from AFU) from the PCIe SS may come out of order when the size is >64B.
For AFU/application completion cycles using the OFS FIM for Agilex FPGA: * All requests in the TX direction will be Memory Read/Write. * Requester ID is generated within the FIM. * That tag must be generated by the AFU/application. * Prefix is not supported. * MM Mode is not supported. * Slot Number is 0. * VF Active, VF Number and PF number are obtained from the Data Mover Header Packet.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#4-platform-interface-manager","title":"4 Platform Interface Manager","text":"The FIM interfaces to an application in the AFU region through AXI4-Stream channels. This format allows the AFU to access the host channel's raw interface without any translation.
As a FIM developer, you have the option to provide the raw data format associated with the host interface channel to the workload or AFU developer or you can provide an intermediate protocol using Platform Interface Manager Components or your own custom interface.
If you expose the raw AXI4-Stream interface of the FIM, workload developers also have the option to convert to a desired protocol using the PIM resources as well.
Refer to the Workload Developer Guide for more information on using the PIM in your design.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#5-interconnect-fabric","title":"5 Interconnect Fabric","text":"There are three types of interconnect fabric in the OFS FIM design: * AXI4-Stream PF/VF mux/demux fabric * AFU Peripheral Fabric (APF) * Board Peripheral Fabric (BPF)
Figure 5-1 Interonnect Fabric Diagram
TLP packets sent from upstream PCIe Subsystem on AXI4-Stream channel are demultiplexed in the AXI4-Stream PF/VF mux/demux fabric and routed to the respective PF/VF function based on the PF/VF information in the TLP header, such as vf_active or the PF/VF number. In the opposite direction, TLP packets from downstream PF/VF function are muxed in the fabric and sent to PCIe subsystem over AXI4-Stream channel.
All host MMIO requests targeting PF0 BAR0 are routed to the ST2MM module. The ST2MM converts MMIO TLP packets into AXI-Lite memory requests and places the requests onto AFU Peripheral Fabric (APF). AFU peripherals, such as OFS managed AFU features and ST2MM, and Board Peripheral Fabric (BPF) are interconnected by APF. The BPF is the interconnect fabric one hierarchy below APF which connects all the board peripherals. Both APF and BPF allow multiple AXI4-Lite master and slave interconnect topology.
If you are modifying the APF or BPF connections, you must re-generate the fabrics. OFS provides a helper script to perform this task.
For modifying the PF/VF mux you may edit the PCIe OFS Settings (OFSS) file to implement the desired PF/VF settings.
For details on these modifications, please refer to the shell developer guide for your target board:
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#51-afu-peripheral-fabric-apf","title":"5.1 AFU Peripheral Fabric (APF)","text":"The AFU Peripheral Fabric (APF) is a 64-bit Arm AXI4-lite compliant interconnect fabric that connects AFU peripheral modules to board peripheral modules through the Board Peripheral Fabric (BPF).
The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by the APF is listed below. All components are mapped to PF0 BAR0 and implement AXI-lite slave interface. Note that none of the features in the APF mapping are designed to act as a master.
Table 5-1 APF Address Mapping
Address Size (Byte) Feature 0x00000\u20130x3FFFF 256K Board Peripherals(See BPF address mapping table) 0x40000 \u2013 0x4FFFF 64K ST2MM 0x50000 \u2013 0x5FFFF 64K Reserved 0x60000 \u2013 0x60FFF 4K UART 0x61000 \u2013 0x6FFFF 4K Reserved 0x70000 \u2013 0x7FFFF 56K PR Gasket: PR Gasket DFH (4K)\u00a0\u00a0\u00a0\u00a0Control and status (4K)\u00a0\u00a0\u00a0\u00a0Port DFH (4K)\u00a0\u00a0\u00a0\u00a0User Clock (4K)\u00a0\u00a0\u00a0\u00a0Remote STP (52K) 0x80000 \u2013 0x80FFF 4K AFU Error Reporting"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#52-board-peripheral-fabric-bpf","title":"5.2 Board Peripheral Fabric (BPF)","text":"The Board Peripheral Fabric is the 64-bit AXI4-Lite compliant interconnect fabric that connects board peripheral modules to APF. The fabric is clocked by clk_csr and has a read allowance and write allowance of 1, i.e. only 1 active write/read is allowed in the fabric at any single time.
The address mapping for components interconnected by BPF is listed below. All components are mapped to PF0 BAR0 and implement AXI4-lite slave interface. The Master column indicates if a component also implements AXI4-lite master interface which can send requests to the BPF.
Table 5-2 BPF Address Mapping
Address Size (Byte) Feature 0x00000 \u2013 0x0FFFF 64K FME (FME, Error, etc) 0x10000 \u2013 0x10FFF 4K PCIe Interface 0x11000 \u2013 0x11FFF 4K Reserved 0x12000 \u2013 0x12FFF 4K QSFP Controller 0 0x13000 \u2013 0x13FFF 4K QSFP Controller 1 0x14000 \u2013 0x14FFF 4K Ethernet Subsystem 0x15000 - 0x15FFF 4K External Memory Interface 0x16000 - 0x19FFF 40K Reserved 0x20000 \u2013 0x3FFFF 128K PMCI Controller"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#53-arm-amba-4-axi4-stream-pfvf-muxdemux","title":"5.3 Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux","text":"The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF Mux/Demux routes the PCIe TLP packets from the PCIe subsystem AXI4-Stream RX channel to downstream PF/VF based on the pf_num and vf_num information in the PCIe TLP header.
The Arm\u00ae AMBA\u00ae 4 AXI4-Stream PF/VF mux arbitrates PCIe TLP packets from downstream PF/VF to the PCIe SS AXI-S TX channel. The PF/VF Mux/Demux is an M X N switch that allows any M port to target any N port, and any N port to target any M port, where M is the number of host/upstream ports, and N is the numbers functions/downstream ports.
The fpga top package file, /src/afu_top/mux/top_cfg_pkg.sv, contains the PF/VF parameters and mappings.
Figure 5-2 PF/VF Mux
The PF/VF mux integration is part of afu_top (/src/afu_top/mux/top_cfg_pkg.sv). There are two independent TX PF/VF MUX trees, labeled \"A\" and \"B\".
Both an A and a B port are passed to each AFU component with a unique PF/VF. You can design your AFU components to send all requests to the primary A port or partition requests across both A and B ports. A typical high-performance AFU sends read requests to the B port and everything else to the A port, giving the arbiter freedom to keep both the host TX and RX channels busy.
In the reference FIM provided for Agilex OFS, the A and B TX trees have been multiplexed down to a single channel for A and another for B. The A/B multiplexer merges them into a single TX stream that will be passed to the tag remapper.
The tag remapper provides unique tags as required by the PCIe specification. Tags are not provided by the PCIe Subsystem FPGA IP. When creating your own AFU you can leverage this module to generate unique tags.
Note that the primary PF/VF Mux A supports RX and TX ports. For the secondary PF/VF Mux B only TX ports are supported and the RX input to the Mux is tied off.
The default mapping is shown below:
Table 5-3 PF/VF Mapping
Module PF/VF Mapping AXI4 Stream to Memory Mapped Module (ST2MM) PF0 Memory Host Exerciser (HE_MEM) PF0-VF0 HSSI Host Exerciser (HE_HSSI) PF0-VF1 Memory Traffic Generator (HE_MEM_TG) PF0-VF2 Virtio Loopback Stub (Virtio_LB) PF1-VF0 PCIe Loopback (HE_LB) PF2 Virtio Loopback Stub (Virtio_LB) PF3 HPS Copy Engine Module PF4 For information on how to modify the PF/VF mapping for your own design, refer to the Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#54-afu-interface-handler","title":"5.4 AFU Interface Handler","text":"The AFU Interface Handler resides inline between the PCIe AXI4-Stream Adapter and the AXI4-Stream PF/VF Demux/Mux logic. Its main function is to provide: * Unique PCIe tags \u2013 Each PCIe transaction shares the 512 tags across all VFs in the AFU region * AFU error logging for all VFs in the AFU region
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#541-unified-tag-remapping","title":"5.4.1 Unified Tag Remapping","text":"When a FPGA function sends out a read cycle, it allocates a unique tag which is subsequently used to identify the read completion. The tag is considered busy; it cannot be assigned to another read cycle until read completion. While a tag may be unique within a unit, two different units could unknowingly send out two read cycles of the same tag. The PCIe subsystem requires unique tags for all read cycles irrespective of their origins. Therefore, a mechanism is needed to uniquify tag globally across different units.
OFS contains a tag remapper (/ofs-fim-common/src/common/tag_remap/tag_remap.sv) that intercepts the read cycle, finds a globally unique tag, and replaces the original tag value. It also restores the original tag value when returning completion to the read requester. tag_remap is placed between the AXI4-Stream interface of the PCIE subsystem and the PF/VF Mux/Demux.
The logic is described as follows:
- A sub-module (
/ofs-fim-common/src/common/tag_remap/ofs_fim_tag_pool.sv) maintains a pool of available tags. - TX read requests are held until a tag is available from the pool by setting tvalid=0 to the host, and tready=0 to the PF/VF Mux/Demux.
- When a TX read is dispatched, the tag is marked busy in the pool.
- The original tag is stored in tag_reg, so it can be recovered when returning a completion to the unit/function.
- Because completion to a read request can split into multiple smaller transfer sizes, responses are monitored and the final completion is detected using PCIe TLP rules.
- Tags are released in the pool only when all requested data are transferred.
- When the completion returns, the original tag is restored from
tag_reg.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#542-afu-error-handling","title":"5.4.2 AFU Error Handling","text":"In this OFS design, the AFU Interface Handler handles error logging for all VFs in the AFU. Errors handled are as follows
Table 5-4 AFU Error Descriptions
Checker Field Description AFU protocol checker (PCIe TLP) Malformed TLP AFU PCIe TLP contains unsupported format type AFU protocol checker (PCIe TLP) MaxPayloadError AFU memory write payload size exceeds max_payload_length limit AFU protocol checker (PCIe TLP) MaxReadReqSizeError AFU memory read payload size exceeds max_read_request_size limit AFU protocol checker (PCIe TLP) MaxTagError AFU memory read request tag value exceeds the maximum supported tag count AFU protocol checker (PCIe TLP) UnalignedAddrErr The address field in AFU memory write/read request TLP is not DW-aligned. AFU protocol checker (PCIe TLP) UnexpMMIOResp AFU is sending a MMIO read response with no matching MMIO read request. AFU protocol checker (PCIe TLP) MMIOTimedOut AFU is not responding to a MMIO read request within the pre-defined response timeout period. AFU protocol checker (PCIe TLP) MMIODataPayloadOverrun The number of data payload sent by AFU for a MMIO response (cplD) is more than the data length specified in the response. AFU protocol checker (PCIe TLP) MMIOInsufficientData The number of data payload sent by AFU for a memory write request is more than the data length specified in the request. AFU protocol checker (PCIe TLP) TxMWrDataPayloadOverrun The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU protocol checker (PCIe TLP) TxMWrInsufficientData The number of data payload sent by AFU for a memory write request is less than the data length specified in the request. AFU Protocol Checker (Arm\u00ae AMBA\u00ae 4 AXI4) TxValidViolation Three checkers are implemented in the FIM to catch errors and protocol violations."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#55-tlp-to-axi4-lite-memory-mapped-bridge-st2mm","title":"5.5 TLP to AXI4-Lite Memory Mapped Bridge (ST2MM)","text":"ST2MM implements the following key features:
- Host MMIO bridge * Maps MMIO TLP packets received from the PCIe Subsystem over streaming interface to AXI4-Lite memory-mapped request. The memory-mapped request is sent to AFU or Board peripherals over APF and BPF. * Maps AXI4-lite MM response received from AFU or Board peripherals to TLP packets and send the packets over ST streaming channel to host HIA subsystem.
- Sends MMIO response of all 0\u2019s for MMIO read to unused BAR region.
- Interrupt * Sends interrupt packets to the PCIe subsystem when interrupt requests are received from the peripherals. Interrupts can be requested by a peripheral through a memory write to interrupt CSR registers in the ST2MM.
Figure 5-3 ST2MM Module
ST2MM implements both AXI4-lite master and slave interfaces that are connected to the designated slave and master port on APF. Host memory requests are sent on the ST2MM master interface to AFP where the requests are routed to the targeted peripherals.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#6-mmio-regions","title":"6 MMIO Regions","text":"The FIM and AFU expose their functionalities to the host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). An MMIO region is an address space within a base address register (BAR) region to which features are memory mapped.
For example, when a feature is mapped to an MMIO region, the CSR registers of that feature are located within the address range of that region. There can be multiple MMIO regions within a BAR region.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#61-feature-region","title":"6.1 Feature Region","text":"A group of related CSRs can be categorized as a feature region. A feature region is contained within a single PCIe BAR and cannot span across two BAR region boundaries.
A Device Feature Header (DFH) is a block of registers that mark the start of the feature region and sub-feature region, and you must place it at the first address of the region. Each DFH starts at 4KB boundary. A DFH register contains information that OPAE software requires to enumerate the feature. It also has an offset field that points to the next DFH in a feature list. OPAE software traverses the linked list of DFHs in each BAR region to discover all the features implemented on the platform.
The EOL field in a DFH Start marks the end of a DFH list and is only set in the DFH of the last feature in the feature list. The feature type field in the DFH is used to differentiate between the different types of feature region. Basic building blocks (BBB) and private features are always a child of an AFU or FPGA Interface Unit (FIU) and must be contained within an AFU or FIU, respectively.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#611-device-feature-header-dfh-structure","title":"6.1.1 Device Feature Header (DFH) Structure","text":"All DFHs must follow a specific structure to be compatible with OPAE software. Note that only features you want to be exposed to the OPAE software must have a DFH. For the latest information on DFH structure, please refer to the FPGA DFL Framework Overview.
6.2 Control and Status Registers
All the Control and Status Registers (CSRs) in the FIM are 64-bit registers with the following MMIO write, and MMIO read support.
Table 6-5: CSR MMIO Read and Write Support
Request Memory Attribute Payload size Memory Ordering MMIO Write UC 4B or 8B Strongly ordered MMIO Read UC 4B or 8B Strongly ordered The FIM does not reorder the MMIO requests or responses. For MMIO writes, there is no reordering of requests in FIM, and uncacheable (UC) ordering rules are followed. Similarly, for MMIO reads, there is no re-ordering of requests or responses in the FIM. An AFU may opt to re-order the MMIO read responses but the FIM does not enforce read response ordering.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#621-software-access-to-registers","title":"6.2.1 Software Access to Registers","text":" - Software accesses 64-bit registers as aligned quadwords. For example, to modify a field (bit or byte) in a 64-bit register, the entire quadword is read, the appropriate field(s) are modified, and the entire quadword is written back.
- When updating registers through multiple accesses (whether in software or due to hardware disassembly), certain registers may have specific requirements on how the accesses must be ordered for proper behavior. These are documented as part of the respective register descriptions.
- For compatibility with future extensions or enhancements, software must assign the last read value to all \u201cReserved and Preserved\u201d (RsvdP) fields when written. In other words, any updates to a register must be read so that the appropriate merge between the RsvdP and updated fields occurs. Also, software must assign a value of zero for \u201cReserved and Zero\u201d (RsvdZ) fields when written.
- PCIe locked operations to FPGA hardware registers are not supported. Software must not issue locked operations to access FPGA hardware registers.
In the following two cases, the FIM terminates MMIO Read requests by sending a completion with the data (CplD) specified below: * MMIO Timeout: This occurs when the AFU does not respond within a set timeout. The timeout value is currently configured to 512 pclks (clk_2x). In this case, the FIM returns all 1s.
- Illegal MMIO Accesses: This occurs when the read is accessing undefined registers in the FIM or if an AFU access violation. An example of an access violation is when a PF attempts to access the AFU when it is set to VF mode, or when a VF attempts to access the AFU when it is set to PF mode. In this case, the FME will returns all 0s.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#622-register-attribute-definition","title":"6.2.2 Register Attribute Definition","text":"Table 6-6: OFS Register Attribute Definitions
Attribute Expansion Description RW Read/Write This bit can be read or written by software. RO Read Only The bit is set by hardware only. Software can only read this bit. Writes do not have any effect. RW1C Read/ Write 1 to Clear Software can read or clear this bit. The software must write 1 to clear this bit. Writing zero to RW1C bit has no effect. Note that a multi-bit RW1C field may exist. In this case, all bits in the field are cleared if a 1 is written to any of the bits. RW1S Read/ Write 1 to Set Software can read this bit. Writing a 1 to the bit sets it to 1. Writing a 0 has no effect. It is not possible for software to set this bit to 0. The 1 to 0 transition can only be performed by HW. RW1CS Read/Write 1 to Clear Sticky Software can read and clear this bit. Writing a 1 to a bit clears it, while writing a 0 to a bit has no effect. This bit is only reinitialized to its default value by a power-on reset. RWD Read/Write Sticky across Hard Reset The bit can be read or written by SW. This bit is sticky or unchanged by any reset type, including Hard Reset. The bit gets cleared only with power on. *S Sticky across Soft Reset The bit will be sticky or unchanged by soft reset. These bits are only re-initialized to their default value by a power-on reset. *D Sticky across Hard Reset The bit is sticky or unchanged by or unchanged by any reset type, including hard reset. The bit gets cleared only with power on. Rsvd Reserved Reserved for future definitions. Currently don\u2019t care bits. RsvdP Reserved and Protected Reserved for future RW implementations. The software must preserve the value of this bit by read modify write. RsvdZ Reserved and Zero Reserved for future RW1C implementations. The software must write zero to this bit."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#623-csr-offset-in-bars","title":"6.2.3 CSR Offset in BARs","text":"The table below captures the FIM and AFU features in the supported BAR regions. The highlighted offset indicates the first DFH in the DFH list of a BAR region where device driver starts the DFH traversal.
Table 6-7: PF0 BAR0 Features
Offset Feature CSR set 0x0_0000 FME 0x0_1000 Thermal Management 0x0_3000 Global Performance 0x0_4000 Global Error 0x1_0000 PCIe 0x1_2000 QSFP0 0x1_3000 QSFP1 0x1_4000 HSSI (Ethernet) 0x1_5000 EMIF 0x2_0000 PMCI 0x4_0000 ST2MM 0x6_0000 UART 0x7_0000 Port Gasket PR logic 0x7_1000 Port Gasket Port interface 0x7_3000 Port Gasket Remote Signal Tap 0x8_0000 AFU Interface Handler- AFU Errors Table 6-8: PF0 BAR4 Features
Offset Feature CSR set 0x0_3000 MSI-X Vector Tables"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#7-clocks","title":"7 Clocks","text":"The following table provides external clock sources which correspond to pins on the FPGA that drive blocks of internal logic or are used as a reference clock for internal PLLs. The names in the table are used in the top.sv or are the logical clock names used by their respective functional blocks.
Table 7-1: External Clock Source
Clock Frequency Description N6001 fseries-dk iseries-dk SYS_REFCLK 100 MHz Reference clock to system IOPLL (sys_pll) which provides FIM system clocks. \u2714 \u2714 \u2714 PCIE_REFCLK0 100MHz PCIe reference clock 0 \u2714 \u2714 \u2714 PCIE_REFCLK1 100MHz PCIe reference clock 1 \u2714 \u2716 \u2714 qsfp_ref_clk 156.25 MHz Ethernet Reference Clock \u2714 \u2714 \u2714 ddr4_mem[0].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2714 \u2714 ddr4_mem[1].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2714 \u2714 ddr4_mem[2].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2716 \u2714 ddr4_mem[3].ref_clk 150MHz Refclk for 32-bit EMIF channel 0 \u2714 \u2716 \u2714 ddr4_hps.ref_clk 150 MHz Refclk for HPS EMIF \u2714 \u2714 \u2716 sdm_config_clk | FPGA_OSC_CLK1 125 MHz SDM Configuration Clock \u2714 \u2714 \u2714 hps_refclk | HPS_OSC_CLK 25 MHz Refclk for HPS \u2714 \u2714 \u2716 Table 7-2: Internal Clocks
Internal clock generated by the IOPLL as outclk outputs.
outclk# Clock Frequency Description outclk0 clk_sys 500 MHz1 System clock. Primary clock for PCIe Datapath outclk1 clk_100m 100 MHz For RemoteSTP and user clock, or any logic that requires a 100 MHz clock. outclk2 clk_sys_div2 250 MHz1 System clock div2 outclk3 clk_ptp_slv 155.56 MHz Unused outclk4 clk_50m 50 MHz Drives Virtual UART outclk5 clk_sys_div4 125 MHz1 System clock div4 outclk6 clk_350m 333.33 MHz Unused 1 The system clock frequency can be changed using OFSS files at build time.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#8-reset","title":"8 Reset","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#81-reset-signals","title":"8.1 Reset Signals","text":"The FIM system reset signals are driven according to their respective requirements derived from their use cases. The behavioral code defining the reset operation is located in the file rst_ctrl.sv. The FIM System Reset Drivers table below provides a list of the input and feedback signals that compose the various resets.
Table 8-1: FIM System Reset Drivers
Reset Description PCIE_PERST_n pin Active low PCIe reset pin from the PCIe card edge that may be set by the host machine for cold or warm resets. nCONFIG pin Active low input to the FPGA that causes the FPGA to lose its configuration data, enter a reset state, and tri-state all I/O pins. Host software must reload the FPGA FIM after nCONFIG is activated. ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. pcie_reset_status Active high reset status from PCIe hard IP. When driven high, this signal indicates that the PCIe IP core is not ready for usermode. pll_locked Active high SYS IOPLL locked signal pcie_cold_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Cold Reset sequence has been completed. pcie_warm_rst_ack_n Reset Acknowledge signal from the PCIe subsystem that lets the Reset Controller know that the Warm Reset sequence has been completed. Upon power-on, the reset module in the FIM holds the FIM in reset until all the reset conditions are de-activated:
nPERST signal is asserted.- The
INIT_DONE pin is driven high to indicate core configuration is complete. - The SYS IOPLL is locked.
- The reset status from PCIe hard IP is de-asserted indicating the IP is ready for transfer.
The reset module places the FIM back into reset if any of these conditions becomes active again. The only way to invoke a system reset to the FIM after power-up is to de-assert the nPERST pin either by performing a warm reboot or through PCIe driver intervention. There are soft reset signals set aside to allow software to reset the Port, AFU and partial reconfiguration IP.
THe following table lists the reset outputs from the rst_ctrl.sv block.
\u200b Table 8-2: FIM System Reset Outputs
Reset Description ninit_done Active low signal indicating FPGA core configuration is complete and FPGA has entered usermode. This signal is provided by the configuration monitor block in rst_ctrl.sv. rst_n_sys System general reset synchronous to clk_sys. This is a warm reset of the FPGA logic. Sticky bits in the FME and other CSR registers will not be reset by this signal. rst_n_100m System general reset synchronous to clk_100m. rst_n_50m System general reset synchronous to clk_50m. rst_n_sys_div2 System general reset synchronous to clk_sys_div2. rst_n_ptp_slv System general reset synchronous to clk_ptp_slv. pwr_good_n Hardware reset conditioned by ninit_done and the pll_locked signal. The signal is generally set when power has been cycled or a physical hardware fault has occurred, requiring a reset of the FPGA logic. This signal is synchronous to clk_sys. pcie_cold_rst_ack_n Hardware reset conditioned by the pcie_reset_status which is a summary reset driven by the PCIe Hard IP logic tile on the FPGA die. This signal is synchronous to clk_sys. pcie_warm_rst_ack_n Soft reset conditioned by nPERST when the pcie_reset_status is not asserted, meaning a warm reset is desired. Cold reset sticky bits in the PCIe subsystem will not be reset by this signal."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#9-interrupts","title":"9 Interrupts","text":"The OFS platform supports interrupts through MSI-X feature. The MSI-X Interrupt feature handles FME and AFU interrupts. FME interrupts are primarily used to notify the host of error events happened in the FIM. When any of the bits in the FME error status registers is set, an interrupt request is sent to the MSI-X module. There are FME error status registers for OFS for Agilex FPGA features. An AFU sends interrupt to the MSI-X module in the PCIE SS on the AXI interrupt request channel. The MSI-X table entries and PBA vectors are implemented in the PCIE SS.
Please refer to the AXI Streaming IP for PCI Express User Guide for more information.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#10-external-memory-interface-emif","title":"10 External Memory Interface (EMIF)","text":"The table below shows the capabilities of the memory populated on the boards for each reference design. Note that ECC is only enabled by default for HPS memory channels. You may enable ECC on channels that support it by modifying the memory subsystem. Refer to the Shell Developer Guides for instructions.
Table 10-1 Memory Subsystem Configuration for OFS Agilex PCIe Attach Target Boards
Reference Design Target Board Memory Channel FPGA I/O Bank Width ECC Width ECC Default Speed Size n6001 0 2A x32 No ECC Disabled 2400 4 GB 1 3B x32 x8 Disabled 2400 4 GB 2 2B x32 No ECC Disabled 2400 4 GB 3 3A x32 x8 Disabled 2400 4 GB HPS 3D x32 x8 Enabled 2400 1 GB fseries-dk 0 2C, 2D x64 x8 Disabled 2400 8 GB 1 2E, 2F x64 No ECC Disabled 2400 8 GB HPS 3D x32 x8 Enabled 2400 8 GB iseries-dk 0 3C,3D x64 x8 Disabled 2666 8 GB 1 3A,3B x64 x8 Disabled 2666 8 GB 2 2C,2F x64 x8 Disabled 2666 8 GB 3 2B,2E x64 x8 Disabled 2666 8 GB The diagram below shows the general connectivity of the EMIF.
Figure 10-1: EMIF Interfaces
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#101-emif-csr","title":"10.1 EMIF CSR","text":"The CSRs for the memory subsystem reside at address 0x15000 in the BPF. The CSRs are defined in ofs-fim-common/src/fpga_family/agilex/mem_ss/xls/EMIF_CSR.xls.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#11-ethernet-subsystem","title":"11 Ethernet Subsystem","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#111-ethernet-subsystem-overview","title":"11.1 Ethernet Subsystem Overview","text":"The Ethernet Subsystem (hssi_ss) provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
The table below shows the validated Ethernet configurations for each target board.
Table 11-1: Validated Ethernet Configurations for example OFS FIMs for Agilex FPGAs
Reference Design Target Board Ethernet Configuration n6001 2x4x25 GbE 2x4x10 GbE 2x100 GbE fseries-dk 2x4x25 GbE iseries-dk 2x4x25 GbE 2x200 GbE 2x400 GbE You may use OFS Settings (OFSS) files to select one of these configurations at build time. Please refer to the Shell Developer Guides for instructions.
For more information on the Ethernet Subsystem IP, please refer to the Ethernet Subsystem Intel FPGA IP User Guide.
The table below describes how the QSFP lanes are mapped to the Ethernet Subsystem ports.
Table 11-2: Transceiver Subsystem Port Mapping
Ethernet-SSPort Number Reference Design Target Board n6001 fseries-dk iseries-dk 2x4x25 GbE 2x4x10 GbE 2x100 GbE 2x4x25 GbE 2x4x25 GbE 2x200 GbE 1x400 GbE 0 QSFPA0:3 25 GbE QSFPA0:3 10 GbE QSFPA0:3 100GCAUI-4 QSFPDD0:7 25 GbE QSFPDD10:7 25 GbE - - 1 25 GbE 10 GbE - 25 GbE 25 GbE - - 2 25 GbE 10 GbE - 25 GbE 25 GbE - - 3 25 GbE 10 GbE - 25 GbE 25 GbE - - 4 QSFPB0:3 25 GbE QSFPB0:3 10 GbE QSFPB0:3 100GCAUI-4 25 GbE 25 GbE - - 5 25 GbE 10 GbE - 25 GbE 25 GbE - - 6 25 GbE 10 GbE - 25 GbE 25 GbE - - 7 25 GbE 10 GbE - 25 GbE 25 GbE - - 8 - - - - - QSFPDD10:3 200GAUI-4 QSFPDD10:7 400GAUI-8 9 - - - - - - - 10 - - - - - - - 11 - - - - - - - 12 - - - - - QSFPDD14:7 200GAUI-4 - 13 - - - - - - - 14 - - - - - - - 15 - - - - - - - 16 N/A - - - - 17 N/A - - - - 18 N/A - - - - 19 N/A - - - - Figure 11-1: Transceiver Subsystem Block Diagram
A host exerciser, named he-hssi, is provided in the pr_slot of the AFU partition. The Ethernet Subsystem interface to the AFU has an Arm\u00ae AMBA\u00ae 4 AXI4 data and sideband interface.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#112-parameterization-options","title":"11.2 Parameterization Options","text":"The Ethernet Subsystem features are highly parameterized to provide the various features/configurations required for the different FIMs. The HSSI OFSS files found in ofs-agx7-pcie-attach/tools/ofss_config/hssi can be used to change the configuration of the HSSI-SS. You may also create your own OFSS file for custom configuration. The provided OFSS files are described in the following table:
Table: Provided HSSI OFSS Files
OFSS File Name Location Type Description Supported Board hssi_8x10.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x10 GbE N6001 hssi_8x25.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 8x25 GbE N6001 hssi_2x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 2x100 GbE N6001 hssi_1x400_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 1x400 GbE iseries-dk hssi_4x100.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be 4x100 GbE N6000 hssi_8x25_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP configuration to be F-Tile 8x25 GbE fseries-dk | iseries-dk hssi_2x200_ftile.ofss $OFSS_ROOTDIR/tools/ofss_config/hssi hssi Defines the Ethernet-SS IP to be 2x200 GbE iseries-dk The following parameters are defined in ofs-fim-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_plat_if_pkg.sv:
- MAX_NUM_ETH_CHANNELS: This indicates how many maximum ethernet channels are supported for platfrom
- NUM_QSFP_PORTS_USED: Indicates number of QSFP cages on the board that are used by the target HSSI configuration
- NUM_ETH_CHANNELS: Number of ethernet ports used by target hssi configuration. E.g. for 8x25G, 8 HSSI ports are active on HSSI IP
- NUM_QSFP_LANES: Number of lanes per QSFP cage
- NUM_LANES: Number of XCVR Lanes per Port used by the configuration. For ex. for 100GCAUI-4, 4 lanes per HSSI port are used
- ETH_PACKET_WIDTH: Datawidth of client side AXI-ST interface coming from HSSI SS IP. This is not an user configurabale parameter. User need update this value to reflect HSSI SS IP client side data width for selected configuration.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#113-ofs-ethernet-subsystem-interface-module","title":"11.3 OFS Ethernet Subsystem Interface Module","text":"A wrapper around the Transceiver Subsystem integrates the following features: * DFH registers * Control & status registers
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1131-ethernet-subsystem-control-and-status-register-csr-map","title":"11.3.1 Ethernet Subsystem Control and Status Register (CSR) Map","text":"The Ethernet Subsystem connects to the BPF which is memory mapped to PF0 BAR0. The Ethernet Subsystem CSR space in the FIM consists of two parts:
- HSSI Subsystem CSRs assigned from offset 0x000 to 0x7FF
- HSSI Wrapper CSRs are added to FIM at offset 0x800 to 0xFFF
The PCIe subsystem uses AXI Memory Mapped accesses to read and write HSSI Subsystem Control and Status Registers in the FIM. The HSSI Subsystem CSR Map structure is designed to scale according to IP capabilities.
The Ethernet Subsystem CSR Map can be found ofs-agx7-pcie-attach/ipss/hssi/HSSI_SS_CSR.xls.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#114-ethernet-subsytem-software","title":"11.4 Ethernet Subsytem Software","text":"There are two pieces of software related to running the Ethernet Subsystem: The Linux* dfl network driver and the user space OPAE Tools.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1141-ethernet-subsystem-linux-driver","title":"11.4.1 Ethernet Subsystem Linux Driver","text":"The Ethernet subystem is exposed as a feature in the PCIe PF BAR0 region. It has a Device Feature Header (DFH) specifying the interface.
The primary functionality of the driver is to interact with the ethernet MAC and PHY through an indirect register access mailbox implemented by the HSSI_RCFG_CMD0, HSSI_RCFG_DATA0 registers described above. To aid in RTL bringup, the driver provides a debugfs interface directly to the indirect register access mailbox. For each HSSI interface in the system there would be a directory with the following form containing two files, regaddr and regval: /sys/kernel/debug/dfl-fme.X.Y
To read a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then read the value as string out of regval file. To write a register offset in the MAC/PHY write the offset to the regaddr file as a C hex string (e.g. 0x04) and then write the value as a C hex string to regval file.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1142-ethernet-subsystem-opae-user-space-tool","title":"11.4.2 Ethernet Subsystem OPAE User Space Tool","text":"User space OPAE Tools that are part of OPAE SDK provide support for the Ethernet Subsystem. You can use the --help option to print more information about each of these commands:
- hssi: Provides a means of interacting with the 10G and 100G HSSI AFUs through the host exerciser.
- hssiloopback: Enables and disables Ethernet loopback.
- hssistats: Provides MAC statistics.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#12-partial-reconfiguration","title":"12 Partial Reconfiguration","text":"Partial Reconfiguration (PR) is an Altera FPGA technology that allows users to reconfigure parts of the FPGA device dynamically, while the remainder of the device continues to operate. In a non-partial reconfiguration flow, any change to the design requires full reprogramming of the entire configuration RAM (CRAM) arrays in the device. With partial reconfiguration, you can dynamically reprogram one or more CRAM frames. A partial reconfiguration design has a static region, and a PR region, which can be modified to implement new logic. The portion of the CRAM on the chip to be reconfigured is contained within a PR region.
For the PR flow, your design must be partitioned into static region and reconfigurable region. The static region is the area of your FPGA that is not reconfigured without reprogramming the entire FPGA. An area of the chip that you plan to partially reconfigure is a PR region.
The Port Gasket contains all the PR specific modules and logic, such as PR slot reset/freeze control, user clock, remote STP etc. For this reference example only one PR slot is supported.
The Figure below shows the fundamental concepts required to support PR in OFS platform. There are 4 OFS management DFH \u2013 PR, Port, User Clock and Remote STP in Port Gasket that is exposed to OFS software. These platform capabilities are generally used in conjunction to Partial Reconfiguration. The Figure below also demonstrates the concepts of adding a new interface to the PR region.
Figure 12-1 Partial Reconfiguration Gasket
The isolation logic is provided on the output signals of the PR region to ensure they don\u2019t glitch and affect the interfacing logic in the Static Region (SR). The isolation logic is controlled by the PR Freeze logic during PR operation.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#user-clock-support","title":"User Clock Support","text":"The reference platform provides two user configurable clock (uclk_usr, uclk_usr_div2) for use by the AFU. These clocks are generated by a reconfigurable IOPLL. The control and status of these clocks is expose through two pairs of command and status registers (USR_CLK_FREQ_CMD0 / USR_CLK_FREQ_STS0 & USR_CLK_FREQ_CMD1 / USR_CLK_FREQ_STS1). The first pair controls the fPLL reconfiguration functionality. The second pair controls a clock frequency measurement block.
The following are the default values of the userclk.
uclk_usr: 312.5 MHz
uclk_usr_div2: 156.2 MHz
Table 12-1 usr_clk_* Acceptable Programming Range
Fabric Frequency Range uclk_usr (H) uclk_usr_div2 (L) Details 0-400 MHz 0-800 MHz 0-400 MHz Clocks set on 2x relationship for L<400 MHz 400-800 MHz 400-800 MHz 400-800 MHz Clks are equal for L>400 MHz"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#pll-reconfiguration","title":"PLL Reconfiguration","text":"The blue bit stream hardware exposes the low level IOPLL reconfiguration interfaces directly to software control. Through the USR_CLK_FREQ_CMD0 register software can select the reference clock, assert the PLL power down pin and issue reads and writes on the IOPLL Avalon-mm reconfiguration bus. Through the USR_CLK_FREQ_STS0 register software can query the IOPLL active reference clock, locked status and readdata returned from the IOPLL AVMM interface for read requests.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#clock-frequency-counter","title":"Clock Frequency Counter","text":"The user clocks, generated by the reconfigurable IOPLL, are connected to a frequency measurement circuit. Software selects which of the two clocks to measure by setting the clock select bit in the USER_CLK_FREQ_CMD1 register. After 10ms software can read the frequency, in 10 KHz units, from the USER_CLK_FREQ_STS1 register. Reading this register before 10ms has passed will return undefined data but otherwise has no effect.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#configuring-user-clocks","title":"Configuring User Clocks","text":"To program the user clock to the desired frequency, user will set the clock-frequency-low and clock-frequency-high fields in the PR AFU GBS .json file to the desired frequency value. During PR, SW will try to provide the closest possible frequency to the value specified in the .json file.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#13-host-exercisers","title":"13 Host Exercisers","text":"The Host Exerciser (HE) modules are responsible for generating traffic with the intent of exercising the specific interface they are designed to connect to. They are intended to test the interface in simulation and hardware, enable measurement of bandwidth and other performance KPIs and, in come cases, provide an example of data movement between interfaces (PCIe to DDR for e.g.) for adoption into a customer application.
### 13.1 HE-LB and HE-MEM Host Exerciser
The Host Exerciser Loopback module is a traffic generator that can be attached both to host memory, over PCIe (HE-LB), and to local memory, such as board-attached DDR (HE-MEM). The Host Exerciser (HE) is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth as well as demonstrating data path correctness. The FIM picks up the HE-LB module behind the PIM (Platform Interface Manager). The PIM converts the AXI with TLP out of the PCIe SS to standard Arm\u00ae AMBA\u00ae 4 AXI4 (MM for completions and Lite for CSR) interfaces. The figure below shows how the HE-LB is instantiated in the FIM.
Figure 13-1 HE-LB Dataflow Diagram
The exerciser has two primary modes: loopback, in which read data is fed back as writes, and read/write mode, in which reads and writes are generated independently. Furthermore, the host_exerciser software provided with OPAE that drives the RTL has two major modes: \"lpbk\" and \"mem\". These modes only select the UUID of the HE AFU, with lpbk selecting a version configured with no local memory (56e203e9-864f-49a7-b94b-12284c31e02b) and mem seeking a version with local memory (8568ab4e-6ba5-4616-bb65-2a578330a8eb). The behavior of each engine with and without local memory is described below.
Figure 13-2 HE Engine Hierarchy
For more details of HE-LB and HE-MEM IP, please refer to ofs-fim-common/src/common/he_lb/README.md
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#132-hssi-host-exerciser-he-hssi","title":"13.2 HSSI Host Exerciser (HE-HSSI)","text":"HE-HSSI is an Ethernet AFU that handles client side ethernet traffic. The reference HE-HSSI has following features:
- HE-HSSI provides a compatible interface with the Ethernet Subsystem.
- Includes a traffic generator and checker or monitor
- Provides pause signals to the Ethernet Subsystem for XON and XOFF generation
- Generates traffic or incoming traffic that can be looped back into transmit path by enabling loopback mode, which will bypass traffic generator
- At the HE-HSSI interface boundary the Ethernet data interface is AXI4-Stream with 64-bit data at eth_clk clock
- The Traffic generator and checker modules have a 64-bit data interface at eth_clk clock.
- The traffic generator supports the following modes:
- Fixed length or Random Length
- Incremental pattern or Random pattern
- Traffic checker does a 32-bit crc check in 10/25G. In the 100G configuration, there is no data integrity check, only a packet counter. (TBD)
- The CSR of this AFU is accessible through AXI4-Stream PCIe TLP interface
- The PCIe TLP to CSR Interface Conversion module converts PCIe TLPs into simple CSR interface
- The CSR space of the traffic generator and checker modules are accessed in an indirect way using mailbox registers
- If used for more than one channel, each channel has a separate traffic generator and traffic checker with separate CSR space.
- Reads and Writes to individual traffic controller CSR spaces can be done by selecting that particular channel using channel select register.
The HE-HSSI Ethernet block diagram is below.
Figure 13-3: HE-HSSI Block Diagram Block Diagram
The diagram below shows the path followed depending on if you enable loopback mode in HE-HSSI or not. In loopback mode, traffic is looped back into the transmit path which will bypass traffic. Alternatively, the traffic generates traffic.
Figure 13-4: HE-HSSI Operation Mode Traffic Patterns
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1322-he-hssi-csr-map","title":"13.2.2 HE-HSSI CSR Map","text":"The reference HSSI AFU contains the following registers and a similar arrangement of register space can be implemented for other usecase specific HSSI AFUs. * AFU DFH Register: Device feature header for the AFU (AFU_DFH) * AFU ID Registers: 128-bit UUID for the AFU which occupies two 64-bit registers (AFU_ID_L, AFU_ID_H) * Mailbox Registers: Command and Data register for traffic controller register access. It follows the standard access method defined for OFS. Access method and implementation is same as Reconfiguration Interface defined for the HSSI FIM. (TRAFFIC_CTRL_CMD, TRAFFIC_CTRL_DATA) * Channel Select Register: Channel select register for traffic controller mailbox access. It is used in cases where more than one channel is in the AFU, else it defaults to zero, meaning channel-0 is selected. (TRAFFIC_CTRL_PORT_SEL) * Scratchpad Register: Scratchpad register for CSR access checking. (AFU_SCRATCHPAD)
The CSR excel for HE-HSSI module can be found at ofs-common/src/common/he_hssi/HE_HSSI_CSR.xls.
13.3 HE-Null Overview
This module is a simple stub that is used to replace various HE and other blocks in the FIM whenever they are bypassed using the build script command line options such as null_he_lb, null_he_hssi, null_he_mem and null_he_mem_tg. To find out more about these compiler directives, refer to the Shell Developer Guides.
Table 13-1 HE-Null DFH
REGISTER NAME ADDRESS ACCESS DEFAULT DESCRIPTION DFH 0x0000 RO 0x0000_0000_1000_0000 DFH register GUID_L 0x0008 RO 0xaa31f54a3e403501 Lower 64b of GUID GUID_H 0x0010 RO 0x3e7b60a0df2d4850 Upper 64b of GUID SCRATCHPAD 0x0018 RW 0x0 Scratchpad Figure 13-5: HE-Null Block Diagram
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14-reliability-accessibility-serviceability-ras-and-error-handling-tbd","title":"14 Reliability, Accessibility, Serviceability (RAS) and Error Handling (TBD)","text":" - Downstream AFU checker: Identifies AFU violations. For example, these checker flags violations of the interface specification.
- Upstream software or PCIe link checker: Identifies invalid traffic from PCIe that violates FIM or PCIe specifications. For example, this checker flags an application sending traffic if it violates the FIM specification or creates a PCIe link issue by causing completion timeout or malformed TLP.
- FIM - Checks for bugs in the FIM fabric.
Errors reported by the checker are logged in either the FME error registers or Port error registers, or both, as shown in the table below. For more details on each of the registers, please refer to ofs-fim-common/src/common/fme/xls/n6001/FME_CSR.xls or the SystemVerilog file: ofs-common/src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#141-fme-errors","title":"14.1 FME Errors","text":""},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1411-fme_error0","title":"14.1.1 FME_ERROR0","text":"The FME_ERROR0 register flags CoreFIM FME errors in the Global Error (GLBL_ERROR) private feature. The error bits in this register are sticky bits. You can only clear these bits through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in FME_ERROR0_MASK register masks the error.
Table 14-2: FME Error Types
Error Type Description Fabric errors FIFO underflow/overflow condition in CoreFIM. These errors only occur if you have introduced bugs into the FIM or during very rare single event upset (SEU) or SEU-like events. Invalid port access A port can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the Port. If it finds a PF is trying to access a port that is mapped to a VF or vice-versa, an error will be reported. Invalid AFU access An AFU can either be mapped to a PF or a VF, and not both. The checker reads the AfuAccessCtrl field in the FME CSR PORT0_OFFSET register to determine the access type of the AFU associated with the Port. If it finds a PF is trying to access an AFU that is mapped to a VF or vice-versa, an error is reported and a fake response is sent back to the requester to avoid a completion timeout on the host."},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1412-pcie0_error","title":"14.1.2 PCIE0_ERROR","text":"The PCIe Avalon-ST to AXI4-Stream bridge monitors the PCIe link for errors and logs any such errors in the PCIE0_ERROR register (in PCIE0 feature region) and PCIE0_ERROR register in the GLBL_ERR private feature. The error bits in the PCIE0_ERROR register are sticky bits that you can only clear through software or through a system reset. Writing a 1 to the error field in the register clears the corresponding error bit. Writing a 1 to the corresponding bit in PCIE0_ERROR0_MASK masks the error. (TBD)
If you have other external FME features, you can add similar _ERROR registers to this space. Please refer to the following spreadsheet in the release branch found at: ofs-fim-common/src/fpga_family/agilex/pcie_ss/PCIE_CSR.xls or the SystemVerilog file: ofs-fim-common/src/fpga_family/agilex/pcie_ss/pcie_csr.sv for more details on this register.
NOTE: The PCIE0_ERROR register is located in both the Global Error external feature memory space and a separate PCIe external feature memory space. OPAE software supports the PCIe external feature memory space beginning at offset 0x40000 for OFS EA and going forward. PCIe registers beginning at 0x4000 in the Global Error external feature memory space is there for backward compatibility to the Intel FPGA PAC D5005 v2.0.1 Acceleration Stack. (TBD)
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1413-fme_first_error-fme_next_error","title":"14.1.3 FME_FIRST_ERROR, FME_NEXT_ERROR","text":"The FME_FIRST_ERROR register flags which of the FME error reporting registers, such as FME_ERROR0, PCIE0_ERROR0, has reported the first error occurrence. The error fields of the first error register are then continuously logged into the FME_FIRST_ERROR register until a system reset or software clears all the errors in that first error register. Likewise, the FME_NEXT_ERROR indicates which of the FME error reporting registers (except the first error register) has reported the next occurrence of error after the first error register. The error fields of the next error register are continuously logged into the FME_NEXT_ERROR register until a system reset or software clears all the errors in the second error register.
Please refer to the file in the ofs-fim-common repository folder: src/common/fme/fme_csr.sv for individual register field descriptions or the SystemVerilog file src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1414-reliability-accessibility-serviceability-ras-error-status","title":"14.1.4 Reliability, Accessibility, Serviceability (RAS) Error Status","text":"The RAS feature in CoreFIM labels errors as non-fatal, fatal or catastrophic based on their impact to the system. * A non-fatal error usually originates from software or an AFU. With a non-fatal error, the user application may still be able to recover from the error by performing a soft reset on the AFU, fixing the user application software if necessary, and clearing the error. On the other hand, a fatal or catastrophic error is non-recoverable and requires the platform to be reset. * Non-fatal errors are logged in the RAS_NOFAT_ERR_STAT register and fatal or catastrophic errors are logged in the RAS_CATFAT_ERR_STAT register.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14141-non-fatal-errors","title":"14.1.4.1 Non-Fatal Errors","text":"The RAS_NOFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It logs the high-level status of non-fatal errors in the hardware. Unlike the error bits in the PCIE0_ERROR and FME_ERROR0 registers which are RW1C (software can write a 1 to clear the error), the error bits in this register are read-only and can only be cleared by system reset. Software has an option to mask the error using RAS_NOFAT_ERR_MASK. Please refer to the following file in the ofs-fim-common resository: src/common/fme/xls/n6000/FME_CSR.xls for individual register field descriptions or the SystemVerilog file: src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14142-catastrophic-fatal-errors","title":"14.1.4.2 Catastrophic & Fatal Errors","text":"The RAS_CATFAT_ERR_STAT is a read-only status register that is specifically added for the RAS feature. It captures the high-level status of errors that can only be recovered with a system reset. Therefore, the error bits in the RAS_CATFAT_ERR_STAT register are read-only and can only be cleared by system reset or masked through RAS_CATFAT_ERR_MASK. Please refer to the following file in the ofs-fim-common resository: src/common/fme/xls/n6000/FME_CSR.xls for individual register field descriptions or the SystemVerilog file: src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1415-ras-error-injection","title":"14.1.5 RAS Error Injection","text":"For software testing purposes, you can inject non-fatal, fatal and catastrophic errors into the platform through the RAS_ERROR_INJ register. These errors are reported in the RAS_CATFAT_ERR_STAT and RAS_NOFAT_ERR_STAT registers. Please refer to the following file in the ofs-fim-common resository: src/common/fme/xls/n6000/FME_CSR.xls for individual register field descriptions or the SystemVerilog file: src/common/fme/fme_csr.sv.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1416-fme-error-interrupts","title":"14.1.6 FME Error Interrupts","text":"In an event of an FME error, the MSI-X module in the FIM generates an interrupt so the host can decide on the next course of action. The FIM does not stall upstream and downstream traffic after the FME error. However, a port error triggered by invalid request from a user AFU stalls all the traffic going from AFU to PCIe. The interrupt capability is discoverable by querying the NumbSuppInterrupt field of the PORT_CAPABILITY register in the Port private feature. The MSI-X vector number is recorded in the InterruptVectorNumber field of the GLBL_ERROR_CAPABILITY register of the Global Error external feature.
An FME error interrupt is generated in response to the FME_ERROR0, PCIE0_ERROR0, RAS_NOFAT_ERR_STAT or RAS_CATFAT_ERR_STAT registers recording a new, unmasked, error per the rules defined by CoreFIM interrupt feature.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1417-fme-error-handling","title":"14.1.7 FME Error Handling","text":"When the host receives an FME error interrupt, it must take the recommended actions described below to bring the system back to its normal operation.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14171-catastrophicfatal-error","title":"14.1.7.1 Catastrophic/Fatal Error","text":"A system reset is mandatory for any catastrophic or fatal error.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#14172-non-fatal-error","title":"14.1.7.2 Non-Fatal Error","text":"When software receives a non-fatal error interrupt which does not require a system reset, it can take the following steps to clear the error after software handles the error: 1. Set the *_ERROR_MASK register to all 1\u2019s to mask all errors 2. Clear the *_FIRST_ERROR register 3. Clear the *_ERROR register 4. Set *_ERROR_MASK register to all 0\u2019s to enable all errors
- Result: The *_ERROR & *_FIRST_ERROR registers begin capturing new errors.
NOTE: A system reset can only clear RAS Error status registers.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#142-mmio-requests","title":"14.2 MMIO Requests","text":"The FIM is designed to gracefully handle MMIO request scenarios.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1421-unsupported-functions-and-bars","title":"14.2.1 Unsupported Functions and BARs","text":"The PCIe hard IP in the FIM guarantees that only TLP packets for the functions and BARs supported by the FIM (as configured in PCIe HIP IP instantiation) are sent to the FIM.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1422-mmio-request-decoding","title":"14.2.2 MMIO Request Decoding","text":"The packet router and memory decoder in the FIM ensure that only legal MMIO requests are forwarded to the targeted MMIO region. Full address and BAR decoding is done both in the packet router and the memory decoder to ensure the requests are forwarded to the designated CSR region as defined in the MMIO Regions section of this manual. Any unsolicited/illegal MMIO requests are dropped, and an error is reported back to host through the FME error register.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1423-unused-fmeport-csr-regions","title":"14.2.3 Unused FME/Port CSR Regions","text":"All the CSR slaves in the FIM which are mapped to the FME or Port CSR regions must always respond to MMIO read requests targeting its associated CSR region. A CSR slave must return all 0s for MMIO reads to its unused CSR region such as a reserved space or a region where no CSR register is implemented for the address. The FIM ensures MMIO reads to the FME or Port CSR regions that are not mapped to any CSR slave always gets a response of all 0s. The memory decoder module and fake responder module in the FIM provide this guaranteed response.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1424-unsupported-mmio-request","title":"14.2.4 Unsupported MMIO Request","text":"Any MMIO requests targeting FME or Port CSR regions with a length or address alignment that are not supported by the FIM are dropped, and an error is logged in PCIE0_ERROR register. The MMIO checker module in the FIM guarantees this response. When an unsupported MMIO read request to the FIM CSR region is detected, the FIM sends back a CPL (completion without data) with error status (UR) back to host.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1425-afu-access-violation","title":"14.2.5 AFU Access Violation","text":"AFU access violations refer to the scenarios where a PF is attempting to access the MMIO region of an AFU bound to a VF (virtualization enabled), or when a VF is trying to access the MMIO region of an AFU bound to a PF (virtualization disabled). When such a violation is detected, the FIM drops the request and logs the error in the FME_ERROR0 register. If the request is an MMIO read request, the FIM returns a fake response to the host.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#1426-afu-mmio-response-timeout","title":"14.2.6 AFU MMIO Response Timeout","text":"An AFU MMIO Response timeout functions in the same manner described in the MMIO Response Timeout section.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#15-ofs-design-hierarchy","title":"15 OFS Design Hierarchy","text":"Files for design, build, and unit test simulation are found at https://github.com/OFS/ofs-agx7-pcie-attach, release branch release/ofs-2024.2.
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#151-design-guidance","title":"15.1 Design Guidance","text":"The OFS FIM is designed with configurability and scalability in mind. Please refer to the shell devloper guide for your target board for detailed design guidance. You can find detailed instructions for customizing the FIM design.
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xR-tile, F-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (2xF-tile) FPGAs
- Shell Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach (P-tile, E-tile) FPGAs
"},{"location":"hw/n6001/reference_manuals/ofs_fim/mnl_fim_ofs_n6001/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122. \u00a0
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/","title":"Getting Started Guide: Open FPGA Stack for Intel\u00ae Agilex FPGAs Targeting the Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Last updated: November 19, 2024
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#10-introduction","title":"1.0 Introduction","text":"The purpose of this document is to help users get started in evaluating the 2024.2-1 version of the Open FPGA Stack (OFS) for the Intel Agilex FPGA targeting the Intel N6000/1-PL FPGA SmartNIC Platform. After reviewing the document a user shall be able to:
- Set up their server environment according to the Best Known Configuration (BKC)
- Build and install the OFS Linux Kernel drivers
- Build and install the Open Programmable Acceleration Engine Software Development Kit (OPAE SDK) software on top of the OFS Linux kernel drivers
- Load and Verify the Firmware and FIM versions loaded on their boards
- Verify the full stack functionality offered by the OFS solution
- Know where to find additional information on other OFS ingredients
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell targeting Intel N6000/1-PL FPGA SmartNIC Platforms. These platforms are Development Kits intended to be used as a starting point for evaluation and development.
Note: Code command blocks are used throughout the document. Commands that are intended for you to run are preceded with the symbol '$', and comments with '#'. Full command output may not be shown.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#12-terminology","title":"1.2 Terminology","text":""},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-1-terminology","title":"Table 1: Terminology","text":"Term Description AER Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. AFU Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region BBB Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. BKC Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against. BMC Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors. CSR Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software. DFL Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration. FIM FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FME FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform. HEM Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Intel VT-d Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. IOCTL Input/Output Control, System calls used to manipulate underlying device parameters of special files. JTAG Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. MMIO Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators. OFS Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs. OPAE SDK Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. PAC Programmable Acceleration Card: FPGA based Accelerator card PIM Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. PR Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page. RSU Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration. SR-IOV Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance. TB Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. UVM Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework. VFIO Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to user space."},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#13-references-and-versions","title":"1.3 References and Versions","text":"The OFS 2024.2-1 Release targeting the Intel\u00ae FPGA SmartNIC N6000/1-PL is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions which compose this release.
The following table highlights the hardware which makes up the Best Known Configuration (BKC) for the OFS 2024.2-1 release.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-2-hardware-bkc","title":"Table 2: Hardware BKC","text":"Component Version 1 x Intel\u00ae FPGA SmartNIC N6001-PL, SKU2 1 x Supermicro Server SYS-220HE 1 x Intel FPGA Download Cable II (Only Required for manual flashing) 1 x 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing) The following table highlights the versions of the software which compose the OFS stack. The installation of the OPAE SDK on top of the Linux DFL drivers will be discussed in their relevant sections in this document.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-3-software-version-summary","title":"Table 3: Software Version Summary","text":"Component Version FPGA Platform Intel\u00ae FPGA SmartNIC N6001-PL, release notes: https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 under \"Known Issues\" OPAE SDK 2.13.0-3 Backport Kernel Drivers intel-1.11.0-2 OneAPI-ASP ofs-2024.2-1 OFS FIM Source Code for N6001 ofs-2024.2-1 OFS FIM Common Resources Tag: ofs-fim-common-1.1.0-rc2 OFS Platform AFU BBB ofs-2024.2-1 Intel Quartus Prime Pro Edition Design Software* Quartus Prime Pro Version 24.1 for Linux Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platform you have. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers to be installed as shown in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-4-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 4: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"SKU Mapping SKU Value Primary Difference fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table highlights the programmable firmware versions that are supported on the Intel N6000/1-PL FPGA SmartNIC Platform in the OFS 2024.2-1 release. Programming and verifying these components is discussed in their respective sections.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-5-intel-fpga-smartnic-n60001-pl-programmable-component-version-summary","title":"Table 5: Intel\u00ae FPGA SmartNIC N6000/1-PL Programmable Component Version Summary","text":"Component Version PR Interface ID a791757d-38a6-5159-a7fc-e1a61157a07b Bitstream ID 360571656856467345 BMC RTL 3.15.2 BMC NIOS FW 3.15.2"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#14-reference-documents","title":"1.4 Reference Documents","text":"Documentation is collected on https://ofs.github.io/ofs-2024.2-1/.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#15-board-installation-and-server-setup","title":"1.5 Board Installation and Server Setup","text":"Platform installation instructions, server BIOS settings and regulatory information can be found in the Board Installation Guide: OFS for Acceleration Development Platforms.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-6-supermicro-server-bmc-bkc","title":"Table 6: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4)"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#20-ofs-stack-architecture-overview-for-reference-platform","title":"2.0 OFS Stack Architecture Overview for Reference Platform","text":""},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#21-hardware-components","title":"2.1 Hardware Components","text":"The OFS hardware architecture decomposes all designs into a standard set of modules, interfaces, and capabilities. Although the OFS infrastructure provides a standard set of functionality and capability, the user is responsible for making the customizations to their specific design in compliance with the specifications outlined in the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
OFS is a hardware and software infrastructure that provides an efficient approach to developing a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#211-fpga-interface-manager","title":"2.1.1 FPGA Interface Manager","text":"The FPGA Interface Manager (FIM), or shell of the FPGA provides platform management functionality, clocks, resets, and interface access to the host and peripheral features on the acceleration platform. The OFS architecture for Intel Agilex FPGA provides modularity, configurability and scalability. The primary components of the FPGA Interface Manager or shell of the reference design are:
- PCIe Subsystem - a hierarchical design that targets the P-tile PCIe hard IP and is configured to support Gen4 speeds and Arm AXI4-Stream Data Mover functional mode.
- Ethernet Subsystem - provides portability to different Ethernet configurations across platforms and generations and reusability of the hardware framework and software stack.
- Memory Subsystem - composed of 5 DDR4 channels; one HPS DDR4 bank, x40 (x32 Data and x8 ECC), 1200 MHz, 1GB each, and four Fabric DDR4 banks, x32 (no ECC), 1200 MHz, 4GB
- Hard Processor System - 64-bit quad core ARM\u00ae Cortex*-A53 MPCore with integrated peripherals.
- Reset Controller
- FPGA Management Engine - Provides a way to manage the platform and enable acceleration functions on the platform.
- AFU Peripheral Fabric for AFU accesses to other interface peripherals
- Board Peripheral Fabric for master to slave CSR accesses from Host or AFU
- Platform Management Controller Interface (PMCI) to the board management controller
The FPGA Management Engine (FME) provides management features for the platform and the loading/unloading of accelerators through partial reconfiguration. Each feature of the FME exposes itself to the kernel-level OFS drivers on the host through a Device Feature Header (DFH) register that is placed at the beginning of Control Status Register (CSR) space. Only one PCIe link can access the FME register space in a multi-host channel design architecture at a time.
Note: For more information on the FIM and its external connections, refer to the Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#212-afu","title":"2.1.2 AFU","text":"An AFU is an acceleration workload that interfaces to the FIM. The AFU boundary in this reference design comprises both static and partial reconfiguration (PR) regions. You can decide how you want to partition these two areas or if you want your AFU region to only be a partial reconfiguration region. A port gasket within the design provides all the PR specific modules and logic required to support partial reconfiguration. Only one partial reconfiguration region is supported in this design.
Similar to the FME, the port gasket exposes its capability to the host software driver through a DFH register placed at the beginning of the port gasket CSR space. In addition, only one PCIe link can access the port register space.
You can compile your design in one of the following ways:
- Your entire AFU resides in a partial reconfiguration region of the FPGA.
- The AFU is part of the static region and is compiled as a flat design.
- Your AFU contains both static and PR regions.
In this design, the AFU region is comprised of:
- AFU Interface handler to verify transactions coming from AFU region.
- PF/VF Mux to route transactions to and from corresponding AFU components: ST2MM module, Virtio LB stub, PCIe loopback host exerciser (HE-LB), HSSI host exerciser (HE-HSSI), Memory Host Exerciser (HE-MEM), Traffic Generator to memory (HE-MEM-TG), Port Gasket (PRG) and HPS Copy Engine.
- AXI4 Streaming to Memory Map (ST2MM) Module that routes MMIO CSR accesses to FME and board peripherals.
- Host exercisers to test PCIe, memory and HSSI interfaces (these can be removed from the AFU region after your FIM design is complete to provide more resource area for workloads)
- Basic HPS Copy Engine to copy second-stage bootloader and Linux OS image from Host DDR to HPS DDR.
- Port gasket and partial reconfiguration support.
- Component for handling PLDM over MCTP over PCIe Vendor Defined Messages (VDM)
The AFU has the option to consume native packets from the host or interface channels or to instantiate a shim provided by the Platform Interface Manager (PIM) to translate between protocols.
Note: For more information on the Platform Interface Manager and AFU development and testing, refer to the Workload Developer Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#22-ofs-software-overview","title":"2.2 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack and on kernel.org.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS Backport DFL driver software provides the bottom-most API to FPGA platforms for this release. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The Backport DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
Build and installation instructions are detailed in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit the opae reference page.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access. You have two options to install OPAE as discussed below - using pre-built packages offered by Intel, or building the source code locally.
The OPAE SDK can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions on the OFS 2024.2-1 Release Page.
You can choose to build and install the OPAE SDK from source, which is detailed in the Software Installation Guide: Open FPGA Stack for PCIe Attach.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#44-fpga-device-access-permissions","title":"4.4 FPGA Device Access Permissions","text":"Access to FPGA accelerators and devices is controlled using file access permissions on the Intel\u00ae FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
sudo chmod a+rw /dev/dfl-port.0\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#45-memlock-limit","title":"4.5 Memlock limit","text":"Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#46-opae-tools-overview","title":"4.6 OPAE Tools Overview","text":"The following section offers a brief introduction including expected output values for the utilities included with OPAE. A full explanation of each command with a description of its syntax is available in the opae-sdk GitHub repo. The following command outputs were captured using an N6001 device.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#461-board-management-with-fpgainfo","title":"4.6.1 Board Management with fpgainfo","text":"The fpgainfo utility displays FPGA information derived from sysfs files.
Displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac, security. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
Note: Your BItstream ID and PR Interface Id may not match the below examples.
The following examples walk through sample outputs generated by fpgainfo.
$ sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : a2b5fd0e7afca4ee6d7048f926e75ac2\nUser1 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\nUser2 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\n
$ sudo fpgainfo bmc\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\n( 1) VCCRT_GXER_0V9 Voltage : 0.91 Volts\n( 2) FPGA VCCIO_1V2 Voltage : 1.21 Volts\n( 3) Inlet 12V Aux Rail Current : 0.87 Amps\n( 4) FPGA E-Tile Temperature [Remote] : 47.00 Celsius\n( 5) AVDD_ETH_0V9_CVL Voltage : 1.48 Volts\n( 6) FPGA E-TILE Temperature #3 : 51.00 Celsius\n...\n(77) FPGA FABRIC Remote Digital Temperature#3 : 47.00 Celsius\n(78) MAX10 & Board CLK PWR 3V3 Inlet Current : 0.97 Amps\n(79) CVL Non Core Rails Inlet Current : 0.01 Amps\n(80) FPGA Core Voltage Phase 0 VR Temperature : 49.50 Celsius\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#462-sensor-monitoring-with-fpgad","title":"4.6.2 Sensor Monitoring with fpgad","text":"The fpgad is a service that can help you protect the server from crashing when the hardware reaches an upper non-recoverable or lower non-recoverable sensor threshold (also called as fatal threshold). The fpgad is capable of monitoring each of the 80 sensors reported by the Board Management Controller. This service is only available once the installation instructions in sections 3.2 Building and Installing the OFS DFL Kernel Drivers and 4.1 OPAE SDK Build Environment Setup have been completed . Note: Qualified OEM server systems should provide the required cooling for your workloads. Therefore, using fpgad may be optional.
When the opae-tools-extra-2.13.0-3.x86_64 package is installed, fpgad is placed in the OPAE binaries directory (default: /usr/bin). The configuration file fpgad.cfg is located at /etc/opae. The log file fpgad.log which monitors fpgad actions is located at /var/lib/opae/. The fpgad periodically reads the sensor values and if the values exceed the warning threshold stated in the fpgad.conf or the hardware defined warning threshold, it masks the PCIe Advanced Error Reporting (AER) registers for the Intel N6000/1-PL FPGA SmartNIC Platform to avoid system reset. Use the following command to start the fpgad service:
Use the following command to start the fpgad service:
$ sudo systemctl start fpgad\n
The configuration file only includes the threshold setting for critical sensor 12V Aux Rail Voltage (sensor 29). This sensor does not have a hardware defined warning threshold and hence fpgad relies on the configuration file. The fpgad uses information contained within this file to mask the PCIe AER register when the sensor reaches the warning threshold. You may create another entry below the 12V Aux Voltage entry for any other sensors on the board. The updated configuration file includes a new entry for (18) Board Front Side Temperature with arbitrary values:
{\n\"configurations\": {\n\"fpgad-xfpga\": {\n\"configuration\": {\n},\n\"enabled\": true,\n\"plugin\": \"libfpgad-xfpga.so\",\n\"devices\": [\n[ \"0x8086\", \"0xbcc0\" ],\n[ \"0x8086\", \"0xbcc1\" ]\n]\n},\n\"fpgad-vc\": {\n\"configuration\": {\n\"cool-down\": 30,\n\"get-aer\": [\n\"setpci -s %s ECAP_AER+0x08.L\",\n\"setpci -s %s ECAP_AER+0x14.L\"\n],\n\"disable-aer\": [\n\"setpci -s %s ECAP_AER+0x08.L=0xffffffff\",\n\"setpci -s %s ECAP_AER+0x14.L=0xffffffff\"\n],\n\"set-aer\": [\n\"setpci -s %s ECAP_AER+0x08.L=0x%08x\",\n\"setpci -s %s ECAP_AER+0x14.L=0x%08x\"\n],\n\"config-sensors-enabled\": true,\n\"sensors\": [\n{\n\"name\": \"12V AUX Voltage\",\n\"low-warn\": 11.40,\n\"low-fatal\": 10.56\n},\n{\n\u201cname\u201d: \u201c3V3 VR Temperature\u201d,\n\u201clow-warn\u201d: 50.00,\n\u201clow-fatal\u201d: 100.00\n}\n]\n},\n\"enabled\": true,\n\"plugin\": \"libfpgad-vc.so\",\n\"devices\": [\n[ \"0x8086\", \"0x0b30\" ],\n[ \"0x8086\", \"0x0b31\" ],\n[ \"0x8086\", \"0xaf00\" ],\n[ \"0x8086\", \"0xbcce\" ]\n]\n}\n},\n\"plugins\": [\n\"fpgad-xfpga\",\n\"fpgad-vc\"\n]\n}\n
You can monitor the log file to see if upper or lower warning threshold levels are hit. For example:
$ tail -f /var/lib/opae/fpgad.log | grep \u201csensor.*warning\u201d\nfpgad-vc: sensor ' Columbiaville Die Temperature ' warning\n
You must take appropriate action to recover from this warning before the sensor value reaches upper or lower fatal limits. On reaching the warning threshold limit, the daemon masks the AER registers and the log file will indicate that the sensor is tripped. Sample output: Warning message when the 'CVL Core0 Voltage VR Temperature' exceeds the upper warning threshold limit
$ tail -f /var/lib/opae/fpgad.log fpgad-vc: sensor 'CVL Core Voltage VR Temperature' warning.\nfpgad-vc: saving previous ECAP_AER+0x08 value 0x057ff030 for 0000:b0:02.0\nfpgad-vc: saving previous ECAP_AER+0x14 value 0x0000f1c1 for 0000:b0:02.0\nfpgad-vc: sensor 'CVL Core Voltage VR Temperature' still tripped.\n
If the upper or lower fatal threshold limit is reached, then a power cycle of server is required to recover the Intel N6000/1-PL SmartNIC FPGA Platform. AER is unmasked by the fpgad after the sensor values are within the normal range which is above the lower warning or below the upper warning threshold.
To stop fpgad:
$ sudo systemctl stop fpgad.service\n
To check status of fpgad:
$ sudo systemctl status fpgad.service\n
Optional: To enable fpgad to re-start on boot, execute
$ sudo systemctl enable fpgad.service\n
For a full list of systemctl commands, run the following command:
$ systemctl -h\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#463-updating-with-fpgasupdate","title":"4.6.3 Updating with fpgasupdate","text":"The fpgasupdate tool updates the Intel Max10 Board Management Controller (BMC) image and firmware (FW), root entry hash, and FPGA Static Region (SR) and user image (PR). The fpgasupdate tool only accepts images that have been formatted using PACsign. If a root entry hash has been programmed onto the board, then you must also sign the image using the correct keys. Refer to the Security User Guide: Open FPGA Stack for information on created signed images and on programming and managing the root entry hash.
The Intel\u00ae FPGA SmartNIC N6000/1-PL ships with a factory, user1, and user2 programmed image for both the FIM and BMC FW and RTL on all cards. The platform ships with a single FIM image that can be programmed into either user1 or user2, depending in the image selected.
Use the following chart for information on the Bitstream ID and Pr Interface ID, two unique values reported by fpgainfo which can be used to identify the loaded FIM.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-7-fim-version-summary-for-ofs-20242-1-release","title":"Table 7: FIM Version Summary for OFS 2024.2-1 Release","text":"FIM Version Bitstream ID Pr Interface ID File Name Download Location ofs-2024.2-1 360571656856467345 a791757d-38a6-5159-a7fc-e1a61157a07b ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2024.2-1 Release Page ofs-n6001-0.9.0-rc2 0x50102025AD3DD11 92ec8960-2f2f-5544-9804-075d2e8a71a1 ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2.3.0 Release Page OFS-2.3.0 0x50102022267A9ED f59830f7-e716-5369-a8b0-e7ea897cbf82 ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2.3.0 Release Page OFS-2.2.0 0x501020295B081F0 8c157a52-1cf2-5d37-9514-944af0a060da ofs_top_page[\u00bd]_unsigned_user[\u00bd].bin ofs-2.2.0-beta Release Page"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-8-bmc-version-summary-for-ofs-20242-1-release","title":"Table 8: BMC Version Summary for OFS 2024.2-1 Release","text":"BMC FW and RTL Version File Name Download Location 3.15.2 AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu n/a -
Example loading a new version of the BMC RTL and FW.
$ sudo fpgasupdate AC_BMC_RSU_user_retail_3.11.0_unsigned.rsu <PCI ADDRESS>\n[2022-04-14 16:32:47.93] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:32:47.93] [INFO ] updating from file /home/user/AC_BMC_RSU_user_retail_3.11.0_unsigned.rsu with size 904064 [2022-04-14 16:32:47.94] [INFO ] waiting for idle [2022-04-14 16:32:47.94] [INFO ] preparing image file [2022-04-14 16:33:26.98] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [904064/904064 bytes][Elapsed Time: 0:00:00.00] [2022-04-14 16:33:26.98] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [Elapsed Time: 0:00:26.02] [2022-04-14 16:33:53.01] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:33:53.01] [INFO ] Secure update OK [2022-04-14 16:33:53.01] [INFO ] Total time: 0:01:05.07\nsudo rsu bmcimg\n
-
Example for loading a Static Region (SR) update image. This process will take up to 20 minutes.
$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:42:31.58] [INFO ] updating from file ofs_top_page1_pacsign_user1.bin with size 19928064 [2022-04-14 16:42:31.60] [INFO ] waiting for idle [2022-04-14 16:42:31.60] [INFO ] preparing image file [2022-04-14 16:42:38.61] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] [2022-04-14 16:42:54.63] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] [2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:49:11.03] [INFO ] Secure update OK [2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#464-signing-images-with-pacsign","title":"4.6.4 Signing Images with PACSign","text":"PACSign is an OPAE utility which allows users to insert authentication markers into bitstreams targeted for the Intel\u00ae FPGA SmartNIC N6000/1-PL. PACSign also allows users updating their Static Region (SR) to designate which partition of flash (user1, user2, factory) to overwrite given a specific FIM binary image. All binary images must be signed using PACSign before fpgasupdate can use them for an update. Assuming no Root Entry Hash (REH) has been programmed on the device, the following examples demonstrate how to prepend the required secury authentication data, and specifiy which region of flash to update. More information, including charts detailing the different certification types and their required options, are fully described in the PACsign README on GitHub.
For more information on PACSign and on general security practices surrounding the Intel N6001-PL FPGA SmartNIC device, visit the Security User Guide: Intel Open FPGA Stack.
PACSign can be run on images that have previously been signed. It will overwrite any existing authentication data.
The following example creates an unsigned SR image from an existing signed SR binary update image, targeting the user1 partition in flash.
$ PACSign SR -t UPDATE -s 0 -H openssl_manager -i ofs_top_page1_pacsign_user1.bin -o new_image.bin\nNo root key specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo CSK specified. Generate unsigned bitstream? Y = yes, N = no: y\nNo root entry hash bitstream specified. Verification will not be done. Continue? Y = yes, N = no: y\n2021-10-18 14:42:54,490 - PACSign.log - WARNING - Bitstream is already signed - removing signature blocks\n
--->
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#465-loading-images-with-rsu","title":"4.6.5 Loading Images with rsu","text":"The rsu performs a Remote System Update operation on a N6000/1-PL device, given its PCIe address. An rsu operation sends an instruction to the device to trigger a power cycle of the card and forces reconfiguration from flash for either the BMC or FPGA image.
The Intel\u00ae FPGA SmartNIC N6000/1-PL contains two regions of flash you may store FIM images. These locations are referred to as user1 and user2. After an image has been programmed with fpgasupdate in either of these regions you may choose to perform an rsu to switch. This operation indicates to the BMC which region to configure the FPGA device from after power-on.
If the factory image has been updated, Intel strongly recommends the user to immediately RSU to the factory image to ensure the image is functional.
The user can determine which region of flash was used to configure their FPGA device using the command fpgainfo fme and referring to the row labelled Boot Page.
$ sudo fpgainfo fme | grep Boot\nBoot Page : user1\n
Swapping between user1 and user2 skips load times that are created when using fpgasupdate to flash a new FIM image.
rsu Overview
Mode 1: RSU
Perform RSU (remote system update) operation on a development platform given its PCIe address. An RSU operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either BMC, Retimer, SDM, (on devices that support these) or the FPGA.
Mode 2: Default FPGA Image
Set the default FPGA boot sequence. The --page option determines the primary FPGA boot image. The --fallback option allows a comma-separated list of values to specify fallback images.
The following example will load an image stored in user2.
$ sudo rsu fpga --page=user2 0000:b1:00.0\n2022-04-15 09:25:22,951 - [[pci_address(0000:b1:00.0), pci_id(0x8086, 0xbcce)]] performing RSU operation\n2022-04-15 09:25:22,955 - [[pci_address(0000:b0:02.0), pci_id(0x8086, 0x347a)]] removing device from PCIe bus\n2022-04-15 09:25:22,998 - waiting 10 seconds for boot\n2022-04-15 09:25:33,009 - rescanning PCIe bus: /sys/devices/pci0000:b0/pci_bus/0000:b0\n2022-04-15 09:25:34,630 - RSU operation complete\n
Note: As a result of using the rsu command, the host rescans the PCI bus and may assign a different Bus/Device/Function (B/D/F) value than the originally assigned value.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#466-verify-fme-interrupts-with-hello_events","title":"4.6.6 Verify FME Interrupts with hello_events","text":"The hello_events utility is used to verify FME interrupts. This tool injects FME errors and waits for error interrupts, then clears the errors.
Sample output from sudo hello_events.
$ sudo hello_events\nWaiting for interrupts now...\ninjecting error\nFME Interrupt occurred\nSuccessfully tested Register/Unregister for FME events!\nclearing error\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#467-host-exercisor-modules","title":"4.6.7 Host Exercisor Modules","text":"The reference FIM and unchanged FIM compilations contain Host Exerciser Modules (HEMs). These are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. There are three HEMs present in the Intel OFS Reference FIM - HE-LPBK, HE-MEM, and HE-HSSI. These exercisers are tied to three different VFs that must be enabled before they can be used. Execution of these exercisers requires you bind specific VF endpoint to vfio-pci. The host-side software looks for these endpoints to grab the correct FPGA resource.
Refer to the Intel Shell Technical Reference Manual: OFS for Agilex\u00ae 7 PCIe Attach FPGAs for a full description of these modules.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-9-module-pfvf-mappings","title":"Table 9: Module PF/VF Mappings","text":"Module PF/VF ST2MM PF0 HE-MEM PF0-VF0 HE-HSSI PF0-VF1 HE-MEM_TG PF0-VF2 HE-LB Stub PF1-VF0 HE-LB PF2 VirtIO LB Stub PF3 HPS Copy Engine PF4"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#4671-he-mem-he-lb","title":"4.6.7.1 HE-MEM / HE-LB","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc. Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: - Latency (AFU to Host memory read) - MMIO latency (Write+Read) - MMIO BW (64B MMIO writes) - BW (Read/Write, Read only, Wr only)
Host Exerciser Loopback Memory (HE-MEM) AFU is used to exercise use of FPGA connected DDR, data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host.
HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller. Both exercisers rely on the user-space tool host_exerciser. When using the Intel N6001-PL FPGA SmartNIC Platform, optimal performance requires the exercisers be run at 400 MHz.
Execution of these exercisers requires you to bind specific VF endpoint to vfio-pci. The following commands will bind the correct endpoint for a device with B/D/F 0000:b1:00.0 and run through a basic loopback test.
Note: While running the opae.io init command listed below, if no output is present after completion then the command has failed. Double check that Intel VT-D and IOMMU have been enabled in the kernel as discussed in step 12 in section 3.1 Intel OFS DFL Kernel Driver Environment Setup.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci Binding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci iommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188 Assigning /dev/vfio/188 to DCPsupport Changing permissions for /dev/vfio/188 to rw-rw----\n\n$ sudo host_exerciser --clock-mhz 400 lpbk\nstarting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 2895\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 9.055 GB/s\n Test lpbk(1): PASS\n
The following example will run a loopback throughput test using one cacheline per request.
$ sudo pci_device 0000:b1:00.0 vf 3\n$ sudo opae.io init -d 0000:b1:00.2 user:user\n$ sudo host_exerciser --clock-mhz 400 --mode trput --cls cl_1 lpbk\nstarting test run, count of 1\nAPI version: 4\nBus width: 64 bytes\nAFU clock from command line: 400 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 512\nHost Exerciser numWrites: 513\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 1663\nTotal number of Reads sent: 512\nTotal number of Writes sent: 512\nBandwidth: 15.763 GB/s\n Test lpbk(1): PASS\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#4672-traffic-generator-afu-test-application","title":"4.6.7.2 Traffic Generator AFU Test Application","text":"Beginning in OPAE version 2.0.11-1+ the TG AFU has an OPAE application to access & exercise traffic, targeting a specific bank. The supported arguments for test configuration are:
- Number of test loops: --loops
- Number of read transfers per test loop: -r,--read
- Number of write transfers per test loop: -w,--write
- Burst size of each transfer: -b,--bls
- Address stride between each transfer: --stride
- Target memory TG: -m,--mem-channel
Below are some example commands for how to execute the test application. To run the preconfigured write/read traffic test on channel 0:
$ mem_tg tg_test\n[2024-06-26 10:27:31.317] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 0:\nTG PASS\nMem Clock Cycles: 127\nDEBUG: wcnt_ 1\nDEBUG: rcnt_ 1\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1\nDEBUG: num_ticks 127\nWrite BW: 0.151181 GB/s\nRead BW: 0.151181 GB/s\nThread on channel 0 exited with status 0\n[2024-06-26 10:27:31.318] [tg_test] [info] Test tg_test(1): PASS\n
Target channel 1 with a 1MB single-word write only test for 1000 iterations
$ mem_tg --loops 1000 -r 0 -w 2000 -m 1 tg_test\n[2024-06-26 10:28:17.861] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 4379946\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 0\nDEBUG: bcnt_ 1\nDEBUG: loop_ 1000\nDEBUG: num_ticks 4379946\nWrite BW: 8.76723 GB/s\nRead BW: 0 GB/s\n\nThread on channel 1 exited with status 0\n
Target channel 2 with 4MB write/read test of max burst length for 10 iterations
$ mem_tg --loops 10 -r 8 -w 8 --bls 255 -m 2 tg_test\n[2024-06-26 10:29:04.653] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 2:\nTG PASS\nMem Clock Cycles: 89462\nDEBUG: wcnt_ 8\nDEBUG: rcnt_ 8\nDEBUG: bcnt_ 255\nDEBUG: loop_ 10\nDEBUG: num_ticks 89462\nWrite BW: 4.37817 GB/s\nRead BW: 4.37817 GB/s\n\nThread on channel 2 exited with status 0\n
$ sudo mem_tg --loops 1000 -r 2000 -w 2000 --stride 2 --bls 2 -m 1 tg_test\n[2024-06-26 10:29:27.509] [tg_test] [info] starting test run, count of 1\nMemory channel clock frequency unknown. Assuming 300 MHz.\nChannel 1:\nTG PASS\nMem Clock Cycles: 17565290\nDEBUG: wcnt_ 2000\nDEBUG: rcnt_ 2000\nDEBUG: bcnt_ 2\nDEBUG: loop_ 1000\nDEBUG: num_ticks 17565290\nWrite BW: 4.37226 GB/s\nRead BW: 4.37226 GB/s\n\nThread on channel 1 exited with status 0\n[2024-06-26 10:29:27.568] [tg_test] [info] Test tg_test(1): PASS\n
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#4673-he-hssi","title":"4.6.7.3 HE-HSSI","text":"HE-HSSI is responsible for handling client-side ethernet traffic. It wraps the 10G and 100G HSSI AFUs, and includes a traffic generator and checker. The user-space tool hssi exports a control interface to the HE-HSSI's AFU's packet generator logic.
The hssi application provides a means of interacting with the 10G and with the 100G HSSI AFUs. In both 10G and 100G operating modes, the application initializes the AFU, completes the desired transfer as described by the mode- specific options, and displays the ethernet statistics by invoking ethtool --statistics INTERFACE.
The following example walks through the process of binding the VF corresponding with the HE-HSSI exerciser to vfio-pci, sending traffic, and verifying that traffic was received.
1. Create 3 VFs in the PR region.
$ sudo pci_device b1:00.0 vf 3
2. Verify all 3 VFs were created.
$ lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
3. Bind all of the PF/VF endpoints to the vfio-pci driver.
$ sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\n$ sudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\n$ sudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\n
4. Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
$ sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
The following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-10-accelerator-pfvf-and-guid-mappings","title":"Table 10: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6000/1-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb 5. Check Ethernet PHY settings with fpgainfo.
$ sudo fpgainfo phy -B 0xb1 IIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\n//****** HSSI information ******//\nHSSI version : 1.0\nNumber of ports : 8\nPort0 :25GbE DOWN\nPort1 :25GbE DOWN\nPort2 :25GbE DOWN\nPort3 :25GbE DOWN\nPort4 :25GbE DOWN\nPort5 :25GbE DOWN\nPort6 :25GbE DOWN\nPort7 :25GbE DOWN\n
6. Set loopback mode.
$ sudo hssiloopback --loopback enable --pcie-address 0000:b1:00.0 args Namespace(loopback='enable', pcie_address='0000:b1:00.0', port=0)\nsbdf: 0000:b1:00.0\nFPGA dev: {'segment': 0, 'bus': 177, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:b1:00.0'}\nargs.hssi_grps{0: ['dfl_dev.6', ['/sys/bus/pci/devices/0000:b1:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.6/uio/uio0']]}\nfpga uio dev:dfl_dev.6\n\n--------HSSI INFO START-------\nDFH :0x3000000010002015\nHSSI ID :0x15\nDFHv :0.5\nguidl :0x99a078ad18418b9d\nguidh :0x4118a7cbd9db4a9b\nHSSI version :1.0\nFirmware Version :1\nHSSI num ports :8\nPort0 :25GbE\nPort1 :25GbE\nPort2 :25GbE\nPort3 :25GbE\nPort4 :25GbE\nPort5 :25GbE\nPort6 :25GbE\nPort7 :25GbE\n--------HSSI INFO END-------\n\nhssi loopback enabled to port0\n
7. Send traffic through the 10G AFU.
$ sudo hssi --pci-address b1:00.6 hssi_10g --num-packets 100 10G loopback test\nport: 0\neth_loopback: on\n he_loopback: none\n num_packets: 100\npacket_length: 64\nsrc_address: 11:22:33:44:55:66\n (bits): 0x665544332211\n dest_address: 77:88:99:aa:bb:cc\n (bits): 0xccbbaa998877\n random_length: fixed\n random_payload: incremental\n rnd_seed0: 5eed0000\n rnd_seed1: 5eed0001\n rnd_seed2: 25eed\n eth:\n\nNo eth interface, so not honoring --eth-loopback.\n0x40000 ETH_AFU_DFH: 0x1000010000001000\n0x40008 ETH_AFU_ID_L: 0xbb370242ac130002\n0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea\n0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000\n0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000\n0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000\n0x40048 AFU_SCRATCHPAD: 0x0000000045324511\n\n0x3c00 number_packets: 0x00000064\n0x3c01 random_length: 0x00000000\n0x3c02 random_payload: 0x00000000\n0x3c03 start: 0x00000000\n0x3c04 stop: 0x00000000\n0x3c05 source_addr0: 0x44332211\n0x3c06 source_addr1: 0x00006655\n0x3c07 dest_addr0: 0xaa998877\n0x3c08 dest_addr1: 0x0000ccbb\n0x3c09 packet_tx_count: 0x00000064\n0x3c0a rnd_seed0: 0x5eed0000\n0x3c0b rnd_seed1: 0x5eed0001\n0x3c0c rnd_seed2: 0x00025eed\n0x3c0d pkt_length: 0x00000040\n0x3cf4 tx_end_tstamp: 0x000003d2\n0x3d00 num_pkt: 0xffffffff\n0x3d01 pkt_good: 0x00000064\n0x3d02 pkt_bad: 0x00000000\n0x3d07 avst_rx_err: 0x00000000\n0x3d0b rx_sta_tstamp: 0x00000103\n0x3d0c rx_end_tstamp: 0x0000053b\n0x3e00 mac_loop: 0x00000000\n\nHSSI performance:\n Selected clock frequency : 402.832 MHz\n Latency minimum : 642.948 ns\n Latency maximum : 896.155 ns\n Achieved Tx throughput : 18.4528 GB/s\n Achieved Rx throughput : 16.7101 GB/s\n\nNo eth interface, so not showing stats.\n
The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. The hssi_loopback utility tests both external and internal loopbacks.
The hssistats tool provides the MAC statistics.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#50-upgrading-the-intel-fpga-smartnic-n60001-pl-with-20242-1-version-of-the-bmc-and-fim","title":"5.0 Upgrading the Intel\u00ae FPGA SmartNIC N6000/1-PL with 2024.2-1 Version of the BMC and FIM","text":"If your Intel\u00ae FPGA SmartNIC N6000/1-PL does not have the 2024.2-1 version of the FIM and BMC, use this section to begin your upgrade process. The upgrade process depends on both the OPAE SDK and kernel drivers, which were installed in the Software Installation Guide: PCIe Attach. Use the output of fpgainfo and compare against the table below to determine if an upgade is necessary.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-11-fim-version-summary-for-intel-ofs-20242-1-release","title":"Table 11: FIM Version Summary for Intel OFS 2024.2-1 Release","text":"FIM Version Bitstream ID Pr Interface ID File Name Download Location 1 360571656856467345 a791757d-38a6-5159-a7fc-e1a61157a07b ofs_top_page[1 / 2]_unsigned_user[1 / 2].bin ofs-2024.2-1 Release Page"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#table-12-bmc-version-summary-for-intel-ofs-20242-1-release","title":"Table 12: BMC Version Summary for Intel OFS 2024.2-1 Release","text":"BMC FW and RTL Version File Name Download Location 3.15.2 AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu n/a Sample output of fpgainfo with matching values:
Intel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: 3.15.2\nBoard Management Controller Build version: 3.15.2\n//****** FME ******//\nObject Id : 0xED00001\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 360571656856467345\nBitstream Version : 5.0.1\nPr Interface Id : a791757d-38a6-5159-a7fc-e1a61157a07b\nBoot Page : user1\nFactory Image Info : a2b5fd0e7afca4ee6d7048f926e75ac2\nUser1 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\nUser2 Image Info : a791757d-38a6-5159-a7fc-e1a61157a07b\n
- If your output does not match the table above, download the appropriate FIM image from the Intel OFS 2024.2-1 (Intel Agilex) release page. Once downloaded transfer the file over to the server and use the fpgasupdate utility to perform an upgrade of the BMC.
$ sudo fpgasupdate AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu\n[2022-04-14 16:32:47.93] [WARNING ] Update starting. Please do not interrupt. [2022-04-14 16:32:47.93] [INFO ] updating from file /home/user/AC_BMC_RSU_user_retail_3.15.2_unsigned.rsu with size 904064 [2022-04-14 16:32:47.94] [INFO ] waiting for idle [2022-04-14 16:32:47.94] [INFO ] preparing image file [2022-04-14 16:33:26.98] [INFO ] writing image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [904064/904064 bytes][Elapsed Time: 0:00:00.00] [2022-04-14 16:33:26.98] [INFO ] programming image file (100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [Elapsed Time: 0:00:26.02] [2022-04-14 16:33:53.01] [INFO ] update of 0000:b1:00.0 complete [2022-04-14 16:33:53.01] [INFO ] Secure update OK [2022-04-14 16:33:53.01] [INFO ] Total time: 0:01:05.07\nsudo rsu bmcimg\n
2. Load the new FIM image. ```bash\n$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin <PCI ADDRESS>\n[2022-04-14 16:42:31.58] [WARNING ] Update starting. Please do not interrupt. \n[2022-04-14 16:42:31.58] [INFO ] updating from file ofs_top_page1_pacsign_user1.bin with size 19928064 \n[2022-04-14 16:42:31.60] [INFO ] waiting for idle \n[2022-04-14 16:42:31.60] [INFO ] preparing image file \n[2022-04-14 16:42:38.61] [INFO ] writing image file \n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588] [19928064/19928064 bytes][Elapsed Time: 0:00:16.01] \n[2022-04-14 16:42:54.63] [INFO ] programming image file \n(100%) [\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588][Elapsed Time: 0:06:16.40] \n[2022-04-14 16:49:11.03] [INFO ] update of 0000:b1:00.0 complete \n[2022-04-14 16:49:11.03] [INFO ] Secure update OK \n[2022-04-14 16:49:11.03] [INFO ] Total time: 0:06:39.45\nsudo rsu fpga --page=user1 <PCI ADDRESS>\n```\n
- Verify output of fpgainfo matches the table above.
"},{"location":"hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/adp_board_installation_guidelines/","title":"Board Installation Guidelines: Intel\u00ae FPGA SmartNIC N6000/1-PL, Intel\u00ae FPGA PAC D5005","text":"Last updated: November 19, 2024
"},{"location":"n6001/adp_board_installation_guidelines/#10-introduction","title":"1.0 Introduction","text":""},{"location":"n6001/adp_board_installation_guidelines/#11-about-this-document","title":"1.1 About This Document","text":"The purpose of this document is to educate users of the following acceleration platforms on board installation and server environment setup - the Intel\u00ae FPGA SmartNIC N6000/1-PL and the Intel\u00ae FPGA PAC D5005. After reading the document a user shall be able to:
- Safely install and remove an ADP
- Set up their server BIOS with the recommended settings
- Learn about thermal cooling requirements for their platform
This document will include notes where board installation or information differs between platforms. If no such note exists, you may assume it is applicable to all three platforms.
"},{"location":"n6001/adp_board_installation_guidelines/#12-audience","title":"1.2 Audience","text":"The information in this document is intended for customers evaluating the PCIe Attach shell on supported ADP platforms. This reference is a starting point for evaluation and development of the OFS compliant shell designs and workloads. This document will not cover software installation or platform validation, as those are shown in their respective documents.
Note: Code command blocks are used throughout the document. Full command output may not be shown for the sake of brevity.
"},{"location":"n6001/adp_board_installation_guidelines/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"n6001/adp_board_installation_guidelines/#13-required-hardware-for-installation","title":"1.3 Required Hardware for Installation","text":""},{"location":"n6001/adp_board_installation_guidelines/#table-2-intel-n60001-pl-fpga-smartnic-platform-sku-mapping","title":"Table 2: Intel N6000/1-PL FPGA SmartNIC Platform SKU Mapping","text":"The following table highlights the differences between N6000/1 PL FPGA SmartNIC platforms (SKU1/SKU2). Use this table to identify which version of the N6000/1-PL FPGA SmartNIC platforms you have if you are unsure. The board identification printed by the fpgainfo fme commands depends on both the OPAE SDK and Linux DFL drivers from sections, whose installation is covered in the [Software Installation Guide: OFS for PCIe Attach FPGAs].
SKU Mapping SKU Value Primary Difference fpgainfo Identification N6000 Q1613314XXXXX PCIe Gen 4 1x16 mechanical bifurcated 2x8 logical to host, with one PCIe Gen 4x8 endpoint reserved for Intel E810-C-CAM2 NIC, the other reserved for FIM \"Intel Acceleration Development Platform N6000\" N6001 Q0216514XXXXX PCIe Gen 4 1x16 mechanical and logical connection between host and FIM \"Intel Acceleration Development Platform N6001\" The following table provides a picture reference for the hardware components discussed in the rest of the document.
"},{"location":"n6001/adp_board_installation_guidelines/#table-3-hardware-bkc","title":"Table 3: Hardware BKC","text":"Component Image Intel\u00ae FPGA SmartNIC N6001-PL (SKU2) Supermicro Server SYS-220HE Intel FPGA Download Cable II (Only Required for manual flashing) 2x5 Extension header - Samtech Part No: ESQ-105-13-L-D (Only Required for manual flashing) In addition to the above, all OFS ADP platforms require an auxillary power cable for the 12 V-Auxiliary 2x4 PCIe* power connector. This cable will differ between server vendors - review the pinout of the power connector on the Intel\u00ae FPGA Programmable Acceleration Card D5005 Data Sheet or Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) as a reference for ordering. Although this is not always the case, often the standard 2x4 PCIe power connector that is required to enable a GPU in your server will also work for an FPGA-based ADP.
"},{"location":"n6001/adp_board_installation_guidelines/#20-initial-server-setup","title":"2.0 Initial Server Setup","text":""},{"location":"n6001/adp_board_installation_guidelines/#21-server-information-for-intel-fpga-smartnic-n60001-pl","title":"2.1 Server Information for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"Both the server BIOS and BMC need to match the versions listed below in Table 4: Supermicro Server BMC BKC. These updates only apply for this specific Best Known Configuration (BKC) - other server manufacturers may require different BIOS updates. Please consult your server's user guide and release notes for update information.
"},{"location":"n6001/adp_board_installation_guidelines/#table-4-supermicro-server-bmc-bkc","title":"Table 4: SuperMicro Server BMC BKC","text":"Component Version BIOS Version American Megatrends International, LLC(1.4) Information about the server\u2019s currently loaded firmware can be found on the BMC web portal dashboard. Accessing this page requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d. During boot the BMC\u2019s login IP will be presented on the screen.
Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case. It is recommended the user change their BMC\u2019s default username as soon as they are able.
After logging in you should be able to review information about the BMC and BIOS by referring to the System box, visible upon initial loading of the page. Double check that the values match those in Table 4. If they do not, you may download the appropriate versions from the SuperMicro product page by selecting the BIOS option and downloading the most recent \u201cBundled Software File Name\u201d. Follow the BMC and BIOS update instructions included in the SuperMicro manuals page in the document X12/H12 BMC Manual in Appendix A.2 Updating Firmware Using BMC Web GUI.
If using a different server model, refer to that server\u2019s user guide for instructions on remote system management. Ensure that any system you end up using meets all the following requirements:
- Main Board: PCI Express 3.0 (D5005) or 4.0 (N6000/1) compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
"},{"location":"n6001/adp_board_installation_guidelines/#22-server-information-for-intel-fpga-pac-d5005","title":"2.2 Server Information for Intel\u00ae FPGA PAC D5005","text":"Refer to sections 2.1-2.3 of the Intel Acceleration Stack Quick Start Guide: Intel FPGA Programmable Acceleration Card D5005 for a complete overview of the physical installation process and ESD precautions for the D5005 platform.
Ensure that the system meets all the following requirements before proceeding to install the Intel\u00ae FPGA PAC D5005 into a server.
- Main Board: PCI Express 3.0 compliant motherboard with at least one dual-width x16 PCIe slot available for card installation
- Board Power Supply: Auxiliary Power (12V)
Detailed mechanical for information can be found on the D5005 Data Sheet and in section 4.0 Mechanical Information of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837).
"},{"location":"n6001/adp_board_installation_guidelines/#30-server-settings","title":"3.0 Server Settings","text":""},{"location":"n6001/adp_board_installation_guidelines/#31-bios-settings","title":"3.1 BIOS Settings","text":"You must enable Intel VT-x/VT-d technologies for the PCIe slot housing your ADP. The following steps are known to work on a SuperMicro SYS-220HE server platform.
-
To enter the Supermicro server\u2019s BIOS setup page, reboot, and press \\<Delete> when prompted. You can browse the tabs / options with a combination of arrow keys along with \\<Escape> and \\<Enter>.
-
Navigate right to the Advanced tab, then select the following menu options: Chipset Configuration -> North Bridge -> IIO Configuration -> Intel VT for Directed I/O (VT-d).
-
If not already, enable the option Intel VT for Directed I/O (VT-d).
"},{"location":"n6001/adp_board_installation_guidelines/#31-server-fan-speed","title":"3.1 Server Fan Speed","text":"The recommended fan speed setting is to use the 100% preset. If using a different server model, refer to that server\u2019s user guide for instructions on changing fan speed. The following steps will help users on the SuperMicro platform.
- Log in to the SuperMicro server BMC. (This requires an Ethernet cable to be attached to an open port on the server labelled \u201cIPMI\u201d.)
- During boot the BMC\u2019s login IP will be presented on the screen. Open this IP address in a browser and enter your login credentials. The default username is ADMIN, and the default password has been printed on the service tag that pulls out from the front of the server case.
- On the left menu select System -> Component Info, select the Fan tab, under Advanced Settings click the circle next to Full Speed.
"},{"location":"n6001/adp_board_installation_guidelines/#32-cooling-requirements","title":"3.2 Cooling Requirements","text":"Please refer to sections 8.1 and 8.2 of the Intel FPGA Programmable Acceleration Card D5005 Data Sheet or section 6.0 of the Intel FPGA SmartNIC N6001-PL Data Sheet - SKU2 (content ID=723837) for guidance on cooling specifications that must be met when using these platforms. Failure to adhere to these guidelines may result in thermal runaway and/or performance degradation.
"},{"location":"n6001/adp_board_installation_guidelines/#40-board-installation-procedure","title":"4.0 Board Installation Procedure","text":""},{"location":"n6001/adp_board_installation_guidelines/#41-pcie-slot-mappings-for-intel-fpga-smartnic-n60001-pl","title":"4.1 PCIe Slot Mappings for Intel\u00ae FPGA SmartNIC N6000/1-PL","text":"The Intel N6000/1-PL FPGA SmartNIC Platforms are officially verified in the upper middle PCIe x16 slot (Slot 3). If using a different slot, refer to the information in Table 5 PCIe Slot Mapping for which port settings to change in server BIOS.
"},{"location":"n6001/adp_board_installation_guidelines/#table-5-pcie-slot-mapping","title":"Table 5: PCIe Slot Mapping","text":"CPU Number Port Number (in BIOS) PCIe Slot CPU1 Port 2 5 and 6 CPU1 Port 4 7 and 8 CPU2 Port 2 1 and 2 CPU2 Port 4 3 and 4"},{"location":"n6001/adp_board_installation_guidelines/#42-installation-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.2 Installation Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe installation of an ADP platform into a supported server. While an Intel\u00ae FPGA SmartNIC N6001-PL is shown in the images below, this procedure applies to all three platforms.
- Position the board over the selected connector on the motherboard.
- Press down gently and firmly to seat the card in the PCIe slot, and then secure the bracket to the system chassis with the retention screw.
"},{"location":"n6001/adp_board_installation_guidelines/#table-6-adp-installation-procedure","title":"Table 6: ADP Installation Procedure","text":"Callout Description 1 Retention screw 2 Press down here gently 3 Press down here gently 4 Motherboard Do not bend the card while inserting into a slot. Do not apply much pressure in regions 2 or 3 while inserting.
"},{"location":"n6001/adp_board_installation_guidelines/#43-removal-procedure-for-the-intel-fpga-pac-d5005-and-intel-fpga-smartnic-n60001-pl-into-a-server","title":"4.3 Removal Procedure for The Intel\u00ae FPGA PAC D5005 and Intel\u00ae FPGA SmartNIC N6000/1-PL into a Server","text":"The following instructions will help to ensure safe removal of the platforms from a supported server.
- Disconnect all power cords from the server power supply(s).
- Remove the retention bracket screw.
- Carefully lift the card out of the PCIe slot.
"},{"location":"n6001/adp_board_installation_guidelines/#table-7-adp-removal-procedure","title":"Table 7: ADP Removal Procedure","text":"Callout Description 1 Retention screw 2 Pull up here gently 3 Motherboard Do not bend the card while removing it from the slot.
"},{"location":"n6001/sw_install_pcie_attach/","title":"Software Installation Guide: Open FPGA Stack for PCIe Attach","text":"Last updated: November 19, 2024
"},{"location":"n6001/sw_install_pcie_attach/#10-about-this-document","title":"1.0 About This Document","text":"The purpose of this document is to help users get started in setting up their local environments and installing the most recent release of the PCIe Attach software stack on the host. This document will not cover the process of board installation or platform bring-up. After reviewing this document, a user shall be able to:
- Set up a server environment according to the Best Known Configuration (BKC)
- Build and install the OPAE Software Development Kit (SDK) on the host
- Build and install the Linux DFL driver stack on the host
"},{"location":"n6001/sw_install_pcie_attach/#11-audience","title":"1.1 Audience","text":"The information in this document is intended for customers evaluating a PCIe Attach shell. The PCIe Attach shell design is supported on a number of board offerings, including the Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile), Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile), Intel\u00ae FPGA SmartNIC N6000/1-PL, and Intel\u00ae FPGA PAC D5005.
Note: Code command blocks are used throughout the document. Comments are preceded with '#'. Full command output may not be shown for the sake of brevity.
"},{"location":"n6001/sw_install_pcie_attach/#table-1-terminology","title":"Table 1: Terminology","text":"Term Abbreviation Description Advanced\u00a0Error\u00a0Reporting AER The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting. (link) Accelerator\u00a0Functional\u00a0Unit AFU Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.\u00a0Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region. Basic Building Block BBB Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID. Best\u00a0Known\u00a0Configuration BKC The software and hardware configuration Intel uses to verify the solution. Board\u00a0Management\u00a0Controller BMC Supports features such as board power managment, flash management, configuration management, and board telemetry monitoring and protection. The majority of the BMC logic is in a separate component, such as an Intel\u00ae Max\u00ae 10 or Intel Cyclone\u00ae 10 device; a small portion of the BMC known as the PMCI resides in the main Agilex FPGA. Configuration and\u00a0Status\u00a0Register CSR The generic name for a register space which is accessed in order to interface with the module it resides in (e.g. AFU, BMC, various sub-systems and modules). Data Parallel C++ DPC++ DPC++ is Intel\u2019s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware. Device\u00a0Feature\u00a0List DFL The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.\u00a0This concept is the foundation for the OFS software framework. (link) FPGA\u00a0Interface\u00a0Manager FIM Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring. FPGA\u00a0Management\u00a0Engine FME Performs reconfiguration and other FPGA management functions. Each FPGA device only has one FME which is accessed through PF0. Host\u00a0Exerciser\u00a0Module HEM Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc. Input/Output Control IOCTL System calls used to manipulate underlying device parameters of special files. Intel\u00a0Virtualization\u00a0Technology for\u00a0Directed I/O Intel VT-d Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization. Joint Test Action Group JTAG Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology. Memory\u00a0Mapped\u00a0Input/Output MMIO The memory space users may map and access both control registers and system memory buffers with accelerators. oneAPI Accelerator Support Package oneAPI-asp A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and OFS shell components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS. Open\u00a0FPGA\u00a0Stack OFS OFS is a software and hardware infrastructure providing an efficient approach to develop a custom FPGA-based platform or workload using an Intel, 3rd party, or custom board. Open\u00a0Programmable\u00a0Acceleration\u00a0Engine Software Development Kit OPAE SDK The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space. Platform\u00a0Interface\u00a0Manager PIM An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols. Platform Management Controller Interface PMCI The portion of the BMC that resides in the Agilex FPGA and allows the FPGA to communicate with the primary BMC component on the board. Partial Reconfiguration PR The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. For OFS designs, the PR region is referred to as the pr_slot. Port N/A When used in the context of the fpgainfo port command it represents the interfaces between the static FPGA fabric and the PR region containing the AFU. Remote System Update RSU The process by which the host can remotely update images stored in flash through PCIe. This is done with the OPAE software command \"fpgasupdate\". Secure Device Manager SDM The SDM is the point of entry to the FPGA for JTAG commands and interfaces, as well as for device configuration data (from flash, SD card, or through PCI Express* hard IP). Static Region SR The portion of the FPGA design that cannot be dynamically reconfigured during run-time. Single-Root\u00a0Input-Output\u00a0Virtualization SR-IOV Allows the isolation of PCI Express resources for manageability and performance. SYCL SYCL SYCL (pronounced \"sickle\") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics. Test Bench TB Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output. Universal Verification Methodology UVM A modular, reusable, and scalable testbench structure via an API framework. In the context of OFS, the UVM enviroment provides a system level simulation environment for your design. Virtual\u00a0Function\u00a0Input/Output VFIO An Input-Output Memory Management Unit (IOMMU)/device agnostic framework for exposing direct device access to userspace.\u00a0(link)"},{"location":"n6001/sw_install_pcie_attach/#table-2-software-and-component-version-summary-for-ofs-pcie-attach","title":"Table 2: Software and Component Version Summary for OFS PCIe Attach","text":"The OFS PCIe Attach release is built upon tightly coupled software and Operating System version(s). The repositories listed below are where the source code resides for each of the components discussed in this document.
Component Version Download Link Host Operating System RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10 link OPAE SDK 2.13.0-3 2.13.0-3 Linux DFL intel-1.11.0-2 intel-1.11.0-2"},{"location":"n6001/sw_install_pcie_attach/#table-3-release-pages-for-each-pcie-attach-platform","title":"Table 3: Release Page(s) for each PCIe Attach Platform","text":"This is a comprehensive list of the platform(s) whose software build and installation steps are covered in this document.
Platform Release Page Link Stratix\u00ae 10 FPGA https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Intel\u00ae FPGA SmartNIC N6001-PL https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA F-Series Development Kit (2x F-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 Agilex\u00ae 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1"},{"location":"n6001/sw_install_pcie_attach/#12-server-requirements","title":"1.2 Server Requirements","text":""},{"location":"n6001/sw_install_pcie_attach/#121-host-bios","title":"1.2.1 Host BIOS","text":"These are the host BIOS settings required to work with the OFS stack, which relies on SR-IOV for some of its functionality. Information about any given server's currently loaded firmware and BIOS settings can be found through its remote access controller, or by manually entering the BIOS by hitting a specific key during power on. Your specific server platform will include instructions on proper BIOS configuration and should be followed when altering settings. Ensure the following has been set:
- Intel VT for Directed I/O (VT-d) must be enabled
Specific BIOS paths are not listed here as they can differ between BIOS vendors and versions.
"},{"location":"n6001/sw_install_pcie_attach/#122-host-server-kernel-and-grub-configuration","title":"1.2.2 Host Server Kernel and GRUB Configuration","text":"While many host Linux kernel and OS distributions may work with this design, only the following configuration(s) have been tested. You will need to download and install the OS on your host of choice; we will build the required kernel alongside the Linux DFL driver set.
- OS: RedHat\u00ae Enterprise Linux\u00ae (RHEL) 8.10
- Kernel: 4.18.0-dfl
"},{"location":"n6001/sw_install_pcie_attach/#20-ofs-software-overview","title":"2.0 OFS Software Overview","text":"The responsibility of the OFS kernel drivers is to act as the lowest software layer in the FPGA software stack, providing a minimalist driver implementation between the host software and functionality that has been implemented on the development platform. This leaves the implementation of IP-specific software in user-land, not the kernel. The OFS software stack also provides a mechanism for interface and feature discovery of FPGA platforms.
The OPAE SDK is a software framework for managing and accessing programmable accelerators (FPGAs). It consists of a collection of libraries and tools to facilitate the development of software applications and accelerators. The OPAE SDK resides exclusively in user-space, and can be found on the OPAE SDK Github.
The OFS drivers decompose implemented functionality, including external FIM features such as HSSI, EMIF and SPI, into sets of individual Device Features. Each Device Feature has its associated Device Feature Header (DFH), which enables a uniform discovery mechanism by software. A set of Device Features are exposed through the host interface in a Device Feature List (DFL). The OFS drivers discover and \"walk\" the Device Features in a Device Feature List and associate each Device Feature with its matching kernel driver.
In this way the OFS software provides a clean and extensible framework for the creation and integration of additional functionalities and their features.
Note: A deeper dive on available SW APIs and programming model is available in the Software Reference Manual: Open FPGA Stack, on kernel.org, and through the Linux DFL wiki pages.
"},{"location":"n6001/sw_install_pcie_attach/#30-ofs-dfl-kernel-drivers","title":"3.0 OFS DFL Kernel Drivers","text":"OFS DFL driver software provides the bottom-most API to FPGA platforms. Libraries such as OPAE and frameworks like DPDK are consumers of the APIs provided by OFS. Applications may be built on top of these frameworks and libraries. The OFS software does not cover any out-of-band management interfaces. OFS driver software is designed to be extendable, flexible, and provide for bare-metal and virtualized functionality. An in depth look at the various aspects of the driver architecture such as the API, an explanation of the DFL framework, and instructions on how to port DFL driver patches to other kernel distributions can be found on https://github.com/OPAE/linux-dfl/wiki.
An in-depth review of the Linux device driver architecture can be found on opae.github.io.
The DFL driver suite can be automatically installed using a supplied Python 3 installation script. This script ships with a README detailing execution instructions, and currently only supported the PCIe Attach release. Its usage is detailed in the relevant Quick Start Demonstration Guideline for your platform and will not be covered here.
"},{"location":"n6001/sw_install_pcie_attach/#31-ofs-dfl-backport-kernel-driver-installation-environment-setup","title":"3.1 OFS DFL Backport Kernel Driver Installation Environment Setup","text":"All OFS DFL kernel driver primary release code for this release resides in the Linux DFL Backport GitHub repository. This repository is open source and does not require any special permissions to access. It includes a snapshot of the Linux kernel with most of the OFS DFL drivers included in /drivers/fpga/*. Download, configuration, and compilation will be discussed in this section. Refer back to section 1.2.2 Host Server Kernel and GRUB Configuration for a list of supported Operating System(s).
You can choose to install the DFL kernel drivers by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your platform's release page, skip to section 3.3 Installing the OFS DFL Kernel Drivers from Pre-Built Packages. Regardless of your choice you will need to follow the two steps in this section to prepare your server environment for installation.
This installation process assumes the user has access to an internet connection to clone specific GitHub repositories, and to satisfy package dependencies.
-
It is recommended you lock your Red Hat release version to 8.10 to prevent accidental upgrades. Update installed system packages to their latest versions. We need to enable the code-ready-builder and EPEL repositories.
subscription-manager release --set=8.10\nsudo dnf update\nsubscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\nsudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n
-
Install the following package dependencies if building and installing drivers from source. If you do not require the use of a proxy to pull in downloads using dnf, you can safely remove those parameters from the following commands:
If you require the use of a proxy, add it to DNF using by editing the following file\nsudo nano /etc/dnf/dnf.conf\n# Include your proxy by adding the following line, replacing the URL with your proxy's URL\n# proxy=http://proxy.server.com:port\nsudo dnf install python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel numactl-devel bison flex automake autoconf libtools\n\npython3 -m pip install --user jsonschema virtualenv pudb pyyaml setuptools pybind11\n\n# If setuptools and pybind11 were already installed\npython3 -m pip install --upgrade --user pybind11 setuptools\n\nsudo dnf install kernel-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-headers-4.18.0-553.5.1.el8_10.x86_64\nsudo dnf install kernel-devel-4.18.0-553.5.1.el8_10.x86_64\n
Now you have the choice to either follow the steps in section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source or 3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages.
"},{"location":"n6001/sw_install_pcie_attach/#32-building-and-installing-the-ofs-dfl-backport-kernel-drivers-from-source","title":"3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
It is recommended you create an empty top level directory for your OFS related repositories to keep the working environment clean. All steps in this installation will use a generic top-level directory at /home/OFS/. If you have created a different top-level directory, replace this path with your custom path.
1. Select the 4.18.0-553.5.1.el8_10 kernel as your next boot target on the build machine, then reboot.
# For multiple boots (until overwritten), use the following\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\n# Or select the kernel you want during boot time\nsudo reboot\n
1. Initialize an empty git repository and clone the DFL driver source code:
```bash\nmkdir /home/OFS/\ncd /home/OFS/\ngit init\ngit clone https://github.com/OFS/linux-dfl-backport\ncd /home/OFS/linux-dfl-backport\ngit checkout tags/intel-1.11.0-2\n```\n\n*Note: The linux-dfl repository is roughly 5 GB in size.*\n
2. Verify that the correct tag/branch have been checked out.
```bash\ngit describe --tags\nintel-1.11.0-2\n```\n\n*Note: If two different tagged releases are tied to the same commit, running git describe tags may report the other release's tag. This is why the match is made explicit.*\n
3. Build the kernel.
```bash\ncd /home/OFS/linux-dfl-backport\nmake && make rpm\n```\n
4. Install the newly compiled RPM package and reboot.
```bash\nsudo rpm -i intel-fpga-dfl-dkms-1.11.0-2.2024.07.25.g276007e.noarch.rpm\n\nsudo reboot\n```\n
5. Verify the DFL drivers have been successfully installed by reading version information directly from /lib/modules. Recall that the name of the kernel built as a part of this section is 4.18.0-dfl. If the user set a different name for their kernel, change this path as needed:
```bash\ncd /usr/lib/modules/4.18.0-dfl/kernel/drivers/fpga\nls\ndfl-afu.ko dfl-fme.ko dfl-fme-region.ko dfl.ko dfl-pci.ko fpga-mgr.ko intel-m10-bmc-sec-update.ko\ndfl-fme-br.ko dfl-fme-mgr.ko dfl-hssi.ko dfl-n3000-nios.ko fpga-bridge.ko fpga-region.ko\n```\n
If an OFS device that is compatible with these drivers is installed on the server, you can double check the driver versions by listing the currently loaded kernel modules with lsmod:
```bash\nlsmod | grep dfl\nuio_dfl 20480 0\ndfl_emif 16384 0\nuio 20480 1 uio_dfl\nptp_dfl_tod 16384 0\ndfl_intel_s10_iopll 20480 0\n8250_dfl 20480 0\ndfl_fme_region 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 2\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 11 dfl_pci,uio_dfl,dfl_fme,intel_m10_bmc_pmci,dfl_fme_br,8250_dfl,qsfp_mem,ptp_dfl_tod,dfl_afu,dfl_intel_s10_iopll,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 20480 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n```\n
6. Four kernel parameters must be added to the boot command line for the newly installed kernel. First, open the file grub:
```bash\nsudo vim /etc/default/grub\n```\n
7. In the variable GRUB_CMDLINE_LINUX add the following parameters in bold: GRUB_CMDLINE_LINUX=\"crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200\".
Note: If you wish to instead set hugepages on a per session basis, you can perform the following steps. These settings will be lost on reboot.
```bash\nmkdir -p /mnt/huge \nmount -t hugetlbfs nodev /mnt/huge \necho 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages \necho 2048 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages \n```\n
8. Save your edits, then apply them to the GRUB2 configuration file.
```bash\nsudo grub2-mkconfig\n```\n
9. Warm reboot. Your kernel parameter changes should have taken affect.
```bash\ncat /proc/cmdline\nBOOT_IMAGE=(hd1,gpt2)/vmlinuz-4.18.0-dfl root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap intel_iommu=on pcie=realloc hugepagesz=2M hugepages=200 rhgb quiet\n```\n
A list of all DFL drivers and their purpose is maintained on the DFL Wiki.
"},{"location":"n6001/sw_install_pcie_attach/#33-installing-the-ofs-dfl-backport-kernel-drivers-from-pre-built-packages","title":"3.3 Installing the OFS DFL Backport Kernel Drivers from Pre-Built Packages","text":"This section assumes you have already read through and followed any relevant environment setup steps in Secion 3.1 3.1 OFS DFL Backport Kernel Driver Installation Environment Setup
To use the pre-built Linux DFL packages, you first need to download the files from your chosen platform's release page under the Artifacts tab. The name will resemble kernel-*.tar.gz. For the backport repository you can also choose to download packages under the GitHub action Build dkms packages, although your version will differ from the release version of the software.
- Download and install the pre-built kernel package.
wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.2-1/kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\ntar xf kernel-4.18.0_dfl_2024.06.14_276007e.tar.gz\n\nsudo rpm -i kernel-4.18.0_dfl_2024.06.14_276007e/intel-fpga-dfl-dkms-1.11.0-2.2024.06.14.g276007e.noarch.rpm\n\n# Set this kernel as your new default boot target\nsudo grubby --set-default /boot/vmlinuz-4.18.0-553.5.1.el8_10.x86_64\nsudo reboot\n
Continue from step 5 of Section 3.2 Building and Installing the OFS DFL Backport Kernel Drivers from Source.
"},{"location":"n6001/sw_install_pcie_attach/#40-opae-software-development-kit","title":"4.0 OPAE Software Development Kit","text":"The OPAE SDK software stack sits in user space on top of the OFS kernel drivers. It is a common software infrastructure layer that simplifies and streamlines integration of programmable accelerators such as FPGAs into software applications and environments. OPAE consists of a set of drivers, user-space libraries, and tools to discover, enumerate, share, query, access, manipulate, and reconfigure programmable accelerators. OPAE is designed to support a layered, common programming model across different platforms and devices. To learn more about OPAE, its documentation, code samples, an explanation of the available tools, and an overview of the software architecture, visit opae.github.io.
The OPAE SDK source code is contained within a single GitHub repository hosted at the OPAE Github. This repository is open source and does not require any permissions to access.
You can choose to install the OPAE SDK by either using pre-built binaries created for the BKC OS, or by building them on your local server. If you decide to use the pre-built packages available on your chosen platform's release page, skip to section 4.3 Installing the OPAE SDK with Pre-built Packages. Regardless of your choice you will need to follow the steps in this section to prepare your server for installation.
You may also choose to use the supplied Python 3 installation script. This script ships with a README detailing execution instructions and is available on the PCIe Attach's platform release page. It can be used to automate installation of the pre-built packages, or to build from source.
"},{"location":"n6001/sw_install_pcie_attach/#41-opae-sdk-installation-environment-setup","title":"4.1 OPAE SDK Installation Environment Setup","text":"This installation process assumes you have access to an internet connection to pull specific GitHub repositories, and to satisfy package dependencies.
"},{"location":"n6001/sw_install_pcie_attach/#table-4-opae-package-description","title":"Table 4: OPAE Package Description","text":"Package Name Description opae OPAE SDK is a collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE. It provides a library implementing the OPAE C API for presenting a streamlined and easy-to-use interface for software applications to discover, access, and manage FPGA devices and accelerators using the OPAE software stack. opae-debuginfo This package provides debug information for package opae. Debug information is useful when developing applications that use this package or when debugging this package. opae-debugsource This package provides debug sources for package opae. Debug sources are useful when developing applications that use this package or when debugging this package. opae-devel OPAE headers, tools, sample source, and documentation opae-devel-debuginfo This package provides debug information for package opae-devel. Debug information is useful when developing applications that use this package or when debugging this package. opae-tools This package contains OPAE base tools binaries opae-extra-tools Additional OPAE tools opae-extra-tools-debuginfo This package provides debug information for package opae-extra-tools. Debug information is useful when developing applications that use this package or when debugging this package. -
Remove any currently installed OPAE packages.
sudo dnf remove opae*\n
-
Initialize an empty git repository and clone the tagged OPAE SDK source code.
cd /home/OFS/\ngit init\ngit clone https://github.com/OFS/opae-sdk opae-sdk\ncd /home/OFS/opae-sdk\ngit checkout tags/2.13.0-3\n
-
Verify that the correct tag/branch have been checkout out.
git describe --tags\n2.13.0-3\n
-
Set up a temporary podman container to build OPAE, which will allow you to customize the python installation without affecting system packages.
sudo dnf install podman\ncd /home/OFS\npodman pull registry.access.redhat.com/ubi8:8.6\npodman run -ti -v \"$PWD\":/src:Z -w /src registry.access.redhat.com/ubi8:8.6\n\n# Everything after runs within container:\n# Enable EPEL\ndnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n\ndnf install --enablerepo=codeready-builder-for-rhel-8-x86_64-rpms -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel doxygen python3-sphinx pandoc rpm-build rpmdevtools python3-virtualenv yaml-cpp-devel libudev-devel libcap-devel make automake autoconf libtools\n\npip3 install --upgrade --prefix=/usr pip setuptools pybind11\n\n./opae-sdk/packaging/opae/rpm/create unrestricted\n\nexit\n
The following packages will be built in the same directory as create:
-
Install the packages you just created.
cd /home/OFS/opae-sdk/packaging/opae/rpm\nrm -rf opae-*.src.rpm sudo dnf localinstall -y opae*.rpm\n
-
Check that all packages have been installed and match expectation:
rpm -qa | grep opae\nopae-2.13.0-3.el8.x86_64.rpm\nopae-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-debugsource-2.13.0-3.el8.x86_64.rpm\nopae-devel-2.13.0-3.el8.x86_64.rpm\nopae-devel-debuginfo-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-2.13.0-3.el8.x86_64.rpm\nopae-extra-tools-debuginfo-2.13.0-3.el8.x86_64.rpm\n
"},{"location":"n6001/sw_install_pcie_attach/#42-installing-the-opae-sdk-with-pre-built-packages","title":"4.2 Installing the OPAE SDK with Pre-Built Packages","text":"You can skip the entire build process and use a set of pre-built binaries supplied by Intel. Visit your chosen platform's release page. Ender the Assets tab you will see a file named opae-2.13.0-3.x86_64-\\<\\<date>>_\\<\\<build>>.tar.gz. Download this package and extract its contents:
tar xf opae-2.13.0-3.x86_64-*.tar.gz\n
For a fast installation you can delete the source RPM as it isn't necessary, and install all remaining OPAE RPMs:
rm opae-*.src.rpm\nsudo dnf localinstall opae*.rpm\n
"},{"location":"n6001/sw_install_pcie_attach/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_dev_afu_host_software/","title":"AFU Host Software Developer Guide","text":"Last updated: November 19, 2024
The host application is used to control the AFU and manage data transfer between the host and the AFU. The host channel provides two interfaces between the host and AFU, MMIO and Host Memory. MMIO is used to read/write the CSR interface of the AFU, and the Host Memory interface is used to share data between the AFU and Host user space.
"},{"location":"n6001/ug_dev_afu_host_software/#1-host-application-flow","title":"1. Host Application Flow","text":"The OPAE SDK provides a library with routines to setup and manage the AFU. The basic host application flow is as follows:
When creating the host application, the following OPAE Header Files are required: - opae/fpga.h - For the OPAE C API library - afu_json_info.h - For AFU information including UUID
// Headers needed for example code\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <assert.h>\n// For uuid_parse() to convert UUID string into binary\n#include <uuid/uuid.h>\n// OPAE C API\n#include <opae/fpga.h>\n// State from the AFU's JSON file, extracted using OPAE's afu_json_mgr script\n#include \"afu_json_info.h\"\n
"},{"location":"n6001/ug_dev_afu_host_software/#11-find-and-connect-to-afu","title":"1.1. Find and connect to AFU","text":"Here is an example function which searches for the AFU based on its UUID. If there is a match, it will connect to the AFU. It will also check to see if the AFU is being run in hardware or simulation (ASE).
// Set as global, to allow MMIO routines to access in ASE mode\nstatic fpga_handle s_accel_handle;\n//\n// Search for an accelerator matching the requested UUID and connect to it.\n// Check to see if running in ASE-Simulation mode\n//\nstatic fpga_handle connect_to_accel(const char *accel_uuid, bool *is_ase_sim)\n{\nfpga_properties filter = NULL;\nfpga_guid guid;\nfpga_token accel_token;\nuint32_t num_matches;\nfpga_handle accel_handle;\nfpga_result r;\n// Don't print verbose messages in ASE by default\n setenv(\"ASE_LOG\", \"0\", 0);\n*is_ase_sim = NULL;\n// Set up a filter that will search for an accelerator\n fpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\n// Convert UUID string into binary\n uuid_parse(accel_uuid, guid);\n// Add the desired UUID to the filter\n fpgaPropertiesSetGUID(filter, guid);\n// Do the search across the available FPGA contexts\n num_matches = 1;\nfpgaEnumerate(&filter, 1, &accel_token, 1, &num_matches);\n// Not needed anymore\n fpgaDestroyProperties(&filter);\nif (num_matches < 1)\n{\nfprintf(stderr, \"Accelerator %s not found!\\n\", accel_uuid);\nreturn 0;\n}\n// Acquire ownership of accelerator\n r = fpgaOpen(accel_token, &accel_handle, 0);\nassert(FPGA_OK == r);\n// While the token is available, check whether it is for HW or for ASE simulation.\n fpga_properties accel_props;\nuint16_t vendor_id, dev_id;\nfpgaGetProperties(accel_token, &accel_props);\nfpgaPropertiesGetVendorID(accel_props, &vendor_id);\nfpgaPropertiesGetDeviceID(accel_props, &dev_id);\n*is_ase_sim = (vendor_id == 0x8086) && (dev_id == 0xa5e);\n// Done with token\n fpgaDestroyToken(&accel_token);\nreturn accel_handle;\n}\n
In main(), the function is called updating the accel_handle and ASE status. AFU_ACCEL_UUID is provided by afu_json_info.h created for the Accelerator Descriptor File:
bool is_ase_sim;\n// Find and connect to the accelerator(s)\ns_accel_handle = connect_to_accel(AFU_ACCEL_UUID, &is_ase_sim);\nif (NULL == s_accel_handle) return 0;\n
"},{"location":"n6001/ug_dev_afu_host_software/#12-map-mmio-optional","title":"1.2. Map MMIO (optional)","text":"Mapping the MMIO provides higher performance on the MMIO access versus the standard OPAE MMIO functions. fpgaMapMMIO() is used to return a pointer to the specified MMIO space of the target AFU in process virtual memory. When running in ASE mode, MMIO mapping isn't supported and the MMIO pointer is set to NULL.
static volatile uint64_t *s_mmio_buf;\nfpga_result r;\nif (is_ase_sim)\n{\nprintf(\"Running in ASE Mode\");\ns_mmio_buf = NULL;\n}\nelse\n{\nuint64_t *tmp_ptr;\nr = fpgaMapMMIO(s_accel_handle, 0, &tmp_ptr);\nassert(FPGA_OK == r);\ns_mmio_buf = tmp_ptr;\n}\n
The below example functions provide MMIO Reads/Writes. When running in hardware the functions will use s_mmio_buf for accessing. When running in ASE mode, indicated by s_mmio_buf being set to NULL, fpgaReadMMIO64() fpgaWriteMMIO64() will be used.
//\n// Read a 64 bit CSR. When a pointer to CSR buffer is available, read directly.\n// Direct reads can be significantly faster.\n// If s_mmio_buf is NULL, in ASE mode and need to use OPAE MMIO functions.\n//\nstatic inline uint64_t readMMIO64(uint32_t idx)\n{\nif (s_mmio_buf)\n{\nreturn s_mmio_buf[idx];\n}\nelse\n{\nfpga_result r;\nuint64_t v;\nr = fpgaReadMMIO64(s_accel_handle, 0, 8 * idx, &v);\nassert(FPGA_OK == r);\nreturn v;\n}\n}\n//\n// Write a 64 bit CSR. When a pointer to CSR buffer is available, write directly.\n//\nstatic inline void writeMMIO64(uint32_t idx, uint64_t v)\n{\nif (s_mmio_buf)\n{\ns_mmio_buf[idx] = v;\n}\nelse\n{\nfpgaWriteMMIO64(s_accel_handle, 0, 8 * idx, v);\n}\n}\n
"},{"location":"n6001/ug_dev_afu_host_software/#13-allocate-shared-memory-buffers","title":"1.3. Allocate Shared Memory Buffers","text":"The below example function creates the shared buffers and provides the physical address for AFU access.
//\n// Allocate a buffer in I/O memory, shared with the FPGA.\n//\nstatic volatile void* alloc_buffer(fpga_handle accel_handle,\n ssize_t size,\n uint64_t *wsid,\n uint64_t *io_addr)\n{\nfpga_result r;\nvolatile void* buf;\nr = fpgaPrepareBuffer(accel_handle, size, (void*)&buf, wsid, 0);\nif (FPGA_OK != r) return NULL;\n// Get the physical address of the buffer in the accelerator\n r = fpgaGetIOAddress(accel_handle, *wsid, io_addr);\nassert(FPGA_OK == r);\nreturn buf;\n}\n
In main(), define the buffers and use the above function to allocate the shared buffers. OPAE supports multiple buffers, and the number of buffers is design dependent. Buffers over 4KB require hugepage support on the host. The buffer address needs to be passed to the AFU over MMIO, for the AFU to correctly access the buffer.
#define BUF_SIZE_IN_BYTES 16384\nvolatile unsigned char *src_buf;\nuint64_t src_wsid;\nuint64_t src_pa;\nvolatile unsigned char *dst_buf;\nuint64_t dst_wsid;\nuint64_t dst_pa;\nsrc_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &src_wsid, &src_pa);\nassert(NULL != src_buf);\ndst_buf = alloc_buffer(s_accel_handle, BUF_SIZE_IN_BYTES, &dst_wsid, &dst_pa);\nassert(NULL != dst_buf);\n
"},{"location":"n6001/ug_dev_afu_host_software/#14-perform-acceleration","title":"1.4. Perform Acceleration","text":"The host application interaction is AFU dependent. Generally, the MMIO interface will be used to setup and control the AFU. While the shared buffers are used to pass data between the host and AFU. Below is an example of setting up the AFU, writing the buffer and retrieving the results from the AFU.
// Loading source buffer with walking ones\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nsrc_buf[i] = 1 << (i & 0x7); // walking ones\n}\n// Send AFU buffer addresses and size // register addresses are based on the AFU CSR interface\nwriteMMIO64(8, src_pa);\nwriteMMIO64(9, dst_pa);\nwriteMMIO64(10, buf_size);\n// Start Acceleration\nwriteMMIO64(11, 1);\n// Wait for AFU to complete acceleration\nwhile(!readMMIO64(12))\n;\n// Read destination buffer and print output\nprintf(\"output: \");\nfor(i=0; i < BUF_SIZE_IN_BYTES; i++)\n{\nprintf(\"%d \", dst_buf[i]);\n}\n
"},{"location":"n6001/ug_dev_afu_host_software/#15-cleanup","title":"1.5. Cleanup","text":"When the acceleration is complete, the host application should release the shared buffers and release ownership of the AFU.
// Release shared buffers\n fpgaReleaseBuffer(s_accel_handle, src_wsid);\nfpgaReleaseBuffer(s_accel_handle, dst_wsid); // Release ownership of accelerator\n fpgaClose(s_accel_handle);\n
"},{"location":"n6001/ug_dev_afu_host_software/#2-building-the-host-application","title":"2. Building the Host Application","text":"A Makefile is used to build the host application. Below is an example Makefile from the [examples AFU] repo with the following updated:
- Path to common_include.mk (from examples-afu)
- TEST name
- Source files: SRCS
- Path to .json file (relative to Makefile directory)
Makefile:
# Path to examples-afu/tutorial/afu_types/01_pim_ifc/common/sw/common_include.mk\ninclude ../../common/sw/common_include.mk\n\n# Primary test name\nTEST = example_afu\n\n# Build directory\nOBJDIR = obj\nCFLAGS += -I./$(OBJDIR)\nCPPFLAGS += -I./$(OBJDIR)\n# Files and folders\nSRCS = $(TEST).c\nOBJS = $(addprefix $(OBJDIR)/,$(patsubst %.c,%.o,$(SRCS)))\nall: $(TEST)\n# AFU info from JSON file, including AFU UUID\nAFU_JSON_INFO = $(OBJDIR)/afu_json_info.h\n$(AFU_JSON_INFO): ../hw/rtl/$(TEST).json | objdir\n afu_json_mgr json-info --afu-json=$^ --c-hdr=$@\n$(OBJS): $(AFU_JSON_INFO)\n$(TEST): $(OBJS)\n$(CC) -o $@ $^ $(LDFLAGS) $(FPGA_LIBS) -lrt -pthread\n\n$(OBJDIR)/%.o: %.c | objdir\n $(CC) $(CFLAGS) -c $< -o $@\nclean:\n rm -rf $(TEST) $(OBJDIR)\nobjdir:\n @mkdir -p $(OBJDIR)\n.PHONY: all clean\n
"},{"location":"n6001/ug_dev_afu_host_software/#3-running-the-host-application","title":"3. Running the Host Application","text":"To run the host application, you will need to:
- Load AFU onto the FIM
- Create VF's
- Bind VF's using the OPAE Drivers
- Run application
See the associated AFU Developer Guide for details.
"},{"location":"n6001/ug_dev_afu_host_software/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_dev_afu_sim_env/","title":"AFU Simulation Environment User Guide","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_dev_afu_sim_env/#1-intended-audience","title":"1. Intended Audience","text":"The Accelerator Functional Unit (AFU) Simulation Environment (ASE) User Guide addresses both beginning and experienced developers. To be successful, you should have knowledge and experience in the following areas:
- C/C++
- Verilog/SystemVerilog
- RTL simulators such as Synopsys\u00ae VCS\u00ae or Siemens\u00ae QuestaSim\u00ae
Alternatively, you can create a team that includes developers who specialize in either RTL or software development. Previous FPGA place and route (PAR) experience is not required to be successful, but PAR experience is also a useful skill.
"},{"location":"n6001/ug_dev_afu_sim_env/#2-introduction","title":"2. Introduction","text":"The ASE provides a consistent transaction-level hardware interface and software API that allows you to develop a production-quality Accelerated Functional Unit (AFU) and host software application. The ASE supports both the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and the Intel Acceleration Stack for programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
To use the ASE Environment you must have source code in a language that RTL simulators can interpret. The following languages are possible:
- Verilog
- SystemVerilog
- VHDL
Note: The ASE supports one AFU and one application at a time. The ASE does not support multiple-slot simulation.
"},{"location":"n6001/ug_dev_afu_sim_env/#21-afu-simulation-environment-ase-overview","title":"2.1. AFU Simulation Environment (ASE) Overview","text":"ASE is a dual-process simulator. One process runs an AFU RTL simulation. The other process connects to software that runs on the RTL AFU simulation. This unified simulation environment reduces AFU hardware and software development time. The OPAE software distribution includes the ASE.
The ASE provides two interfaces:
- Software: OPAE API implemented in the C programming language.
- Hardware: PCIe SS TLP specification implemented in SystemVerilog.
Use these interfaces to deploy your IP on an OFS Integrated FPGA Platform.
"},{"location":"n6001/ug_dev_afu_sim_env/#22-ase-capabilities","title":"2.2. ASE Capabilities","text":" - The ASE provides a protocol checker to ensure protocol correctness. The ASE also provides methods to identify potential issues early, before in-system deployment.
- The ASE can help identify certain lock conditions and Configuration and Status Registers (CSR) address mapping and pointer math errors.
- The ASE tracks memory requested from the accelerator. The memory model immediately flags illegal memory transactions to locations outside of requested memory spaces. Consequently, you can fix incorrect memory accesses early, during the simulation phase.
- The ASE does not guarantee that you can synthesize an AFU. After you verify the AFU RTL functionality in the ASE, use the ASE and the Quartus\u00ae Prime Pro Edition software iteratively to generate the Accelerator Function (AF).
- The ASE does not require administrator privileges. After installing all the required tools, you can run the ASE on a plain vanilla user Linux machine.
"},{"location":"n6001/ug_dev_afu_sim_env/#23-ase-limitations","title":"2.3. ASE Limitations","text":"When using ASE in the application development cycle, consider the following limitations:
- The ASE is a transaction-level simulator. It does not model either Intel UPI- or PCIe-specific packet structures and protocol layers.
- The ASE does not simulate caching and is not a cache simulator. It cannot reliably simulate cache collisions or capacity issues.
- Although ASE models some latency parameters, it cannot model real-time system-specific latency. It is also not an accurate timing simulation of the design or latency and bandwidth of the real system. The ASE models enable you to develop functionally correct accelerators.
- The ASE does not simulate multi-AFU or multi-socket configurations.
"},{"location":"n6001/ug_dev_afu_sim_env/#24-ase-based-afu-design-workflow","title":"2.4 ASE-Based AFU Design Workflow","text":"AFU development using the ASE includes the following four stages:
-
Learning/Training: Learn to use ASE and understand the interface specifications and platform. Review sample code to get an understanding of the PCIe TLP specification and OPAE API function calls. Run samples in an ASE simulation.
-
Development Phase: Use the ASE to develop AFU RTL and software application in a single workflow. Develop RTL from the specification or by modifying existing sample RTL. The ASE includes a behavioral model of the FPGA Interface Manager (FIM) IP that provides immediate feedback on functionality during the development phase. The ASE flags errors in PCIe TLP protocols, transactions, and memory accesses. Consequently, you can fix these errors before moving to the time-consuming bitstream generation phase.
-
Bitstream Generation: Once AFU RTL and software are functionally correct, open the AFU RTL in the Intel Quartus Prime Pro Edition software. Run the place and route (PAR) tools for your platform.
Use the Synthesis reports to correct problems in the AFU RTL. Then, return to the development phase and revalidate in ASE. Bitstream generation can take hours depending on design complexity, area, and so on. After successful bitstream generation, perform timing analysis to check for timing corners, setup and hold violations, clock closure, and so on. After correcting failures found during timing analysis, revalidate in the ASE environment. When the AFU is error-free, generate the Accelerator Function (AF) bitstream that represents the AFU.
-
In-system Deployment: Test the AF in system hardware. Use Signal Tap to debug problems. Develop platform-specific software optimizations.
The AFU RTL code and OPAE software code you create in the ASE is compatible with the Quartus Prime PAR software if the following two conditions are true:
- The AFU RTL code is synthesizable.
- The AFU RTL code meets timing.
In the simulation environment, complete the following steps to create an AF bitstream and program the hardware: 1. Compile the AFU RTL in either the Synopsys\u00ae VCS\u00ae or in the Siemens\u00ae QuestaSim\u00ae simulators. 2. Compile the software application for an ASE-specific implementation of the OPAE API. 3. Synthesize the AFU RTL in the Quartus Prime Pro software to generate a bitstream. 4. Program the hardware using this bitstream.
Note: The ASE only operates using the AFU RTL source code. It cannot take the AF bitstream as input.
"},{"location":"n6001/ug_dev_afu_sim_env/#3-system-requirements","title":"3. System Requirements","text":"The OPAE software release includes the ASE. The current OPAE ASE release supports both Acceleration Stack for the Intel\u00ae Xeon\u00ae Processor with Integrated FPGA and Acceleration Stack for a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors.
The ASE is available only on 64-bit Linux operating systems with one of the following simulators: * Synopsys\u00ae VCS\u00ae Simulator (S-2023.03-SP2-1 or newer) * Siemens\u00ae QuestaSim\u00ae Simulator (2024.1 or newer)
Consult your RTL simulator vendor for Synopsys\u00ae or Siemens\u00ae for specific simulation requirements.
The ASE uses Inter-Process Communication (IPC) constructs. Under most circumstances these constructs operate without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.
Here are the other ASE requirements:
- C-Compiler: gcc 8.5.0 or above
* Boost Development libraries * UUID Development libraries * JSON Development libraries * Please see the dependencies of the OPAE System library build process
- CMake: version 3.15 or above
- Python: version 3.6.8 or above
- Intel Quartus Prime Pro 24.1: The ASE must find the
$QUARTUS_HOME/eda/sim_lib/ directory. You specify this directory during project definition in the Intel Quartus Prime Pro Edition software.
The ASE provides the env_check.sh bash script in the /opae-sim/ase/scripts directory. Run this script to verify the your installation.
Check the RTL simulator product information for supported operating systems, installation notes, and other related information. The RTL simulator must be able to perform the following functions:
- Compilation of the SystemVerilog Direct Programming Interface (DPI) constructs
- Compilation of the standard examples that are included in the installation
- Support for SystemC
"},{"location":"n6001/ug_dev_afu_sim_env/#4-package-description","title":"4. Package Description","text":"The opae-sim source directory tree is:
OPAE_SIM_BASEDIR\n |-- ase\n | |-- api\n | | |-- src\n | |-- cmake\n | |-- in\n| |-- rtl\n | |-- scripts\n | |-- sw\n
This directory tree shows the package structure of the ASE distribution. The following directories implement and run the ASE simulator:
ase: This is the ASE simulator implementation directory. It contains the following subdirectories: * api/src: This directory contains the OPAE Intel ASE implementation as a compiled library. You can link statically or dynamically to this library. * rtl: This directory contains the RTL components of the ASE. You can compile this RTL for either platform. * scripts: This directory contains several useful scripts. Refer to the ASE Scripts Section for more information. * sw: This directory contains the software components of the ASE. All simulations require the software components. The GNU Compiler Collection (GCC) compiles these components.
"},{"location":"n6001/ug_dev_afu_sim_env/#41-ase-scripts","title":"4.1. ASE Scripts","text":"The ASE distribution under the ase/scripts includes several scripts. Use these scripts to initialize, set up, and clean an existing ASE simulation environment.
"},{"location":"n6001/ug_dev_afu_sim_env/#411-simulation-tool-set-up","title":"4.1.1. Simulation Tool Set Up","text":"Use ase/scripts/ase_setup_template.sh as a template script to set up the required tools. This script has many empty placeholders for site- and environment-specific information. Consult your Electronic Design Automation (EDA) tools administrator, or the RTL simulator user guides for help setting up the tools.
"},{"location":"n6001/ug_dev_afu_sim_env/#412-ase-environment-check","title":"4.1.2. ASE Environment Check","text":"This script checks the status of the OS distribution, distro, and available system libraries. This check is a non-exhaustive. It looks for only the most important dependencies, such as the GCC version, GLIBC version, and so on.
$ ./ase/scripts/env_check.sh\n
"},{"location":"n6001/ug_dev_afu_sim_env/#413-afu-simulation-using-the-ase","title":"4.1.3. AFU Simulation Using the ASE","text":"Before configuring the ASE, follow the instructions for building the OPAE SDK and ensure that either the OPAE installed bin or the OPAE build tree bin directory is on your shell's PATH.
To simulate an AFU, replicate the ASE source tree and add the AFU-specific configuration. The OPAE installation includes several scripts to accomplish this task. The primary script, afu_sim_setup, is in the OPAE bin directory.
"},{"location":"n6001/ug_dev_afu_sim_env/#4131-afu_sim_setup","title":"4.1.3.1. afu_sim_setup","text":"The afu_sim_setup script reads a file containing a list of RTL sources (\\<rtl_sources.txt>) and configures a simulation environment for the specified sources. The afu_sim_setup command copies your base ASE environment to the \\<target dir>.
$ afu_sim_setup --sources=<rtl_sources.txt> <target dir>\n
* The only required argument to the afu_sim_setup command is the directory for the new AFU environment. Here are the usage: usage: afu_sim_setup [-h] -s SOURCES [-p PLATFORM] [-t {VCS,QUESTA,MODELSIM}]\n[-f] [--ase-mode ASE_MODE] [--ase-verbose]\ndst\n\nGenerate an ASE simulation environment for an AFU. An ASE environment is\ninstantiated from the OPAE installation and then configured for the specified\nAFU. AFU source files are specified in a text file that is parsed by\nrtl_src_config, which is also part of the OPAE base environment.\n\npositional arguments:\n dst Target directory path (directory must not exist).\n\noptional arguments:\n -h, --help show this help message and exit\n-s SOURCES, --sources SOURCES\n AFU source specification file that will be passed to\n rtl_src_config. See \"rtl_src_config --help\" for the\n file's syntax. rtl_src_config translates the source\nlist into either Quartus or RTL simulator syntax.\n -p PLATFORM, --platform PLATFORM\n FPGA Platform to simulate.\n -t {VCS,QUESTA,MODELSIM}, --tool {VCS,QUESTA,MODELSIM}\nDefault simulator.\n -f, --force Overwrite target directory if it exists.\n --ase-mode ASE_MODE ASE execution mode (default, mode 3, exits on\n completion). See ase.cfg in the target directory.\n --ase-verbose When set, ASE prints each CCI-P transaction to the\n command line. Transactions are always logged to\n work/ccip_transactions.tsv, even when not set. This\n switch sets ENABLE_CL_VIEW in ase.cfg.\n
--help The help argument lists all the arguments to afu_sim_setup.--platform: The platformargument specifies any platform defined in the platform database, including both the Integrated FPGA Platform or the Intel PAC. This argument is generally not required when a hardware platform release is installed. In that case, the OPAE_PLATFORM_ROOT environment variable points to the hardware release, which defines the platform.
afu_sim_setup is a wrapper for the following scripts. You can also access both of these scripts directly:
rtl_src_config: This script transforms the list of RTL sources into simulator configuration files.
generate_ase_environment.py: This script instantiates your simulated platform configuration.
"},{"location":"n6001/ug_dev_afu_sim_env/#4132-rtl_src_configpy","title":"4.1.3.2. rtl_src_config.py","text":"The rtl_src_config script maps a simple text file containing a list of RTL source files to an ASE configuration file for simulation or an Quartus Prime Pro configuration file for synthesis. rtl_src_config also defines preprocessor variables. Source configuration files may be hierarchical, with one file including another. rtl_src_config can construct ASE-based simulation trees or Quartus build trees.
Run rtl_src_config --help for a list of options and the required command syntax.
"},{"location":"n6001/ug_dev_afu_sim_env/#4133-generate_ase_environmentpy","title":"4.1.3.3. generate_ase_environment.py","text":"The /scripts/generate_ase_environment.py generates platform configuration files. afu_sim_setup invokes it automatically. A legacy mode in generate_ase_environment.py performs a brute-force check of the specified AFU RTL directories, attempting to define a compilation. This brute-force mode is imperfect and lists every file ending in .sv, .vs, .vhd, or .v and directories separated by +. It also may fail when compilation is order-dependent.
Run generate_ase_environment.py --help for a list of arguments.
The Synopsys and Siemens RTL simulators generate the following scripts.
- Synopsys: Creates
synopsys_sim.setup and vcs_run.tcl in the configuration directory. - Siemens: Creates
vsim_run.tcl in the configuration directory.
The run-time simultation uses the .tcl files.
Details on generated files: * vlog_files.list: Lists all the Verilog and SystemVerilog files found in the AFU directory path. * vhdl_files.list: Lists all the VHDL files found in the AFU directory path. * ase_sources.mk: Ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM: Sets the platform type to the default type or the type you specify. * Set additional VCS or QUESTA options using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile.
The simulation files use absolute paths when possible. To improve portability across users and groups, substitute environment variables in the generated files that build and run the simulator.
Note: You should manually check this file for correctness before using it in the simulation.
"},{"location":"n6001/ug_dev_afu_sim_env/#414-cleaning-the-ase-environment","title":"4.1.4. Cleaning the ASE Environment","text":"Use the ASE cleanup script located in scripts/ipc_clean.py to kill zombie simulation processes and temporary files left behind by failed simulation processes or crashes.
$ ./ase/scripts/ipc_clean.py\n\n############################################################\n# #\n# ASE IPC Cleanup script #\n# #\n############################################################\nIPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '\nRemoving .ase_ready file ...\n Type 'y' to clean up all zombie ase_simv processes : y\n Going ahead with cleaning up ASE processes opened by user_foo\n $\n
"},{"location":"n6001/ug_dev_afu_sim_env/#5-ase-usage","title":"5. ASE Usage","text":"The AFU ASE is a server-client simulation environment. The AFU RTL is the server process. The software application compiled and linked to the OPAE ASE library is the client process. Communication between server and client uses named pipes. The ASE abstracts most of the simulation infrastructure. You do not need to modify it.
Server Process: * The server process interfaces to 3rd-Party RTL Simulator packages. The server process currently supports Questasim and Synopsys VCS via the SystemVerilog-DPI library and simulator software interface. * Named pipes implement communication to the client. Named pipes also implement control, status and session management. The server process includes a pipe event monitoring engine. * SystemVerilog manages the PCIe interface. All PCIe events are logged and time stamped. * The buffer allocation calls map to POSIX Shared Memory (/dev/shm). The server-client processes share information about these buffers using named pipes.
Note: The Physical addresses generated in ASE are not realistic and are not replicable in-system.
Client Process: * The client implements an OPAE interface and a library to access the ASE platform functionality including MMIO, Buffer management, and session control. The features available depend on the platform you specify at build time. These functions are available using the OPAE API. * The client process also provides a physical memory model that simulates the RTL AFU access to physical addresses. The physical memory model simulates address translation from virtual addresses to physical addresses. * A compiled program compiles and links to the ASE implementation of OPAE library. All OPAE calls route to ASE instead of the OPAE platform driver.
Separate build scripts build the server and client processes.
- Server: A makefile in the
ase directory compiles the ASE server process, containing the ASE Software, SystemVerilog engines and the AFU RTL logic code. - Client: The main
cmake script in the root of the distribution builds the OPAE library implementations for the System and ASE. The cmake script installs the library in the lib directory.
"},{"location":"n6001/ug_dev_afu_sim_env/#51-ase-build-instructions","title":"5.1. ASE Build Instructions","text":"In this section you will set up your server to support ASE by independently downloading and installing OPAE SDK and ASE. Then, set up the required environment variables.
"},{"location":"n6001/ug_dev_afu_sim_env/#511-install-opae-sdk","title":"5.1.1. Install OPAE SDK","text":"Follow the instructions documented in the Software Installation Guide to build and install the required OPAE SDK.
"},{"location":"n6001/ug_dev_afu_sim_env/#512-setup-required-ase-environment-variables","title":"5.1.2. Setup Required ASE Environment Variables","text":"The values set to the following environment variables assume the OPAE SDK and ASE were installed in the default system directories below /usr. Setup these variables in the shell where ASE will be executed. You may wish to add these variables to the script you created to facilitate configuring your environment.
$ export QUARTUS_ROOTDIR=<path to Quartus Root Dir>\n$ export PATH=$QUARTUS_ROOTDIR/bin:$PATH\n$ export OPAE_PLATFORM_ROOT=<path to PR build tree>\n$ export PATH=/usr/bin:$PATH\n$ cd /usr/lib/python*/site-packages\n$ export PYTHONPATH=$PWD\n$ export LIBRARY_PATH=/usr/lib\n$ export LD_LIBRARY_PATH=/usr/lib64\n$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu_bbb directory> ## For VCS, set the following:\n$ export VCS_HOME=<Set the path to VCS installation directory>\n$ export PATH=$VCS_HOME/bin:$PATH\n$ export SNPSLMD_LICENSE_FILE=<License File Server>\n$ export DW_LICENSE_FILE=<DesignWare License File Server>\n\n## For QuestaSIM, set the following:\n$ export MTI_HOME=<path to QuestaSIM installation directory>\n$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH\n$ export LM_LICENSE_FILE=<>\n
"},{"location":"n6001/ug_dev_afu_sim_env/#513-install-ase-tools","title":"5.1.3. Install ASE Tools","text":"ASE is an RTL simulator for OPAE-based AFUs. The simulator emulates both the OPAE SDK software user space API and the AFU RTL interface. The majority of the FIM as well as devices such as PCIe and local memory are emulated with simple functional models.
ASE must be installed separatedly from the OPAE SDK. However, the recommendation is to install it in the same target directory as OPAE SDK. The following steps assume the OPAE SDK was installed in the default system directories below /usr, if installed in a different directory, refer to https://github.com/OFS/opae-sim for build options.
-
Clone the opae-sim repository.
$ cd $OFS_BUILD_ROOT\n$ git clone https://github.com/OFS/opae-sim.git\n$ cd opae-sim\n# Checkout tag and branch based on your OFS release\n$ git checkout tags/<tag> -b <branch>\n
-
Create a build directory and build ASE to be installed under the default system directories along with OPAE SDK.
$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX=/usr ..\n$ make\n
Optionally, if the desire is to install ASE binaries in a different location to the system's default, provide the path to CMAKE through the CMAKE_INSTALL_PREFIX switch, as follows.
$ cmake -DCMAKE_INSTALL_PREFIX=<</some/arbitrary/path>> ..
- Install ASE binaries and libraries under the system directory
/usr. $ sudo make install
"},{"location":"n6001/ug_dev_afu_sim_env/#514-ase-simulator-server-build-instructions","title":"5.1.4. ASE Simulator (Server) Build Instructions","text":"ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces asimulator configuration based on the differentiation key. These keys are set automatically by afu_sim_setup.
$ afu_sim_setup -s ./hw/rtl/sources.txt -t VCS afu_sim\n\nCopying ASE from /usr/local/share/opae/ase...\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nTool Brand: VCS\nLoading platform database: /home/user/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_pr/pr_build_template/hw/lib/platform/platform_db/ofs_agilex_adp.json\nLoading platform-params database: /usr/share/opae/platform/platform_db/platform_defaults.json\nLoading AFU database: /usr/share/opae/platform/afu_top_ifc_db/ofs_plat_afu.json\nWriting rtl/platform_afu_top_config.vh\nWriting rtl/platform_if_addenda.txt\nWriting rtl/platform_if_includes.txt\nWriting rtl/ase_platform_name.txt\nWriting rtl/ase_platform_config.mk and rtl/ase_platform_config.cmake\nASE Platform: discrete (FPGA_PLATFORM_DISCRETE)\n
Change directory to the targeted simuation directory dst and make simulation project.
$ cd afu_sim\n$ make\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\nLocal memory model set to BASIC\nmkdir -p work/verilog_libs\ncd work; quartus_sh --simlib_comp -family agilex7 -tool vcsmx -language verilog -gen_only -cmd_file quartus_vcs_verilog.sh; chmod a+x quartus_vcs_verilog.sh\n...
"},{"location":"n6001/ug_dev_afu_sim_env/#514-ase-runtime-instructions","title":"5.1.4. ASE Runtime Instructions","text":"The ASE server-client simulator makes the server before the client. Use two terminal windows to start the simulation.
- Terminal 1: In the simulation directroy
dst, run make sim. The ASE initializes and the AFU issues a reset and then waits for incoming transactions. The software application must wait until the \"Ready for Simulation\" message displays.
Specify the environment variable ASE_WORKDIR Terminal 1.
# Invoke the simulator\n$ make sim\n#################################################################\n# #\n# OPAE Intel(R) Xeon(R) + FPGA Library #\n# AFU Simulation Environment (ASE) #\n# #\n#################################################################\nSIMULATOR=VCS\nCC=gcc\nFPGA_FAMILY=agilex7\n\nASE platform set to DISCRETE mode\n\n.\n .\n .\n [SIM] Transaction Logger started\n [SIM] Simulator started...\n [SIM] +CONFIG /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg file found !\n [SIM] +SCRIPT /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase_regress.sh file found !\n [SIM] ASE running with seed => 0\n[SIM] PID of simulator is 1822681\n[SIM] Reading /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/ase.cfg configuration file\n [SIM] ASE was started in Mode 3 (Server-Client with Sw SIMKILL (long runs)\n[SIM] ASE Mode: Server-Client mode with SW SIMKILL (long runs)\n[SIM] Inactivity kill-switch ... DISABLED\n [SIM] Reuse simulation seed ... ENABLED\n [SIM] ASE Seed ... 1234\n[SIM] ASE Transaction view ... DISABLED\n [SIM] User Clock Frequency ... 312.500000 MHz, T_uclk = 3200 ps\n [SIM] Amount of physical memory ... 128 GB\n [SIM] Current Directory located at =>\n [SIM] /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] Creating Messaging IPCs...\n [SIM] Information about allocated buffers => workspace_info.log\n [SIM] Sending initial reset...\n .\n .\n .\n [SIM] ASE lock file .ase_ready.pid written in work directory\n [SIM] ** ATTENTION : BEFORE running the software application **\n [SIM] Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>\n [SIM] $SHELL | Run:\n [SIM] ---------+---------------------------------------------------\n [SIM] bash/zsh | export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] tcsh/csh | setenv ASE_WORKDIR /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n [SIM] For any other $SHELL, consult your Linux administrator\n [SIM]\n[SIM] Ready for simulation...\n [SIM] Press CTRL-C to close simulator...\n
You can close Terminal 1 make sim by issuing a SIGTERM to the relevant ase_simv process or by typing CTRL-C.
- Terminal 2: First set the environment variable
ASE_WORKDIR as specified in Terminal 1. In this example ASE_WORKDIR is set to /home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work. Then, start the software application using with_ase, which will run the binary using the ASE simulation library instead of the standard libopae-c.
# Set ASE_WORKDIR environment variable\n$ export ASE_WORKDIR=/home/user/OFS_BUILD_ROOT/example_afu/afu_sim/work\n\n# Run the application\n$ with_ase ./hello_fpga\n
Note: After the application exits, the simulation is complete. Close the simulator to allow the waveform dump process to complete. In Terminal 1, type the CTRL-C command.
[SIM] Closing message queue and unlinking...\n [SIM] Session code file removed\n [SIM] Removing message queues and buffer handles ...\n [SIM] Cleaning session files...\n [SIM] Simulation generated log files\n [SIM] Transactions file | $ASE_WORKDIR/log_ase_events.tsv\n [SIM] Workspaces info | $ASE_WORKDIR/workspace_info.log\n [SIM]\n[SIM] Tests run => 0\n[SIM]\n[SIM] Sending kill command...\n [SIM] Simulation kill command received...\n$finish called from file \"/home/user//OFS_BUILD_ROOT/examples_afu/afu_sim/rtl/pcie_ss_tlp/ase_pcie_ss_emulator.sv\", line 388.\n$finish at simulation time 16396997500\nV C S S i m u l a t i o n R e p o r t\nTime: 16396997500 ps\nCPU Time: 506.240 seconds; Data structure size: 4.3Mb\nWed Mar 13 18:26:28 2024\n
Upon completion, the simulation generates the following files:
- Waveform dump:
make wave opens the waveform for the selected tool.* $ASE_WORKDIR/inter.vpd: VCS Waveform file * $ASE_WORKDIR/vsim.wlf: Questa waveform file.
$ASE_WORKDIR/log_ase_events.tsv: Events log listing all events observed between the host and afu interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.$ASE_WORKDIR/workspace_info.log: Information about buffers the simulation opened.
"},{"location":"n6001/ug_dev_afu_sim_env/#52-ase-makefile-targets","title":"5.2. ASE Makefile Targets","text":"COMMAND DESCRIPTION make Build the HW Model using RTL supplied make sim Run simulator - ASE can be run in one of 4 modes set in ase.cfg - A regression mode can be enabled by writing ASE_MODE = 4 in ase.cfg and supplying an ase_regress.sh script make wave Open the waveform (if created) to be run after simulation completes make clean Clean simulation files make distclean Clean ASE sub-distribution"},{"location":"n6001/ug_dev_afu_sim_env/#53-ase-makefile-variables","title":"5.3. ASE Makefile Variables","text":"Makefile switch DESCRIPTION ASE_CONFIG Directly input an ASE configuration file path (ase.cfg) ASE_SCRIPT Directly input an ASE regression file path (ase_regress.sh, for ASE_MODE=4) SIMULATOR Directly input a simulator brand (select between 'VCS' or 'QUESTA') ASE_DISABLE_CHECKER Legacy - Disable CCI-P protocol checker module (set to '1' might speed up simulation) WARNING => NO warnings on hazards, protocol checks, timeouts will be generated. This option must be ONLY used if the design is already CCI-P compliant and fast simulation of app-specific logic is needed"},{"location":"n6001/ug_dev_afu_sim_env/#54-ase-runtime-configuration-options","title":"5.4. ASE Runtime Configuration Options","text":"The ASE configuration file configures simulator behavior. An example configuration script is available at ase/ase.cfg
Switch Name Default Description ASE_MODE 1 ASE mode has the following valid values: 1 : Standard Server-Client Mode2 : Simulator stops after ASE_TIMEOUT clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ASE_MODE=1) ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOUT clocks of CCI-P interface inactivity. ASE_NUM_TESTS 4 (only if ASE_MODE=4) Number of tests in regression mode. If incorrectly set, the simulator may exit pre-maturely or stall waiting for tests to get started. ENABLE_REUSE_SEED 1 When set to 1, reuses the simulation seed, so that CCI-P transactions replay with the previous addresses. When set to 0, obtains a new seed. ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, enabled when ENABLE_REUSE_SEED is set to 1, otherwise the simulations uses a different seed. At the end of the simulation, the ASE writes the current seed to $ASE_WORKDIR/ase_seed.txt. ENABLE_CL_VIEW 1 The ASE prints all CCI-P transactions. On long simulation runs, setting ENABLE_CL_VIEW to 0 may reduce simulation time. USR_CLK_MHZ 312.50000 Configurable User Clock (Read by simulator as float) PHYS_MEMORY_AVAILABLE_GB 128 Restricts ASE address generation the specified memory range."},{"location":"n6001/ug_dev_afu_sim_env/#55-logging-verbosity-control","title":"5.5. Logging Verbosity Control","text":"ASE provides the following three levels for logging message verbosity. By default, these messages print to stdout:
- ASE_INFO: Prints mandatory information messages required to specify operation.
- ASE_ERR: Prints error messages during operation.
- ASE_MSG: Prints general messages indicating check points in the ASE. Suppress these messages by setting the environment variable
ASE_LOG to 0.
Two log levels are supported in ASE, controlled by env(ASE_LOG)
- ASE_LOG=0 | ASE_LOG_SILENT : Only INFO, ERR messages are posted
- ASE_LOG!=0 | ASE_LOG_MESSAGE : All MSG, INFO, ERR messages are posted
The following command include the ASE_MSG category:
$ ASE_LOG=1 with_ase ./hello_fpga\n
You cannot suppress warnings and errors."},{"location":"n6001/ug_dev_afu_sim_env/#56-troubleshooting-and-error-reference","title":"5.6. Troubleshooting and Error Reference","text":"The following list of ASE errors and warnings is not comprehensive:
Observation Problem Next Steps Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in the ASE. If using ASE_MODE=2 (Daemon with timeout), in the ase.cfg file, increase the ASE_TIMEOUT setting or disable ASE_TIMEOUT. ASE simulation build error - compilation, or linking failed GCC version might be too old. Use the ./scripts/env_check.sh script to identify issues. Synopsys VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problems with incremental compilation. Clean the ASE environment using $ make clean If this command fails, clean the distribution with $ ./distclean.shthen rebuild the simulation. ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the ./scripts/ipc_clean.py script to clean IPC instances. Check if the System Requirements have been met. If problems continue, increase resource limits for your Linux distribution. $ASE_WORKDIR environment variable has not been set up Application cannot find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are in green text. .ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_WORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. $ echo $ASE_WORKDIR Wait for simulator to print: Ready for Simulation ase_sources.mk: No such file or directory ASE Environment has not been generated. Generate an AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. You can use ./scripts/generate_ase_environment.pyto generate these files. An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ./scripts/ipc_clean.py script to clean the simulation temporary files using: $ make clean. Then rebuild the simulator."},{"location":"n6001/ug_dev_afu_sim_env/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_dev_pim_based_afu/","title":"PIM Based AFU Developer Guide","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_dev_pim_based_afu/#1-introduction","title":"1. Introduction","text":"When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.
This guide will walk you through creating a PIM-Based AFU, including:
- AFU Build environment
- Using the PIM to interface with an AFU
- AFU Design
- Software Development
- Packaging the AFU
For more information on the PIM, refer to PIM Core Concepts.
For PIM based examples AFU's to provide templates in designing your own AFU, refer to examples AFU.
For steps on compiling your AFU, please see the associated platform's AFU Developer Guide.
For steps on integrating your AFU into the FIM, please see the associated platform's FIM Developer Guide.
"},{"location":"n6001/ug_dev_pim_based_afu/#2-afu-build-environment","title":"2. AFU Build Environment","text":"The Platform Interface Manager (PIM) acts as a gateway between the board-specific platform and the generic AFU. It manages resources, handles communication protocols, and translates platform-specific signals to a format the AFU can understand. The PIM wraps all FIM devices in a single container as an interface named ofs_plat_if, which is passed to the top-level AFU module ofs_plat_afu.
The below table shows the supported interfaces for each channel type by the PIM.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X A Partial Reconfiguration (PR) build template is used for configuring a PR AFU build and is derived from a synthesized FIM. The template includes the PIM and the afu_synth_setup script, which generates a Quartus build environment for an AFU. The build environment is instantiated from a FIM release and then configured for the specified AFU. The AFU source files are specified in a text file parsed by the script when creating the Quartus project.
The PIM is instantiated in the build environment from an .ini file describing the platform, located at <PR build template>/hw/lib/platform/platform_db/<ofs platform>.ini
Example N6001 FIM .ini file, <pr_build_template>/hw/lib/platform/platform_db/ofs_agilex.ini
;; Platform Interface Manager configuration\n;;\n;; Intel\u00ae Agilex adp board\n;; OFS FIM\n;;\n;; Local memory with AXI-MM interface\n;;\n[define]\nPLATFORM_FPGA_FAMILY_AGILEX=1\nPLATFORM_FPGA_FAMILY_AGILEX7=1\n;; Indicates that ASE emulation of the afu_main interface is offered\nASE_AFU_MAIN_IF_OFFERED=1\nnative_class=none\n;; Early versions of afu_main checked INCLUDE_HSSI_AND_NOT_CVL. When\n;; this macro is set, the presence of HSSI ports in afu_main() is\n;; controlled by INCLUDE_HSSI.\nAFU_MAIN_API_USES_INCLUDE_HSSI=1\n[clocks]\npclk_freq=int'(ofs_fim_cfg_pkg::MAIN_CLK_MHZ)\n;; Newer parameter, more accurate when pclk is not an integer MHz\npclk_freq_mhz_real=ofs_fim_cfg_pkg::MAIN_CLK_MHZ\nnative_class=none\n[host_chan]\nnum_ports=top_cfg_pkg::PG_AFU_NUM_PORTS\nnative_class=native_axis_pcie_tlp\ngasket=pcie_ss\nmmio_addr_width=ofs_fim_cfg_pkg::MMIO_ADDR_WIDTH_PG\nnum_intr_vecs=ofs_fim_cfg_pkg::NUM_AFU_INTERRUPTS\n;; Minimum number of outstanding flits that must be in flight to\n;; saturate bandwidth. Maximum bandwidth is typically a function\n;; of the number flits in flight, indepent of burst sizes.\nmax_bw_active_flits_rd=1024\nmax_bw_active_flits_wr=128\n;; Recommended number of times an AFU should register host channel\n;; signals before use in order to make successful timing closure likely.\nsuggested_timing_reg_stages=0\n[local_mem]\nnative_class=native_axi\ngasket=fim_emif_axi_mm\nnum_banks=ofs_fim_mem_if_pkg::NUM_MEM_CHANNELS\n;; Address width (line-based, ignoring the byte offset within a line)\naddr_width=ofs_fim_mem_if_pkg::AXI_MEM_ADDR_WIDTH-$clog2(ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH/8)\ndata_width=ofs_fim_mem_if_pkg::AXI_MEM_WDATA_WIDTH\necc_width=0\n;; For consistency, the PIM always encodes burst width as if the bus were\n;; Avalon. Add 1 bit: Avalon burst length is 1-based, AXI is 0-based.\nburst_cnt_width=8+1\nuser_width=ofs_fim_mem_if_pkg::AXI_MEM_USER_WIDTH\nrid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nwid_width=ofs_fim_mem_if_pkg::AXI_MEM_ID_WIDTH\nsuggested_timing_reg_stages=2\n[hssi]\nnative_class=native_axis_with_fc\nnum_channels=ofs_fim_eth_plat_if_pkg::MAX_NUM_ETH_CHANNELS\n;; Sideband interface specific to this platform. It is used for passing\n;; state through plat_ifc.other.ports[] that the PIM does not manage.\n[other]\n;; Use the PIM's \"generic\" extension class. The PIM provides the top-level\n;; generic wrapper around ports and the implementation of the type is set below.\ntemplate_class=generic_templates\nnative_class=ports\n;; All PIM wrappers are vectors. Depending on the data being passed through\n;; the interface, FIMs may either use more ports or put vectors inside the\n;; port's type.\nnum_ports=1\n;; Data type of the sideband interface\ntype=ofs_plat_fim_other_if\n;; Import the \"other\" SystemVerilog definitions into the PIM (relative path)\nimport=import/extend_pim\n
The OFS scripts choose the proper subset of PIM sources to map from standard PIM AFU interfaces to physical hardware. Given an input .ini configuration file, gen_ofs_plat_if constructs an ofs_plat_if interface that is tailored to the target platform. Templates make it possible for the source tree to support multiple devices of similar types, such as both DDR and HBM, on a single board.
Each major section in a platform .ini file corresponds to one or more devices of the same type. Same-sized banks of local memory share a single .ini section, with the number of banks as a parameter in the section. The same is true of HSSI ports and, on some multi-PCIe systems, of host channels. All devices in a section must share the same properties. If there are two types of local memory on a board with different address or data widths, they must have their own local memory sections. Separate sections of the same type must be named with monotonically increasing numeric suffixes, e.g. local_memory.0 and local_memory.1. The trailing .0 is optional. host_channel.0 and host_channel are equivalent.
The gen_ofs_plat_if script, which composes a platform-specific PIM given an .ini file, uses the ofs_plat_if/src/rtl/ tree as a template. The script copies sources into the target ofs_plat_if interface within a release, generates some top-level wrapper files and emits rules that import the generated tree for simulation or synthesis.
For more information, refer to PIM Board Vendors
"},{"location":"n6001/ug_dev_pim_based_afu/#21-pim-resources","title":"2.1. PIM Resources","text":"The PIM provides a collection of RTL interfaces and modules. These are copied over from ofs-platform-afu-bbb to <afu build dir>/build/platform/ofs_plat_if/rtl/. The modules brought over are based on the FIM's native interfaces:
- ofs_plat_if.vh: PIM's top level wrapper interface for passing all top-level interfaces into an AFU and is copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ofs_plat_if.vh. The 'ofs_plat_if' file typically contains the definition of the interface signals and parameters that connect the AFU to the PIM. This includes details about the data and control signals that the AFU and PIM use to communicate, such as clocks, host channels or local memory.
- PIM interfaces are defined in base_ifcs and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/base_ifcs. This base interface classes tree is a collection of generic interface definitions (e.g. Avalon and AXI) and helper modules (e.g. clock crossing and pipeline stage insertion).
- PIM modules are defined in ifcs_classes and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/ifc_classes. The PIM-provided modules (aka shims) transform FIM interfaces to PIM interfaces. On the AFU side of its shims, all PIM modules share common base AXI and Avalon interfaces. The PIM modules are classified by the channels they support: - host_chan
- local_memory
- hssi
- Other
- PIM utilities are defined in utils and copied over to
<afu build dir>/build/platform/ofs_plat_if/rtl/utils. Utilities include primitive shims, such as FIFOs, memories, and reorder buffers.
"},{"location":"n6001/ug_dev_pim_based_afu/#3-using-pim-to-interface-with-an-afu","title":"3. Using PIM to interface with an AFU","text":"To interface the PIM with an AFU:
- Create top level module ofs_plat_afu.sv.
- For each Subsystem used by your AFU, create individual channel interfaces using your selected bus protocols and connect the channel PIM Shims based on selected bus protocols.
- PCIe - Host Channel
- Local Memory
- HSSI
- Tie off all unused channels/ports.
- Connect the channel interfaces to the AFU module.
"},{"location":"n6001/ug_dev_pim_based_afu/#31-top-level-module-ofs_plaf_afu","title":"3.1. Top Level Module - ofs_plaf_afu","text":"For a PIM based AFU, start with the required top level module, ofs_plat_afu, which has a single interface, ofs_plat_if, containing all the FIM connections. It should include 'ofs_plat_if.vh' to ensure that the PIM resources are available.
`include \"ofs_plat_if.vh\"\n//\n// Top level PIM-based module.\n//\n\nmodule ofs_plat_afu\n (\n// All platform wires, wrapped in one interface.\n ofs_plat_if plat_ifc\n );\n
The SystemVerilog interface ofs_plat_if wraps all connections to the FIM's devices. The contents of ofs_plat_if may vary from device to device. Portability is maintained by conforming to standard naming conventions. ofs_plat_if is, itself, a collection of interface wrappers to groups of devices. Each PCIe virtual or physical function is treated by the PIM as a separate channel. For more information, refer to PIM AFU Interface
"},{"location":"n6001/ug_dev_pim_based_afu/#32-host-channel","title":"3.2. Host Channel","text":"The host channel serves as the communication pathway between the host and the FPGA. It facilitates the exchange of commands, data, and control signals, allowing the host to interact with the FPGA and manage accelerated functions.
For more information, refer to PIM IFC Host Channel
"},{"location":"n6001/ug_dev_pim_based_afu/#321-create-the-host-channel-interfaces-to-the-afu","title":"3.2.1. Create the host channel interfaces to the AFU","text":"The Host Memory interface is designed to facilitate the communication between the host and the FPGA as it allows the FPGA to access data stored in the host's main memory or to receive data from the host for processing.
The Host Memory supported interface:
- AVMM
- AVMM-RDWR
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`HOST_CHAN_AXI_MEM_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n
The Memory-Mapped I/O (MMIO) allows the host to access and control specific registers or memory locations within the FPGA's address space. This interface is commonly used for configuring and interacting with hardware components through memory-mapped addresses.
The MMIO supported interface:
- AVMM
- AXI-Lite
AXI-Lite example:
ofs_plat_axi_mem_lite_if\n #(\n`HOST_CHAN_AXI_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n
"},{"location":"n6001/ug_dev_pim_based_afu/#322-connect-the-host-channel-to-the-pim-shim","title":"3.2.2. Connect the host channel to the PIM Shim","text":"Using the PIM Shim, host channel FIM interface is bridged over to the AFU's host memory interface and MMIO interface, making it usable for the AFU.
AXI example:
ofs_plat_host_chan_as_axi_mem_with_mmio primary_axi\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n// These ports would be used if the PIM is told to cross to\n // a different clock. In this example, native pClk is used.\n .afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#323-avalon-example","title":"3.2.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
#Host memory \nofs_plat_avalon_mem_rdwr_if\n #(\n`HOST_CHAN_AVALON_MEM_RDWR_PARAMS,\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nhost_mem();\n#MMIO\nofs_plat_avalon_mem_if\n #(\n`HOST_CHAN_AVALON_MMIO_PARAMS(64),\n .LOG_CLASS(ofs_plat_log_pkg::HOST_CHAN)\n)\nmmio64_to_afu();\n#PIM Shim\nofs_plat_host_chan_as_avalon_mem_rdwr_with_mmio primary_avalon\n (\n.to_fiu(plat_ifc.host_chan.ports[0]),\n .host_mem_to_afu(host_mem),\n .mmio_to_afu(mmio64_to_afu),\n\n.afu_clk(),\n .afu_reset_n()\n);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#33-local-memory","title":"3.3. Local Memory","text":"Local memory is off-chip memory connected to an FPGA but not visible to the host as system memory. Local memory is organized in groups and banks. Within a group, all banks have the same address and data widths.
For more information, refer to PIM IFC Local Memory
"},{"location":"n6001/ug_dev_pim_based_afu/#331-create-the-local-memory-interfaces-to-the-afu","title":"3.3.1. Create the local memory interfaces to the AFU","text":"The Local Memory supported interfaces:
- AVMM
- AXI-MM
AXI-MM example:
ofs_plat_axi_mem_if\n #(\n`LOCAL_MEM_AXI_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM),\n .BURST_CNT_WIDTH($clog2(MAX_BURST_SIZE/ofs_plat_host_chan_pkg::DATA_WIDTH_BYTES))\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\n
"},{"location":"n6001/ug_dev_pim_based_afu/#332-connect-local-memory-to-the-pim-shim","title":"3.3.2. Connect local memory to the PIM Shim","text":"Using the PIM Shim, the local memory FIM interface is bridged over to the AFU's local memory interface, making it usable for the AFU.
AXI example:
genvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_axi_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(host_mem.clk),\n .afu_reset_n(host_mem.reset_n)\n);\nend\n endgenerate\n
"},{"location":"n6001/ug_dev_pim_based_afu/#333-avalon-example","title":"3.3.3. Avalon Example","text":"The following examples show the steps for a Avalon MM interface:
ofs_plat_avalon_mem_if\n #(\n`LOCAL_MEM_AVALON_MEM_PARAMS_DEFAULT,\n .LOG_CLASS(ofs_plat_log_pkg::LOCAL_MEM)\n)\nlocal_mem_to_afu[local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS]();\ngenvar b;\ngenerate\n for (b = 0; b < local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS; b = b + 1)\nbegin : mb\n ofs_plat_local_mem_as_avalon_mem\n #(\n.ADD_CLOCK_CROSSING(1)\n)\nshim\n (\n.to_fiu(plat_ifc.local_mem.banks[b]),\n .to_afu(local_mem_to_afu[b]),\n\n.afu_clk(mmio64_to_afu.clk),\n .afu_reset_n(mmio64_to_afu.reset_n)\n);\nend\n endgenerate\n
"},{"location":"n6001/ug_dev_pim_based_afu/#34-high-speed-serial-interface-hssi","title":"3.4. High Speed Serial Interface (HSSI)","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
"},{"location":"n6001/ug_dev_pim_based_afu/#341-create-the-hssi-interfaces-to-the-afu","title":"3.4.1. Create the HSSI interfaces to the AFU","text":"The High-Speed Serial Interface enables high-speed serial communication between the FPGA and external devices. It's commonly used for tasks such as high-speed data streaming, interfacing with storage devices, or connecting to network components.
A vector of HSSI channels holds RX and TX AXI-S data interfaces. In addition to the data streams, each channel has a flow control interface on which pause requests are passed. Within a single channel, the RX, TX and pause interfaces share a clock. The clock is not guaranteed to be common across channels. The PIM provides only an AXI-S data option.
Note: Clock Crossing not supported, parameter and ports are there for standardization
// HSSI Channels\n ofs_plat_hssi_channel_if\n #(\n// Log AXI transactions in simulation\n .LOG_CLASS(ofs_plat_log_pkg::HSSI)\n)\nhssi_to_afu[ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS](); genvar c;\ngenerate\n for (c = 0; c < ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS; c = c + 1)\nbegin : ch\n\nofs_plat_hssi_as_axi_st hssi_shim\n (\n.to_fiu(plat_ifc.hssi.channels[c]),\n .rx_st(hssi_to_afu[c].data_rx), // HSSI->AFU\n .tx_st(hssi_to_afu[c].data_tx), // AFU->HSSI\n .fc(hssi_to_afu[c].fc), // Flow Control\n // These are present in all PIM interfaces, though not available with hssi.\n .afu_clk(),\n .afu_reset_n()\n);\nend\n endgenerate\n
"},{"location":"n6001/ug_dev_pim_based_afu/#35-tie-off-unused-ports","title":"3.5. Tie Off Unused ports","text":"In digital design, unused input ports can lead to unpredictable behavior. To prevent this, unused ports are \"tied off\" to a known state. Tie-offs are passed to the PIM as bit masks in parameters. The mask makes it possible to indicate, for example, that a single local memory bank is being driven.
ofs_plat_if_tie_off_unused\n #(\n// Only using channel 0\n.HOST_CHAN_IN_USE_MASK(1)\n// Use two memory banks\n .LOCAL_MEM_IN_USE_MASK(3)\n// Use 4 HSSI channel\n .HSSI_IN_USE_MASK(15)\n)\ntie_off(plat_ifc);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#36-afu-instantiation","title":"3.6. AFU Instantiation","text":"Instantiate the AFU in ofs_plat_afu.sv and connect to the channel interfaces.
// =========================================================================\n//\n // Instantiate the AFU.\n //\n // =========================================================================\nexample_afu\n #(\n.NUM_LOCAL_MEM_BANKS(local_mem_cfg_pkg::LOCAL_MEM_NUM_BANKS),\n .NUM_ETHERNET_CHANNELS(ofs_fim_eth_if_pkg::NUM_ETH_CHANNELS)\n)\nafu_inst\n (\n.mmio64_to_afu,\n .host_mem,\n .local_mem_to_afu,\n .hssi_to_afu\n );\n
"},{"location":"n6001/ug_dev_pim_based_afu/#4-afu","title":"4. AFU","text":"The AFU requires that each channel uses the interfaces supported by the PIM. The below table shows the supported interfaces for each channel type. The MMIO channel is the only channel required by the FIM, while all other channels are optional and can be tied off.
Channel AXI-MM AXI-Lite Avalon MM Avalon MM Rd/Wr HSSI Channel MMIO X X Host Memory X X X Local Memory X X HSSI X"},{"location":"n6001/ug_dev_pim_based_afu/#41-afu-top-level-module","title":"4.1. AFU top level module","text":"The AFU module should match the interfaces provided by the PIM. Including ofs_plat_if.vh in your module will bring in the base interface classes and channel interfaces:
`include \"ofs_plat_if.vh\"\nmodule example_afu\n #(\nparameter NUM_LOCAL_MEM_BANKS = 2,\n parameter NUM_ETHERNET_CHANNELS = 2\n)\n(\n// CSR interface (MMIO on the host)\nofs_plat_axi_mem_lite_if.to_source mmio64_to_afu,\n\n// Host memory (DMA)\nofs_plat_axi_mem_if.to_sink host_mem_to_afu,\n\n// Local memory interface ofs_plat_axi_mem_if.to_sink local_mem_to_afu[NUM_LOCAL_MEM_BANKS],\n\n// High Speed Serial Interface\n ofs_plat_hssi_channel_if hssi_to_afu [NUM_ETHERNET_CHANNELS]\n);\n
"},{"location":"n6001/ug_dev_pim_based_afu/#42-afu-interfaces","title":"4.2. AFU Interfaces","text":"The AXI-MM and AXI-Lite interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/axi directory.
For AXI-MM and AXI-Lite, the handshaking signals (Ready and Valid) are separated from each of the interfaces (aw, w, b, ar, r). For example, the aw interface is defined as:
t_axi_mem_aw aw;\nlogic awvalid;\nlogic awready;\n
The Avalon MM interfaces are defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/avalon directory. There are two Avalon MM interfaces, a traditional interface (ofs_plat_avalon_mem_if) with shared read and write operations and a split-bus interface (ofs_plat_avalon_mem_rdwr_if) which separates the read and write channels.
The HSSI Channel interface is defined in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/hssi directory. The HSSI channel is comprised of three interfaces, RX AXIS, TX AXIS and flow control. These interfaces are defined in <afu_build dir>/build/ofs-common/src/fpga_family/<device family>/hssi_ss/inc/ofs_eth_fim_if.sv.
Clock and Resets definition and header files are in the <afu_build_dir>/build/platform/ofs_plat_if/rtl/base_ifcs/clocks directory. By default, each channel has its own associated clock and reset which is derived from it connected subsystem. Using the ADD_CROSS_CLOCKING option with the PIM shims, allows the channels to all be on the same clock domain.
// Each interface names its associated clock and reset.\n logic afu_clk;\nassign afu_clk = mmio64_to_afu.clk;\nlogic afu_reset_n;\nassign afu_reset_n = mmio64_to_afu.reset_n;\n
"},{"location":"n6001/ug_dev_pim_based_afu/#43-csr-interface","title":"4.3. CSR Interface","text":"The MMIO is the only required channel for the AFU. Besides providing a control and status interface for the AFU, the MMIO is required to have base registers as described in the Device Feature List Overview, which is used by the OPAE SW.
When using the host channel, the Host creates shared buffers created between the Host CPU and FPGA. The base address of these buffers should be passed to the AFU using the MMIO interface.
"},{"location":"n6001/ug_dev_pim_based_afu/#44-addressing","title":"4.4. Addressing","text":"The interface addressing depends on the interface's bus protocol, the below table shows the addressing based of interface.
Interface Addressing AXI Byte Avalon Word"},{"location":"n6001/ug_dev_pim_based_afu/#45-replicating-interface-parameters","title":"4.5. Replicating Interface Parameters","text":"When creating interfaces in the AFU, using `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS or `OFS_PLAT_AVALON_MEM_IF_REPLICATE_PARAMS allows the interface to have the same parameters as the channel interface.
// The read ports will be connected to the read engine and the write ports unused.\n// This will split the read channels from the write channels but keep\n// a single interface type.\nofs_plat_axi_mem_if\n #(\n// Copy the configuration from host_mem\n `OFS_PLAT_AXI_MEM_IF_REPLICATE_PARAMS(host_mem)\n)\nhost_mem_rd();\n
"},{"location":"n6001/ug_dev_pim_based_afu/#46-systemverilog-packages","title":"4.6. SystemVerilog Packages","text":"The AFU project provides System Verilog packages, which provide configuration details for the different channels.
The Host Channel and Local Memory System Verilog packages are included by default in the Quartus Project:
- Host Channel Package: ofs_plat_host_chan_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/host_chan/afu_ifcs/include/ofs_plat_host_chan_pkg.sv - Local Memory Package: local_mem_cfg_pkg
<afu_build_dir>/build/platform/ofs_plat_if/rtl/ifc_classes/local_mem/local_mem_cfg_pkg.sv
The HSSI Channel System Verilog package is not included by default, therefore it needs to be imported:
- HSSI Channel Package: ofs_fim_eth_if_pkg
<afu_build_dir>/build/ofs-common/src/fpga_family/agilex/hssi_ss/inc/ofs_fim_eth_if_pkg.sv import ofs_fim_eth_if_pkg::*;\n
"},{"location":"n6001/ug_dev_pim_based_afu/#5-host-software-development","title":"5. Host Software Development","text":"The host application is used to control the AFU and manage data transfer between the host and the AFU. Refer to the AFU Host Software Developer Guide for creating a host application.
"},{"location":"n6001/ug_dev_pim_based_afu/#6-packaging-the-afu","title":"6. Packaging the AFU","text":"Besides the RTL and software files, an AFU requires an Accelerator Description File and source list file. These files are used during the build process.
"},{"location":"n6001/ug_dev_pim_based_afu/#61-accelerator-description-file","title":"6.1. Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration.
example_afu.json:
{\n\"version\": 1,\n \"afu-image\": {\n\"power\": 0,\n \"clock-frequency-high\": \"auto\",\n \"clock-frequency-low\": \"auto\",\n \"afu-top-interface\":\n {\n\"class\": \"ofs_plat_afu\"\n},\n \"accelerator-clusters\":\n [\n{\n\"name\": \"example_afu\",\n \"total-contexts\": 1,\n \"accelerator-type-uuid\": \"01234567-89ab-cdef-fedc-ba9876543210\"\n}\n]\n}\n}\n
- power - Accelerator Function power consumption, in watts. Set to 0 for Intel ADP platforms.
- clock-frequency-high - Clock frequency for uclk_usr in MHz. (optional)
- clock-frequency-low - Clock frequency for uclk_usr_div2 in MHz. (optional)
- afu-top-interface:
- class : Set to \"ofs_plat_afu\" for PIM based AFU, \"afu_main\" for native/hybrid AFU's.
- accelerator-clusters:
- name : name of AFU
- total-contexts : Set to '1'
- accelerator-type-uuid : 128-bit Universally Unique Identifier (UUID) used to identify the AFU.
The ASE and synthesis setup scripts call afu_json_mgr to create afu_json_info.vh:
//\n// Generated by afu_json_mgr from \u2026/hw/rtl/example_afu.json\n//\n\n`ifndef __AFU_JSON_INFO__\n`define __AFU_JSON_INFO__\n\n`define AFU_ACCEL_NAME \"example_afu\"\n`define AFU_ACCEL_NAME0 \"example_afu\"\n`define AFU_ACCEL_UUID 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_ACCEL_UUID0 128'h01234567_89ab_cdef_fedc_ba9876543210\n`define AFU_IMAGE_POWER 0\n`define AFU_TOP_IFC \"ofs_plat_afu\"\n`endif // __AFU_JSON_INFO__\n
The Makefile calls the afu_json_mgr to create afu_json_info.h:
//\n// Generated by afu_json_mgr from ../hw/rtl/example_afu.json\n//\n#ifndef __AFU_JSON_INFO__\n#define __AFU_JSON_INFO__\n#define AFU_ACCEL_NAME \" example_afu \"\n#define AFU_ACCEL_NAME0 \" example_afu \"\n#define AFU_ACCEL_UUID \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_ACCEL_UUID0 \"01234567-89AB-CDEF-FEDC-BA9876543210\"\n#define AFU_IMAGE_POWER 0\n#define AFU_TOP_IFC \"ofs_plat_afu\"\n#endif // __AFU_JSON_INFO__\n
"},{"location":"n6001/ug_dev_pim_based_afu/#62-source-list-file","title":"6.2. Source List File","text":"The source list file is used by the ASE and synthesis setup scripts to build the AFU project. It should include the accelerator description file and RTL source files. The file paths are relative to the source list file location.
example sources.txt:
# Paths are relative to sources.txt file\n# Accelerator Descriptor File\nexample_afu.json\n\n# Top level module\nofs_plat_afu.sv\n\n# RTL\nexample_afu.sv\nexample_afu_csr.sv\naccelerator.sv\ndma_engine.sv\n\n# Pointer to software - Information only\n# ../../sw/example_afu.c\n
"},{"location":"n6001/ug_dev_pim_based_afu/#63-directory-structure","title":"6.3. Directory Structure","text":"Below is an example directory structure:
example_afu\n|-- hw\n| |-_ rtl\n| |-- example_afu.json | |-- sources.txt\n| |-- ofs_plat_afu.sv\n| |-- example_afu.sv | |-- example_afu_csr.sv | |-- accelerator.sv | |-- dma_engine.sv\n|-- sw\n |-- example_afu.c\n |-- Makefile\n
"},{"location":"n6001/ug_dev_pim_based_afu/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_docker/","title":"Docker User Guide: Open FPGA Stack: Intel\u00ae Open FPGA Stack","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_docker/#1-introduction","title":"1 Introduction","text":"This document is intended to help you get started in evaluating Open FPGA Stack (Intel\u00ae OFS) using Docker for the Intel\u00ae Platforms. The Intel FPGA platforms can be used as a starting point for evaluation and development. This document covers critical topics related to the initial setup of the Docker solution included with the OFS release.
After reviewing the document, you shall be able to:
- Set up the Intel\u00ae Quartus\u2122 Prime Pro Edition Software in a host server
- Set up the Docker engine
- Build and load your Docker image to the Docker engine
- Run a Docker container with OFS preloaded
The Open FPGA Stack (OFS) Docker image has two main personas:
- Development: You can develop, simulate, and build any component of the OFS. The Docker image enables you to use your laptop or server without having drivers, FPGA Platform, or specific Linux* distribution installed in your host computer. You can follow the development flow provided to run Docker on Linux.
- Deployment: You can program, load binaries, or execute real-time testing using the OPAE and OFS. To do so, the host computer must have the specified software distribution and drivers installed.
"},{"location":"n6001/ug_docker/#12-background-information","title":"1.2 Background Information","text":"A container is a fully functional and portable cloud or non-cloud computing environment that includes an application, associated libraries, and other dependencies. Docker containers do not require a hardware hypervisor, instead using the application layer of the host computer, which means they tend to be smaller, faster to setup, and require fewer resources when compared to a virtual machine (VM).
The OFS provides the flexibility to support various orchestration or management systems, including bare metal, VM, and Docker.
"},{"location":"n6001/ug_docker/#13-relevant-information","title":"1.3 Relevant information","text":" - What is a container?
- Docker vs. Virtual Machines
- Does the Docker container have its own Kernel?
- No, Docker image or Container uses the application layer of the host computer; this functionality is the main reason for docker having lightweight and fast applications.
- Does Docker run on Linux, macOS, and Windows?
- Intel Docker Image can use the PCIe card from the host server?
- Yes, The drivers and additional information could be shared, but this could create potential security concerns (ensure your system is secure).
- Docker security
- Docker subscription
"},{"location":"n6001/ug_docker/#20-prerequisites-and-scope","title":"2.0 Prerequisites and Scope","text":"The OFS release targeting the compatible OFS Platform's is built upon tightly coupled software and firmware versions. Use this section as a general reference for the versions in this release.
The following table highlights the hardware that comprises the Best-Known Configuration (BKC) for the OFS release. For a detailed explanation and safety information regarding the setup go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide. This site walks you through the BIOS configuration changes needed to enable the OFS Platform's.
"},{"location":"n6001/ug_docker/#30-development-installation","title":"3.0 Development Installation","text":"Docker engines have cross-compatibility with multiple systems, but the host server does not require any specific distribution. However, the Quartus\u00ae Prime Pro Edition Version 23.4 requires a specific version. For this guide, Red Hat Linux is used for general instructions.
The OFS Docker image includes all the libraries and tools required by the OFS and OPAE SDK (Python, Perl, CMake, and so on).
"},{"location":"n6001/ug_docker/#31-intel-quartus-prime-software-installation","title":"3.1 Intel Quartus Prime Software Installation","text":"Building AFUs with OFS for Intel Agilex FPGA requires the build machine to have at least 64 GB of RAM.
Go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide for a list of detailed steps for the Quartus\u00ae Prime Pro Edition Version 23.4 installation.
"},{"location":"n6001/ug_docker/#32-docker-engine-installation","title":"3.2 Docker Engine installation","text":""},{"location":"n6001/ug_docker/#rhel-810","title":"RHEL 8.10","text":"The Docker installation steps for RHEL 8.10 are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Also, uninstall Podman and the related dependencies if installed already.
sudo dnf remove docker \\\ndocker-client \\\ndocker-client-latest \\\ndocker-common \\\ndocker-latest \\\ndocker-latest-logrotate \\\ndocker-logrotate \\\ndocker-engine \\\npodman \\\nrunc\n
-
Add the Docker repository to your system:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\n
-
Install the latest version of Docker Engine, containerd, and Docker Compose, or go to the next step to install a specific version.
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"n6001/ug_docker/#ubuntu-2204","title":"Ubuntu 22.04","text":"The Docker installation steps for Ubuntu are the following:
-
Remove old versions; older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
sudo apt-get remove docker docker-engine docker.io containerd runc\n
-
Install packages to allow apt to use a repository
sudo apt-get update\nsudo apt-get install \\\nca-certificates \\\ncurl \\\ngnupg \\\nlsb-release\n
-
Add Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg\n
-
The following command to set up the repository:
echo \\\n\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\n
-
Update the package manager index again:
sudo apt-get update\n
-
Install Docker:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin\n
-
Start the Docker daemon:
sudo systemctl start docker\n
-
Enable the Docker daemon to start on boot:
sudo systemctl enable --now docker\nsudo systemctl enable --now containerd\n
-
Verify that Docker is installed and running:
sudo systemctl status docker\n
You should see a message indicating that the Docker daemon is active and running.
Note: If you want to use Docker as a non-root user, you should add your user to the docker group:
sudo usermod -aG docker your-user\n
You will need to log out and back in for the changes to take effect.
-
Ensure your proxies are setup in case you needed
sudo mkdir -p /etc/systemd/system/docker.service.d nano /etc/systemd/system/docker.service.d/http-proxy.conf\n\n[Service] Environment=\"HTTP_PROXY=http://proxy.example.com:80/\"\nEnvironment=\"HTTPS_PROXY=https://proxy.example.com:443/\"\n#save and close \nsudo systemctl daemon-reload\nsudo systemctl restart docker\n
"},{"location":"n6001/ug_docker/#33-load-docker-image-installation","title":"3.3 Load Docker Image installation","text":"The Dockerfile is released in conjunction with the OFS stack release, and The file needs to be loaded into your host computer to start a docker container.
"},{"location":"n6001/ug_docker/#build-the-image","title":"Build the image","text":" -
You can download the Dockefile from OFS GitHub Docker.
-
Inside the Dockerfile folder, you will find the DockerFile edit and modify the following lines:
ENV no_proxy= #you could use github.com here\nENV http_proxy= #setup proxy\nENV https_proxy= #setup proxy\nENV GITUSER= #setup github user\nENV GITTOKEN= #setup github token\nENV REDUSER= #redhat user \nENV REDPASS= #redhat password\nENV DW_LICENSE_FILE= #DW license\nENV SNPSLMD_LICENSE_FILE= #Synopsys license\nENV LM_LICENSE_FILE= #Quartus License\n
Save the file
-
Create and load the image:
cd Docker_file\ndocker build -t ofs:latest . --no-cache\n
Note: Never remove --no-cache this could cause issues with your environmental variables inside of the container
-
Use the following command to ensure the image is loaded correctly:
sudo docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nofs latest fc80175d13a0 \u221e seconds ago 2.55GB\n
"},{"location":"n6001/ug_docker/#volumen-creation","title":"Volumen creation","text":" -
Docker requires a volume to move data from the host computer (Persistent data) to the docker container and vice versa. To create a docker volume, use the following command:
docker volume create --name DataOFS\n
For more information about Docker volume go here.
Tip: Remember, The docker container has a limited lifecycle; the files and data are lost when the docker is Stopped-> Deleted.
-
Check where the docker volume is mapped in your host server:
docker volume inspect DataOFS\n[\n{\n\"CreatedAt\": \"xxxxxxxxxx\",\n \"Driver\": \"local\",\n \"Labels\": {},\n \"Mountpoint\": \"/var/lib/docker/volumes/DataOFS/_data\",\n \"Name\": \"DataOFS\",\n \"Options\": {},\n \"Scope\": \"local\"\n}\n]\n
-
Inside of your docker container, you can use cp command to copy from your docker to your host:
cp /atmydocker/myfile.txt /dataofs\n
The docker container path is /dataofs the host path is /var/lib/docker/volumes/DataOFS/_data.
"},{"location":"n6001/ug_docker/#34-create-a-container","title":"3.4 Create a container","text":"Now you are ready to start the container, and you should be prepared to run it: 1. First, Let's create the template for the run command, copy your Quartus installation path and paste it under -v (Don't Run the command yet):
docker run --rm -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
- Using the previous example now, you can execute the docker run command.
docker run --rm -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
-
Now the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nbdc1289fb081 ofs:latest \"/bin/bash\" 46 seconds ago Up 45 seconds myOFS\n
Your Container ID is bdc1289fb081.
"},{"location":"n6001/ug_docker/#35-evaluate-ofs-container","title":"3.5 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@bdc1289fb081 /]#\n
The container id is shown when you are in interactive mode [root@bdc1289fb081 /]#.
-
Now verify the variables and Quartus is appropriately set up and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion Quartus Prime Pro Version 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Tip: If you need to de-attach without stopping the container, you can use Ctrl+P or Ctrl+Q. For custom combinations, for example, docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the evaluation script.
-
Let's use option 2 - Check versions of Operating System and Quartus Premier Design Suite (QPDS); remember multiple options could not be available if the DFL drivers and the FPGA Platform is not installed, This example uses the Intel\u00ae FPGA SmartNIC N6001-PL .
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 2\nGo to selection: 2\n###########################################################################################\n#################### Check versions of Operation System, Quartus ##########################\n###########################################################################################\nChecking Linux release\nLinux version 4.18.0-dfl .....\n\n....\n\ncycle complete exiting...\n
-
The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n
Now you can control and compile the design. You can use the interactive or de-attach mode. -
If you need to save the log file and output files use the following command
sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -e\n
all the files are saved under the share volume, DataOFS , /var/lib/docker/volumes/DataOFS/_data
"},{"location":"n6001/ug_docker/#40-deployment","title":"4.0 Deployment","text":"The OFS docker image allows you to connect with your FPGA Platform. The main difference from the development installation process is that you are able to test with real hardware, but you must have a specific requirement to have a fully compatible system.
Information related to host setup please go to Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
"},{"location":"n6001/ug_docker/#41-installation-of-deployment-server","title":"4.1 Installation of Deployment server","text":"Once you ensure the DFL drivers are installed, follow the below steps:
- Follow the steps listed in sections 2.1 to 2.3
- 2.1 Quartus installation
- 2.2 Docker Engine installation
- 2.3 Load Docker Image installation
- The steps required for DFL driver installation are documented Open FPGA Stack (OFS) Collateral Site select your desired platform and select Getting stated guide.
Now you should have all the steps required, and you can run the docker image directly.
"},{"location":"n6001/ug_docker/#42-create-a-container","title":"4.2 Create a container","text":"Now you are ready to start the container, and should be prepared to run it (Note: now we are adding a new flag to allow us to access the PCIe devices \u201c\u2014privileged\u201d) :
-
First, copy your Quartus installation path and paste it under -v:
docker run --rm --privileged -itd --name myOFS -v=<yourintallationfolder>:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n
Example, my Quartus installation is located at \"/home/intelFPGA_pro/24.1\" as a result, my command should be
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\nbdc1289fb0813bb325b55dd11df4eeec252143d6745a6e5772638fbc107d0949\n
Tip: you can change myOFS with any other value. The value is the given name of the container.
Important: The --privileged flag gives all capabilities to the container. When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
:warning: Only use --privileged under development infrastructure, never in production!
-
Execute the docker run command.
docker run --rm --privileged -itd --name myOFS -v=/home/intelFPGA_pro/24.1:/home/intelFPGA_pro/:ro -v=DataOFS:/dataofs ofs:latest /bin/bash\n25b41eb4d232de9c750b52ddc6b92a3db612200e5993f55733b59068898623d7\n
-
Now, the docker container should be available.
# sudo docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n25b41eb4d232 ofs:latest \"/bin/bash\" 13 seconds ago Up 12 seconds myOFS\n
\u200b Your Container ID is 25b41eb4d232.
"},{"location":"n6001/ug_docker/#43-evaluate-ofs-container","title":"4.3 Evaluate OFS container","text":"The OFS container has two possible ways to interact with the container:
-
Interactive mode:
This mode it takes you straight inside the container and uses the command terminal as a regular Linux console.
-
Enable the interactive mode:
docker attach myOFS\n[root@25b41eb4d232 /]#\n
The container id is shown when you are in interactive mode [root@25b41eb4d232 /]#.
-
Now verify the variables and Quartus is appropriately setup and recognized:
quartus_syn --version\n\nQuartus Prime Synthesis\nVersion 24.1\n
-
Everything is set up correctly. Please go to the following link for more information related to the Open FPGA Stack (OFS) Collateral Site select your desired platform and select User Guide, Technical Reference Manual, Developer Guide, or Getting Started Guide.
Tip: If you need to de-attach without stopping the container you can use Ctrl+P or Ctrl+Q. For custom, combinations use for example docker attach --detach-keys=\"ctrl-a\" myOFS and if you press CTRL+A you will exit the container, without killing it.
-
De-attach Mode:
This mode runs your container in the background and allows you to run multiple commands without going inside of the docker container.
-
The OFS Docker image already includes the eval script.
-
Run the script and make a selection, you can directly execute with the following command:
Let's use option 3 - Identify Platform Hardware via PCIe; remember the DFL drivers need be installed.
$ sudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs-agx7-pcie-attach_eval.sh 3\nGo to selection: 3\nPCIe card detected as\n\nb1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)\nb1:00.1 Processing accelerators: Intel Corporation Device bcce\nb1:00.2 Processing accelerators: Intel Corporation Device bcce\nb1:00.4 Processing accelerators: Intel Corporation Device bcce\n\nHost Server is connected to SINGLE card configuration\n\ncycle complete exiting...\n
- The Intel Docker image includes the script ofs_extratool.sh to allow you to change the seed value.
```sh\nsudo docker exec -it myOFS /home/OFS_BUILD_ROOT/ofs_extratool.sh -s 5\n```\n
Now you can control and compile the design using the interactive or de-attach mode.
"},{"location":"n6001/ug_docker/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"n6001/ug_kvm/","title":"Virtual machine User Guide: Open FPGA Stack + KVM","text":"Last updated: November 19, 2024
"},{"location":"n6001/ug_kvm/#document-scope","title":"Document scope","text":"The document describes setting up and configuring a virtual machine to use PCIe devices. Here are the steps that the document may include:
- Install the necessary tools, such as virt-manager, on the host machine. This may involve downloading and installing the software from the internet.
- Enable the virtualization feature on the host machine. This may involve going into the BIOS settings and enabling hardware-assisted virtualization or using a command-line tool to enable it in the operating system.
- Use virt-manager to create a new virtual machine and configure its settings. This may involve choosing a name and operating system for the virtual machine and setting the amount of memory and storage it will use.
- Install the OPAE (Open Programmable Acceleration Engine) tool on the virtual machine. This may involve downloading and installing the OPAE software.
- Install the DFL (Data Field Level) drivers on the virtual machine. These drivers allow the virtual machine to access and use the PCIe devices on the host machine. This may involve downloading and installing the drivers from the internet.
- Once all of the steps have been completed, you should be able to use the virtual machine to access and use the PCIe devices on the host machine. You may need to configure the virtual machine's settings to enable it to use the PCIe devices, such as by assigning a specific device to the virtual machine.
"},{"location":"n6001/ug_kvm/#1-modes-of-operation","title":"1. Modes of Operation","text":"Our current operational framework stipulates two distinct modes of operation for PF/VF configurations. When using a 2 PF enabled FIM design, both the workload and management ports can be interchangeably passed through to a VM or run on bare-metal.
-
Management Mode: This mode necessitates the passthrough of only the FME device (use fpgainfo fme to discover your port number, normally .0). The reason for this is that the Open FPGA Stack (OFS) depends on this address for management. Under this mode, the use of the exerciser and virtual functions is not feasible.
-
Virtual Function Mode: This mode comes into effect when a user needs to utilize the Virtual Functions (VF). The user will convert (example) Physical Function 0 (PF0) to three Virtual Functions (VF). This means the PF will cease to function for management purposes. Once the VFs are set up, they essentially take over the role of the PF in communicating with the Virtual Machines (VMs).
However, this mode is subject to a limitation. If the user needs to execute 'fpgainfo fme' or 'fpgaupdate', they will need to transition from Virtual Function Mode to Management Mode. Conversely, if the user intends to utilize the Virtual Functions, they would need to switch from Management Mode to Virtual Function Mode. It is imperative to bear this limitation in mind when operating within these modes.
"},{"location":"n6001/ug_kvm/#2-enable-virtualization","title":"2. Enable Virtualization","text":"To check if virtualization is enabled on a Red Hat system using lscpu and grep, you can use the following command:
lscpu -e | grep Virtualization\n
This command will run lscpu with the -e or --extended option, which displays information about the CPU and its available virtualization capabilities. Then, it pipes the output to grep with the search pattern \"Virtualization\". If the system has support for virtualization, the output will show the \"Virtualization\" field and its value, for example:
Virtualization: VT-x\n
In this example, the output shows that the system supports Intel VT-x virtualization technology. If the \"Virtualization\" field is empty, the system does not have support for virtualization. Keep in mind that even if the system has support for virtualization, it may not be enabled in the BIOS or the operating system itself.
Check the following for the bios configuration, Enabling Intel VT-d Technology
"},{"location":"n6001/ug_kvm/#3-verify-environment-setup","title":"3. Verify Environment Setup","text":" - Open a terminal window and log in as a user with sudo privileges.
- Check if the virtualization kernel modules are loaded by running the following command:
lsmod | grep kvm\n
-
If the command outputs a list of modules, the virtualization kernel modules are loaded, and virtualization is enabled on your system.
-
The virtualization kernel modules are not loaded if the command does not output anything. You can try loading them manually by running the following command:
sudo modprobe kvm\n
- If the kernel modules are not loaded, and you cannot load them manually, it may be because virtualization is not supported or enabled in your system's BIOS or UEFI settings. You must reboot your system and enter the BIOS or UEFI settings menu to enable virtualization. The exact steps for doing this may vary depending on your system's hardware and BIOS/UEFI version, so consult your motherboard or system documentation for specific instructions.
"},{"location":"n6001/ug_kvm/#4-install-virtual-machine-manager","title":"4. Install Virtual Machine Manager","text":"Virtual Machine Manager (also known as libvirt) can be installed by following the below steps:
- Open a terminal window and log in as a user with sudo privileges.
-
Update your system package index by running the following command:
- Redhat
sudo dnf update\n
* Ubuntu sudo apt update\n
-
Install the libvirt package and any required dependencies by running the following command:
- Redhat
sudo dnf install @virtualization\n
- Ubuntu
sudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager\n
-
Start the libvirtd service and enable it to start automatically at boot time by running the following commands:
sudo systemctl start libvirtd\nsudo systemctl enable libvirtd\n
- Optional: Install the virt-manager package, which provides a GUI application for managing virtual machines, by running the following command:
sudo dnf install virt-manager\n
- Optional: If you want to be able to run virtual machines as a non-root user, add your user to the libvirt group by running the following command, replacing \"USERNAME\" with your username:
sudo usermod -a -G libvirt USERNAME\n
- You can now launch virt-manager by running the command
virt-manager as the non-root user.
Note: By default, virt-manager will only allow non-root users to create and manage virtual machines with limited resources, such as a limited amount of memory and CPU cores. To allow non-root users to create and manage virtual machines with more resources, you need to edit the /etc/libvirt/qemu.conf configuration file and set the user and group values for the dynamic_ownership option to 1. For example:
# Set user and group ownership of dynamic /dev/kvm device nodes\ndynamic_ownership = 1\nuser = \"root\"\ngroup = \"root\"\n
You will also need to restart the libvirtd service for the changes to take effect. You can do this by running the command.
sudo systemctl restart libvirtd\n
- Reboot your server to apply the changes
reboot\n
After completing these steps, you should be able to use the virt-manager GUI application to manage virtual machines on your system.
"},{"location":"n6001/ug_kvm/#5-create-a-vm-using-virt-manager","title":"5. Create a VM Using Virt-Manager","text":"Before creating the virtual machine, ensure the DFL drivers are installed in your host machine. For your chosen OFS platform, refer to the Getting Stared User Guide and associated Software Installation Guidelines for a complete overview of the software installation process. The following are the current available getting started guides for OFS enabled platforms:
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (Intel\u00ae FPGA SmartNIC N6001-PL/N6000-PL)
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (F-Series Development Kit (2xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 PCIe Attach FPGAs (I-Series Development Kit (2xR-Tile, 1xF-Tile))
- Getting Started Guide: OFS for Agilex\u00ae 7 SoC Attach FPGAs
- Getting Started Guide: OFS for Stratix 10\u00ae FPGA PCIe Attach FPGAs
To create a Red Hat 8.10 or Ubuntu 22.04 virtual machine (VM) using virt-manager and share PCI devices with the VM, you will need to perform the following steps:
- Start the
virt-manager GUI by running the following command:
sudo virt-manager&\n
-
Create a new connection from the menu File-> \"Add Connection,\" Use the default options and click \"Connect.\"
-
In the virt-manager window, click the \"New virtual machine\" button.
-
In the \"New VM\" wizard, select \"Local install media (ISO image or CDROM)\" as the installation source, and then click \"Forward.\"
-
Get the Red Hat image from the following link.
https://developers.redhat.com/content-gateway/file/rhel/Red_Hat_Enterprise_Linux_8.10/rhel-8.10-x86_64-dvd.iso
-
Get the Ubuntu image from the following link.
https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso
-
In the next step, Click Browse -> Browse local, select the Red Hat 8.10 ISO image as the installation source and click \"Forward\".
Note: if the system is not detected, disable \"Automatic detected from the installation media/source\" and type ubuntu and select 19.10 (this should be fine for the 22.04); this step is necessary to copy the default values for the specific OS
-
In the next step, specify a name and location for the VM, and select the desired memory and CPU configuration. in our case, 16 cores and 64 GB of RAM; Click \"Forward\" to continue.
-
Select \"enable storage for this virtual machine,\" Select \"Create a new disk for the virtual machine,\" and enter a size for the virtual disk (at least 200~300GB in case you need to compile the design) or create a custom storage.
-
If you need to create custom storage, select \"Select or Create custom storage\" and click \"Manage.\"
-
Click on the \"+\" icon (Bottom left) to create the storage pool.
-
In the \"Create a new storage pool\" dialog, enter a name for the storage pool and select the type of storage pool you want to create; select the Target Path and Click \"Finish.\"
-
Select the pool and later click on the \"+\" icon (The Icon is on the right side of the Volume label) to create the New Storage Volume.
-
In the \"Create Storage Volume\" dialog, Define the name and format (keep with the default qcow2) and select the Max Capacity (at least 200~300GB in case you need to compile the design); click \"Finish\" to create the disk.
-
Once the disk is created, it will appear in your virtual machine's list of storage devices. You can now use this disk just like any other disk. Select from the list and Click \"Choose Volume.\"
-
In the next step, select the \"Customize configuration before install\" option and click \"Finish.\"
"},{"location":"n6001/ug_kvm/#51-passing-devices-to-the-vm","title":"5.1 Passing Devices to the VM","text":"In the \"Overview\" tab, select \"Add Hardware,\" choose \"PCI Host Device\" from the drop-down menu and choose the PCI device you want to share with the VM. Click \"Apply\" to apply the changes, and then click \"Finish\" to create the VM.
Depending on the FIM currently loaded onto your FPGA device, you have access to a few modes of operation. Management Mode and Deployment mode can be used on any FIM that supports a PF/VF split architecture. When using the 2 PF FIM, see 2 PF Mode.
"},{"location":"n6001/ug_kvm/#511-management-mode","title":"5.1.1 Management Mode","text":"This will only allow you to load the binaries to the FPGA, you only need to add the PF listed at the fpgainfo fme command.
fpgainfo fme\n\nfpgainfo fme\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxxx Board Management Controller Build version: xxxx //****** FME ******//\nObject Id : 0xEE00000\nPCIe s:b:d.f : 0000:b1:00.0\n
\u200b
"},{"location":"n6001/ug_kvm/#512-deployment-mode","title":"5.1.2 Deployment Mode","text":"The main idea of this mode is enable the Virtual function used by the Agilex PCIe Attach OFS under the Physical Function 0, This option will allow us to use the Host Exercises.
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Create 3 VFs in the PR region.
sudo pci_device b1:00.0 vf 3
-
Verify all 3 VFs were created.
lspci -s b1:00 b1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01) b1:00.1 Processing accelerators: Intel Corporation Device bcce b1:00.2 Processing accelerators: Intel Corporation Device bcce b1:00.3 Processing accelerators: Red Hat, Inc. Virtio network device b1:00.4 Processing accelerators: Intel Corporation Device bcce b1:00.5 Processing accelerators: Intel Corporation Device bccf b1:00.6 Processing accelerators: Intel Corporation Device bccf b1:00.7 Processing accelerators: Intel Corporation Device bccf
-
Bind all of the PF/VF endpoints to the vfio-pci driver.
sudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to DCPsupport\nChanging permissions for /dev/vfio/187 to rw-rw----\n\nsudo opae.io init -d 0000:b1:00.2 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 188\nAssigning /dev/vfio/188 to DCPsupport\nChanging permissions for /dev/vfio/188 to rw-rw----\n\n...\n\nsudo opae.io init -d 0000:b1:00.7 user:user\nBinding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 319\nAssigning /dev/vfio/319 to DCPsupport\nChanging permissions for /dev/vfio/319 to rw-rw----\n
-
Check that the accelerators are present using fpgainfo. Note your port configuration may differ from the below.
sudo fpgainfo port //****** PORT ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:B1:00.0\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x00\n//****** PORT ******//\nObject Id : 0xE0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.7\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 4dadea34-2c78-48cb-a3dc-5b831f5cecbb\n//****** PORT ******//\nObject Id : 0xC0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.6\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 823c334c-98bf-11ea-bb37-0242ac130002\n//****** PORT ******//\nObject Id : 0xA0B1000000000000\nPCIe s:b:d.f : 0000:B1:00.5\nVendor Id : 0x8086\nDevice Id : 0xBCCF\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 8568ab4e-6ba5-4616-bb65-2a578330a8eb\n//****** PORT ******//\nObject Id : 0x80B1000000000000\nPCIe s:b:d.f : 0000:B1:00.4\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 44bfc10d-b42a-44e5-bd42-57dc93ea7f91\n//****** PORT ******//\nObject Id : 0x40B1000000000000\nPCIe s:b:d.f : 0000:B1:00.2\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n//****** PORT ******//\nObject Id : 0x20B1000000000000\nPCIe s:b:d.f : 0000:B1:00.1\nVendor Id : 0x8086\nDevice Id : 0xBCCE\nSubVendor Id : 0x8086\nSubDevice Id : 0x1771\nSocket Id : 0x01\nAccelerator GUID : 3e7b60a0-df2d-4850-aa31-f54a3e403501\n
The following table contains a mapping between each VF, Accelerator GUID, and component.
"},{"location":"n6001/ug_kvm/#table-16-accelerator-pfvf-and-guid-mappings","title":"Table 16: Accelerator PF/VF and GUID Mappings","text":"Component VF Accelerator GUID Intel N6001-PL FPGA SmartNIC Platform base PF XXXX:XX:XX.0 N/A VirtIO Stub XXXX:XX:XX.1 3e7b60a0-df2d-4850-aa31-f54a3e403501 HE-MEM Stub XXXX:XX:XX.2 56e203e9-864f-49a7-b94b-12284c31e02b Copy Engine XXXX:XX:XX.4 44bfc10d-b42a-44e5-bd42-57dc93ea7f91 HE-MEM XXXX:XX:XX.5 8568ab4e-6ba5-4616-bb65-2a578330a8eb HE-HSSI XXXX:XX:XX.6 823c334c-98bf-11ea-bb37-0242ac130002 MEM-TG XXXX:XX:XX.7 4dadea34-2c78-48cb-a3dc-5b831f5cecbb -
Ensure you add the desired VF in your PCIE devices list.
"},{"location":"n6001/ug_kvm/#513-2-pf-mode","title":"5.1.3 2 PF Mode","text":"For FIMs that support the dual PF architecture, you have the option to pass through any number of PFs into the VM. The VM's software will recognize any management / workload ports and probe them appropriately. This assumes you have the OPAE SDK and Linux DFL drivers installed on both the VM and host.
-
Bind all endpoints you wish to pass through to the VM to the vfio-pci driver on the host.
sudo opae.io init -d 0000:b1:00.0 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\nsudo opae.io init -d 0000:b1:00.1 user:user\nUnbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci\nBinding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci\niommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 187\nAssigning /dev/vfio/187 to user\nChanging permissions for /dev/vfio/187 to rw-rw----\n
-
Pass through any required hardware endpoints, select \"Add Hardware\" -> \"PCI Host Device\".
-
Run the following command on the host and VM to allocate hugepages for workload testing:
echo 4194304 | sudo tee /sys/module/vfio_iommu_type1/parameters/dma_entry_limit\n
"},{"location":"n6001/ug_kvm/#52-virt-manager-configuration-changes","title":"5.2 Virt-Manager Configuration Changes","text":" -
Edit the XML file for your machine and include the following
-
< ioapic driver='qemu'/> inside of features:
<features>\n<acpi/>\n<apic/>\n<ioapic driver='qemu'/>\n</features>\n
-
Inside of devices
<devices>\n........\n ......\n <iommu model='intel'>\n<driver intremap='on' caching_mode='on'/>\n</iommu>\n</devices>\n
-
Ensure the hard limit is setup correctly otherwise you can only pass one device:
<memtune>\n<hard_limit unit='G'>64</hard_limit>\n</memtune>\n
Note: assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices)
-
Save the changes \"Apply\"
-
On the host machine append intel_iommu=on to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.
nano /etc/default/grub\n......\nGRUB_CMDLINE_LINUX=\"....... ... intel_iommu=on\"\n...\n#Refresh the grub.cfg file for these changes to take effect\ngrub2-mkconfig -o /boot/grub2/grub.cfg\nshutdown -r now\n
-
Ensure your devices are enumerated properly.
-
Example in you host system should look like this:
1. Management Mode:
B1:00.0\n
2. Deployment Mode:
B1:00.5\n
-
Under the virtual machine (The PCIe Address is an example you could get a different number):
1. Management Mode:
177:00.0\n
2. Deployment Mode:
177:00.0\n
-
Click on \"Begin Installation.\" and follow the wizard installation of the OS.
-
Once the VM is created, you can start it by selecting it in the virt-manager window and clicking the \"Run\" button. This will boot the VM and start the Red Hat 8.10/Ubuntu installation process. Follow the on-screen instructions to complete the installation.
-
Under your virtual machine, configure your VM proxy:
- Redhat How to apply a system-wide proxy?
- Ubuntu Define proxy settings
- Configure Git to use a proxy
-
To include OPAE in your virtual machine, follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide. To install the DFL drivers, please follow the instructions from the following link Open FPGA Stack (OFS) Collateral Site select your desired platform from the OFS Software tab and select Software Installation Guide.
-
Use the OPAE SDK tool opae.io (under your virtual machine) to check default driver binding using your card under test PCIe B:D.F (Management mode).
sudo fpgainfo fme\n\nIntel Acceleration Development Platform N6001\nBoard Management Controller NIOS FW version: xxx Board Management Controller Build version: xxx\n//****** FME ******//\nObject Id : 0xEC00000\nPCIe s:b:d.f : 0000:177:00.0\n
-
Use the Virtual function (Not supported at management mode)
-
Ensure the DFL kernel drivers is install in your VM system
-
Bind VFs to VFIO driver
$ sudo opae.io init -d 177:00.0\n[sudo] password for rhel8_10:\nBinding (0x8086,0xbccf) at 0000:177:00.0 to vfio-pci\niommu group for (0x8086,0xbccf) at 0000:177:00.0 is 24\n
-
Verify the binding is correct.
$ opae.io ls\n[0000:177:00.0] (0x8086:0xbccf 0x8086:0x1771) Intel Acceleration Development Platform N6001 (Driver: vfio-pci)\n
-
Test the HE mem
host_exerciser mem\n starting test run, count of 1\nAPI version: 2\nBus width: 64 bytes\nLocal memory bus width: 64 bytes\nAFU clock: 470 MHz\nAllocate SRC Buffer\nAllocate DST Buffer\nAllocate DSM Buffer\n Host Exerciser Performance Counter:\n Host Exerciser numReads: 1024\nHost Exerciser numWrites: 1025\nHost Exerciser numPendReads: 0\nHost Exerciser numPendWrites: 0\nHost Exerciser numPendEmifReads: 0\nHost Exerciser numPendEmifWrites: 0\nNumber of clocks: 8276\nTotal number of Reads sent: 1024\nTotal number of Writes sent: 1024\nBandwidth: 3.722 GB/s\n Test mem(1): PASS\n
After the installation, you can use virt-manager to manage and configure the VM to move from Management mode to Deployment or vice versa, including setting up networking, attaching additional storage, and installing additional software. The shared PCI device will be available to the VM, allowing it to use it as if it were connected directly to the physical system.
"},{"location":"n6001/ug_kvm/#notices-disclaimers","title":"Notices & Disclaimers","text":"Intel\u00ae technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. \u00a9 Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group\u2122.
"},{"location":"sw/build_chain/fpga_api/api_build/","title":"Building OPAE SDK Artifacts","text":""},{"location":"sw/build_chain/fpga_api/api_build/#steps","title":"Steps","text":" - Fetch the OPAE SDK source tree
- Configure the OPAE SDK CMake project
- Build OPAE SDK targets
The example below lists commands that can be used to fetch and build OPAE SDK.
# fetch the source\ngit clone https://github.com/OPAE/opae-sdk.git\ncd opae-sdk\n# configure CMake\ncmake ..\n# build\nmake\n
For a list of targets that can be built, type make help from the build directory.
CMake options that may be set during the configuration include the following:
|----------------------------|-----------------------|-------------------------------------|---------------------------------------|----------------|\n| cmake flag | Optional or Mandatory | Purpose | Valid values | Default value |\n|----------------------------|-----------------------|-------------------------------------|---------------------------------------|----------------|\n| -DCMAKE_BUILD_TYPE | Optional | Set compiler flags | Debug/Release/Coverage/RelWithDebInfo | RelWithDebInfo |\n| -DOPAE_BUILD_LEGACY | Optional | Enable/disable opae-legacy.git | ON/OFF | OFF |\n| -DOPAE_BUILD_SPHINX_DOC | Optional | Enable/disable documentation build | ON/OFF | OFF |\n| -DOPAE_BUILD_TESTS | Optional | Enable/disable building unit tests | ON/OFF | OFF |\n| -DOPAE_INSTALL_RPATH | Optional | Enable/disable rpath for install | ON/OFF | OFF |\n| -DOPAE_BUILD_LIBOPAE_CXX | Optional | Enable/disable OPAE C++ bindings | ON/OFF | ON | \n| -DOPAE_WITH_PYBIND11 | Optional | Enable/disable pybind11 binaries | ON/OFF | ON |\n| -DOPAE_BUILD_PYTHON_DIST | Optional | Enable/disable Python Distribution | ON/OFF | OFF |\n| -DOPAE_ENABLE_MOCK | Optional | Enable/disable mocks for unit tests | ON/OFF | OFF |\n| -DOPAE_BUILD_SIM | Optional | Enable/disable opae-sim.git | ON/OFF | OFF |\n
"},{"location":"sw/build_chain/fpga_driver/driver_build/","title":"Building the OPAE Intel FPGA driver (out-of-tree)","text":"The Intel FPGA driver included with OPAE SDK releases is packaged as an RPM or DEB package as well as a source tarball. Starting with OPAE SDK release of 1.4, the driver can be built from source out-of-tree but requires the following packages:
For RPM package managers (Red Hat, CentOS, Fedora, etc.) * kernel-headers * kernel-devel * gcc * make
For DEB package managers (Debian, Ubuntu, etc.) * kernel-headers-generic * gcc * make
After installation of necessary distribution packages, follow the steps in the example below to build the Intel Kernel driver. NOTE The example below references Intel FPGA Kernel driver version 2.0.2. but can be applied to later versions.
tar zxf opae-intel-fpga-driver-2.0.2-1.tar.gz\ncd opae-intel-fpga-driver-2.0.2\nmake\n
"},{"location":"sw/drv_arch/drv_arch/","title":"Open Programmable Accelerator Engine (OPAE) Linux Device Driver Architecture","text":"The OPAE FPGA Linux Device Driver provides interfaces for user-space applications to configure, enumerate, open, and access FPGA accelerators on platforms equipped with Intel FPGA solutions. The OPAE FPGA driver also enables system-level management functions such as FPGA reconfiguration and virtualization.
"},{"location":"sw/drv_arch/drv_arch/#hardware-architecture","title":"Hardware Architecture","text":"The Linux Operating System treats the FPGA hardware as a PCIe* device. A predefined data structure, Device Feature List (DFL), allows for dynamic feature discovery in an Intel FPGA solution.
The Linux Device Driver implements PCIe Single Root I/O Virtualization (SR-IOV) for the creation of Virtual Functions (VFs). The device driver can release individual accelerators for assignment to virtual machines (VMs).
"},{"location":"sw/drv_arch/drv_arch/#fpga-management-engine-fme","title":"FPGA Management Engine (FME)","text":"The FPGA Management Engine provides error reporting, reconfiguration, performance reporting, and other infrastructure functions. Each FPGA has one FME which is always accessed through the Physical Function (PF). The Intel Xeon\u00ae Processor with Integrated FPGA also performs power and thermal management. These functions are not available on the Intel Programmable Acceleration Card (PAC).
User-space applications can acquire exclusive access to the FME using open(), and release it using close(). Device access may be managed by standard Linux interfaces and tools.
If an application terminates without freeing the FME or Port resources, Linux closes all file descriptors owned by the terminating process, freeing those resources.
"},{"location":"sw/drv_arch/drv_arch/#port","title":"Port","text":"A Port represents the interface between two components: * The FPGA Interface Manager (FIM) which is part of the static FPGA fabric * The Accelerator Function Unit (AFU) which is the partially reconfigurable region
The Port controls the communication from software to the AFU and makes features such as reset and debug available.
"},{"location":"sw/drv_arch/drv_arch/#accelerator-function-unit-afu","title":"Accelerator Function Unit (AFU)","text":"An AFU attaches to a Port. The AFU provides a 256 KB memory mapped I/O (MMIO) region for accelerator-specific control registers.
- Use
open() on the Port device to acquire access to an AFU associated with the Port device. - Use
close()on the Port device to release the AFU associated with the Port device. - Use
mmap() on the Port device to map accelerator MMIO regions.
"},{"location":"sw/drv_arch/drv_arch/#partial-reconfiguration-pr","title":"Partial Reconfiguration (PR)","text":"Use PR to reconfigure an AFU from a bitstream file. Successful reconfiguration has the following requirement:
- You must generate the reconfiguration AFU for the exact FIM. The AFU and FIM are compatible if their interface IDs match. You can verify this match by comparing the interface ID in the bitstream header against the interface ID that is exported by the driver in sysfs.
In all other cases PR fails and may cause system instability.
Platforms that support 512-bit Partial Reconfiguration require binutils >= version 2.25.
Close any software programs accessing the FPGA, including those running in a virtualized host before initiating PR. For virtualized environments, the recommended sequence is as follows:
- Unload the driver from the guest
- Release the VF from the guest
Releasing the VF from the guest while an application on the guest is still accessing its resources may lead to VM instabilities. We recommend closing all applications accessing the VF in the guest before releasing the VF.
- Disable SR-IOV
- Perform PR
- Enable SR-IOV
- Assign the VF to the guest
- Load the driver in the guest
"},{"location":"sw/drv_arch/drv_arch/#fpga-virtualization","title":"FPGA Virtualization","text":"To enable accelerator access from applications running on a VM, create a VF for the port using the following process:
-
Release the Port from the PF using the associated ioctl on the FME device.
-
Use the following command to enable SR-IOV and VFs. Each VF can own a single Port with an AFU. In the following command, N is the number of Port released from the PF.
echo N > $PCI_DEVICE_PATH/sriov_numvfs\n
The number, 'N', cannot be greater than the number of supported VFs. This can be read from $PCI_DEVICE_PATH/sriov_totalvfs.
-
Pass the VFs through to VMs using hypervisor interfaces.
-
Access the AFU on a VF from applications running on the VM using the same driver inside the VM.
Creating VFs is only supported for port devices. Consequently, PR and other management functions are only available through the PF.
If assigning multiple devices to the same VM on a guest IOMMU, you may need to increase the hard_limit option in order to avoid hitting a limit of pinned memory. The hard limit should be more than (VM memory size x Number of PCIe devices).
"},{"location":"sw/drv_arch/drv_arch/#driver-organization","title":"Driver Organization","text":""},{"location":"sw/drv_arch/drv_arch/#pcie-module-device-driver","title":"PCIe Module Device Driver","text":"FPGA devices appear as a PCIe devices. Once enumeration detects a PCIe PF or VF, the Linux OS loads the FPGA PCIe device driver. The device driver performs the following functions:
- Walks through the Device Feature List in PCIe device base address register (BAR) memory to discover features and their sub-features and creates necessary platform devices.
- Enables SR-IOV.
- Introduces the feature device infrastructure, which abstracts operations for sub-features and provides common functions to feature device drivers.
"},{"location":"sw/drv_arch/drv_arch/#pcie-module-device-driver-functions","title":"PCIe Module Device Driver Functions","text":"The PCIe Module Device Driver performs the following functions:
- PCIe discovery, device enumeration, and feature discovery.
- Creates sysfs directories for the device, FME, and Port.
- Creates the platform driver instances, causing the Linux kernel to load their respective drivers.
"},{"location":"sw/drv_arch/drv_arch/#fme-platform-module-device-driver","title":"FME Platform Module Device Driver","text":"The FME Platform Module Device Driver loads automatically after the PCIe driver creates the FME Platform Module. It provides the following features for FPGA management:
-
Power and thermal management, error reporting, performance reporting, and other infrastructure functions. You can access these functions via sysfs interfaces the FME driver provides.
-
Partial Reconfiguration. During PR sub-feature initialization, the FME driver registers the FPGA Manager framework to support PR. When the FME receives the relevant ioctl request from user-space, it invokes the common interface function from the FPGA Manager to reconfigure the AFU using PR.
-
Port management for virtualization (releasing/assigning port device).
After a port device is released, you can use the PCIe driver SR-IOV interfaces to create/destroy VFs.
For more information, refer to \"FPGA Virtualization\".
"},{"location":"sw/drv_arch/drv_arch/#fme-platform-module-device-driver-functions","title":"FME Platform Module Device Driver Functions","text":"The FME Platform Module Device Driver performs the the following functions:
- Creates the FME character device node.
- Creates the FME sysfs files and implements the FME sysfs file accessors.
- Implements the FME private feature sub-drivers.
- FME private feature sub-drivers: * FME Header * Partial Reconfiguration * Global Error * Global Performance
"},{"location":"sw/drv_arch/drv_arch/#port-platform-module-device-driver","title":"Port Platform Module Device Driver","text":"After the PCIe Module Device Driver creates the Port Platform Module device, the FPGA Port and AFU driver are loaded. This module provides an interface for user-space applications to access the individual accelerators, including basic reset control on the Port, AFU MMIO region export, DMA buffer mapping service, and remote debug functions.
"},{"location":"sw/drv_arch/drv_arch/#port-platform-module-device-driver-functions","title":"Port Platform Module Device Driver Functions","text":"The Port Platform Module Device Driver performs the the following functions:
- Creates the Port character device node.
- Creates the Port sysfs files and implements the Port sysfs file accessors.
- Implements the following Port private feature sub-drivers. * Port Header * AFU * Port Error * Signal Tap
"},{"location":"sw/drv_arch/drv_arch/#opae-fpga-driver-interface","title":"OPAE FPGA Driver Interface","text":"The user-space interface consists of a sysfs hierarchy and ioctl requests. Most kernel attributes can be accessed/modified via sysfs nodes in this hierarchy. More complex I/O operations are controlled via ioctl requests. The OPAE API implementation, libopae-c, has been designed to use this interface to interact with the OPAE FPGA kernel drivers.
"},{"location":"sw/fpga_api/fpga_api/","title":"OPAE C API Reference","text":"The reference documentation for the OPAE C API is grouped into the following sections:
- Types
- types.h
- types_enum.h
- Enumeration API
- enum.h
- properties.h
- Access API
- access.h
- Event API
- event.h
- MMIO and Shared Memory APIs
- mmio.h
- buffer.h
- umsg.h
- Management API
- manage.h
- Metrics API
- metrics.h
- SysObject
- sysobject.h
- Utilities
- utils.h
- Samples
- hello_fpga.c
- hello_events.c
"},{"location":"sw/fpga_api/fpga_api/#types","title":"Types","text":"The OPAE C API defines a number of types; most prominent are the types fpga_token, fpga_handle, and fpga_properties. All regular types are defined in types.h, while the values of enumeration types are defined in types_enum.h.
"},{"location":"sw/fpga_api/fpga_api/#typesh","title":"types.h","text":"types.h
"},{"location":"sw/fpga_api/fpga_api/#types_enumh","title":"types_enum.h","text":"types_enum.h
"},{"location":"sw/fpga_api/fpga_api/#enumeration-api","title":"Enumeration API","text":"The OPAE enumeration API allows selective discovery of FPGA resources. When enumerating resources, a list of filter criteria can be passed to the respective function to select a subset of all resources in the system. The fpgaEnumerate() function itself then returns a list of fpga_tokens denoting resources, which can be used in subsequent API calls.
Filter criteria are specified using one or more fpga_properties object. These objects need to be created using fpgaGetProperties() (defined in ) before being passed to fpgaEnumerate(). Individual attributes of an fpga_properties object are set using specific accessors, which are also defined in ."},{"location":"sw/fpga_api/fpga_api/#enumh","title":"enum.h","text":"
enum.h
"},{"location":"sw/fpga_api/fpga_api/#propertiesh","title":"properties.h","text":"properties.h
"},{"location":"sw/fpga_api/fpga_api/#access-api","title":"Access API","text":"The access API provides functions for opening and closing FPGA resources. Opening a resource yields an fpga_handle, which denotes ownership and can be used in subsequent API calls to interact with a specific resource. Ownership can be exclusive or shared.
"},{"location":"sw/fpga_api/fpga_api/#accessh","title":"access.h","text":"access.h
"},{"location":"sw/fpga_api/fpga_api/#event-api","title":"Event API","text":"The event API provides functions and types for handling asynchronous events such as errors or accelerator interrupts.
To natively support asynchronous event, the driver for the FPGA platform needs to support events natively (in which case the OPAE C library will register the event directly with the driver). For some platforms that do not support interrupt-driven event delivery, you need to run the FPGA Daemon (fpgad) to enable asynchronous OPAE events. fpgad will act as a proxy for the application and deliver asynchronous notifications for registered events.
"},{"location":"sw/fpga_api/fpga_api/#eventh","title":"event.h","text":"event.h
"},{"location":"sw/fpga_api/fpga_api/#mmio-and-shared-memory-apis","title":"MMIO and Shared Memory APIs","text":"These APIs feature functions for mapping and accessing control registers through memory-mapped IO (mmio.h), allocating and sharing system memory buffers with an accelerator (buffer.h), and using low-latency notifications (umsg.h).
"},{"location":"sw/fpga_api/fpga_api/#mmioh","title":"mmio.h","text":"mmio.h
"},{"location":"sw/fpga_api/fpga_api/#bufferh","title":"buffer.h","text":"buffer.h
"},{"location":"sw/fpga_api/fpga_api/#umsgh","title":"umsg.h","text":"umsg.h
"},{"location":"sw/fpga_api/fpga_api/#management-api","title":"Management API","text":"The management APIs define functions for reconfiguring an FPGA (writing new partial bitstreams) as well as assigning accelerators to host interfaces.
"},{"location":"sw/fpga_api/fpga_api/#manageh","title":"manage.h","text":"manage.h
"},{"location":"sw/fpga_api/fpga_api/#metrics-api","title":"Metrics API","text":"The metrics APIs define functions for discovery/enumeration of metrics information and reading metrics values.
"},{"location":"sw/fpga_api/fpga_api/#metricsh","title":"metrics.h","text":"metrics.h
"},{"location":"sw/fpga_api/fpga_api/#sysobject","title":"SysObject","text":"The SysObject API can be used to get system objects by name. Names used with the SysObject API are driver-specific and may not be compatible across plugins and/or drivers. For example, SysObject names used with the xfpga plugin will apply to the OPAE Linux Kernel driver and refer to sysfs nodes under the sysfs tree for the resource used with the SysObject API.
"},{"location":"sw/fpga_api/fpga_api/#sysobjecth","title":"sysobject.h","text":"sysobject.h
"},{"location":"sw/fpga_api/fpga_api/#utilities","title":"Utilities","text":"Functions for mapping fpga_result values to meaningful error strings are provided by the utilities API.
"},{"location":"sw/fpga_api/fpga_api/#utilsh","title":"utils.h","text":"utils.h
"},{"location":"sw/fpga_api/fpga_api/#samples","title":"Samples","text":"Code samples demonstrate how to use OPAE C API.
"},{"location":"sw/fpga_api/fpga_api/#hello_fpgac","title":"hello_fpga.c","text":"hello_fpga.c
"},{"location":"sw/fpga_api/fpga_api/#hello_eventsc","title":"hello_events.c","text":"hello_events.c
"},{"location":"sw/fpga_api/fpga_cxx_api/","title":"OPAE C++ Core API Reference","text":"The reference documentation for the OPAE C++ Core API is grouped into the following sections:
- Overview
- Goals
- Simplicity
- Extensibility and Interoperability
- Modern C++ Coding Practices
- Error Handling
- Coding Style
- Fundamental Types
- Properties
- pvalue.h
- properties.h
- Resource Classes
- token.h
- handle.h
- shared_buffer.h
- errors.h
- events.h
- sysobject.h
- Exceptions
- except.h
- Misc
- version.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#overview","title":"Overview","text":"The OPAE C++ API enables C++ developers with the means to use FPGA resources by integrating the OPAE software stack into C++ applications.
"},{"location":"sw/fpga_api/fpga_cxx_api/#goals","title":"Goals","text":""},{"location":"sw/fpga_api/fpga_cxx_api/#simplicity","title":"Simplicity","text":"Keep the API as small and lightweight as possible. Although features such as system validation and orchestration are beyond the scope of this API, using this API for their development should be relatively easy.
"},{"location":"sw/fpga_api/fpga_cxx_api/#extensibility-and-interoperability","title":"Extensibility and Interoperability","text":"While keeping to the goal of simplicity, the OPAE C++ API is designed to allow for better reuse by either extending the API or by integrating with other languages.
"},{"location":"sw/fpga_api/fpga_cxx_api/#modern-c-coding-practices","title":"Modern C++ Coding Practices","text":"The OPAE C++ API uses the C++ 11 standard library and makes use of its features whenever practical. The OPAE C++ API is also designed to require the minimum number of third-party libraries/dependencies.
"},{"location":"sw/fpga_api/fpga_cxx_api/#error-handling","title":"Error Handling","text":"The OPAE C++ API is designed to throw exceptions when appropriate. The structure of OPAE C++ exceptions is similar to the error codes in the OPAE C API. This gives users of the API more freedom on error handling while providing better debug information in cases of failure.
"},{"location":"sw/fpga_api/fpga_cxx_api/#coding-style","title":"Coding Style","text":"For formatting of the OPAE C++ API complies with most of the recommendations of the Google C++ style. For example, the OPAE C++ API uses:
- opening braces on the same line as their scope definition
- spaces instead of tabs for indentation
- indentation of two spaces
"},{"location":"sw/fpga_api/fpga_cxx_api/#fundamental-types","title":"Fundamental Types","text":"Basic types for the OPAE C++ API are found in the opae::fpga::types namespace. They serve as an adapter layer between the OPAE C API and the OPAE C++ layer. Aside from providing a C++ binding to the C fundamental types, these types also:
- manage the lifetime and scope of the corresponding C struct.
- For example a C++ destructor will take care of calling the appropriate C function to release the data structure being wrapped.
- provide a friendly syntax for using the OPAE C type.
Most classes in this namespace have a c_type() method that returns the C data structure being wrapped, making it easy to use the OPAE C++ type with the OPAE C API. Alternatively, most classes in this namespace have implicit conversion operators that enable interoperability with the OPAE C API.
"},{"location":"sw/fpga_api/fpga_cxx_api/#properties","title":"Properties","text":"C++ class properties wraps fpga_properties and uses pvalue and guid_t to get and set properties stored in an instance of an fpga_properties. pvalue and guid_t are designed to call an accessor method in the OPAE C API to either read property values or write them. Most accessor methods in the OPAE C API share a similar signature, so pvalue generalizes them into common operations that translate into calling the corresponding C API function. guid_t follows similar patterns when reading or assigning values.
"},{"location":"sw/fpga_api/fpga_cxx_api/#pvalueh","title":"pvalue.h","text":"pvalue.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#propertiesh","title":"properties.h","text":"properties.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#resource-classes","title":"Resource Classes","text":"The token, handle, and shared_buffer classes are used to enumerate and access FPGA resources. properties are used to narrow the search space for token's. Before enumerating the accelerator resources in the system, applications can produce one or more properties objects whose values are set to the desired characteristics for the resource. For example, an application may search for an accelerator resource based on its guid.
Once one or more token's have been enumerated, the application must choose which token's to request. The token is then converted to a handle by requesting that a handle object be allocated and opened for it.
Once a handle has been successfully opened, the application can read and write the associated configuration and status space. Additionally, the application may use the handle to allocate shared_buffer's or to register event's. The shared_buffer and event objects retain a reference to their owning handle so that the handle does not lose scope before freeing the shared_buffer and event objects.
"},{"location":"sw/fpga_api/fpga_cxx_api/#tokenh","title":"token.h","text":"token.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#handleh","title":"handle.h","text":"handle.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#shared_bufferh","title":"shared_buffer.h","text":"shared_buffer.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#errorsh","title":"errors.h","text":"errors.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#eventsh","title":"events.h","text":"events.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#sysobjecth","title":"sysobject.h","text":"sysobject.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#exceptions","title":"Exceptions","text":"When the OPAE C++ API encounters an error from the OPAE C API, it captures the current source code location and the error code into an object of type except, then throws the except. Applications should implement the appropriate catch blocks required to respond to runtime exceptions.
"},{"location":"sw/fpga_api/fpga_cxx_api/#excepth","title":"except.h","text":"except.h
"},{"location":"sw/fpga_api/fpga_cxx_api/#misc","title":"Misc","text":"The version class wraps the OPAE C version API.
"},{"location":"sw/fpga_api/fpga_cxx_api/#versionh","title":"version.h","text":"version.h
"},{"location":"sw/fpga_api/fpga_python_api/","title":"OPAE Python API Reference","text":"The reference documentation for the OPAE Python API and is grouped into the following sections:
- Module Types, Methods, and Constants
- Fundamental Types
- Properties
- Token
- Handle
- Event
- Shared Buffer
- Error
- SysObject
"},{"location":"sw/fpga_api/fpga_python_api/#module-types-methods-and-constants","title":"Module Types, Methods, and Constants","text":""},{"location":"sw/fpga_api/fpga_python_api/#fundamental-types","title":"Fundamental Types","text":""},{"location":"sw/fpga_api/fpga_python_api/#properties","title":"Properties","text":""},{"location":"sw/fpga_api/fpga_python_api/#token","title":"Token","text":""},{"location":"sw/fpga_api/fpga_python_api/#handle","title":"Handle","text":""},{"location":"sw/fpga_api/fpga_python_api/#event","title":"Event","text":""},{"location":"sw/fpga_api/fpga_python_api/#shared-buffer","title":"Shared Buffer","text":""},{"location":"sw/fpga_api/fpga_python_api/#error","title":"Error","text":""},{"location":"sw/fpga_api/fpga_python_api/#sysobject","title":"SysObject","text":""},{"location":"sw/fpga_api/plug_guide/readme/","title":"Plugin Developer's Guide","text":""},{"location":"sw/fpga_api/plug_guide/readme/#overview","title":"Overview","text":"Beginning with OPAE C library version 1.2.0, OPAE implements a plugin-centric model. This guide serves as a reference to define the makeup of an OPAE C API plugin and to describe a sequence of steps that one may follow when constructing an OPAE C API plugin.
"},{"location":"sw/fpga_api/plug_guide/readme/#plugin-required-functions","title":"Plugin Required Functions","text":"An OPAE C API plugin is a runtime-loadable shared object library, also known as a module. On Linux systems, the dl family of APIs from libdl are used to interact with shared objects. Refer to \"man dlopen\" and \"man dlsym\" for examples of using the libdl API.
An OPAE C API plugin implements one required function. This function is required to have C linkage, so that its name is not mangled.
int opae_plugin_configure(opae_api_adapter_table *table, const char *config);\n
During initialization, the OPAE plugin manager component loads each plugin, searching for its opae_plugin_configure function. If none is found, then the plugin manager rejects that plugin. When it is found, opae_plugin_configure is called passing a pointer to a freshly-created opae_api_adapter_table and a buffer consisting of configuration data for the plugin.
The job of the opae_plugin_configure function is to populate the given adapter table with each of the plugin's API entry points and to consume and comprehend the given configuration data in preparation for initialization.
"},{"location":"sw/fpga_api/plug_guide/readme/#opae-api-adapter-table","title":"OPAE API Adapter Table","text":"The adapter table is a data structure that contains function pointer entry points for each of the OPAE APIs implemented by a plugin. In this way, it adapts the plugin-specific behavior to the more general case of a flat C API. Note that OPAE applications are only required to link with opae-c. In other words, the name of the plugin library should not appear on the linker command line. In this way, plugins are truly decoupled from the OPAE C API, and they are required to adapt to the strict API specification by populating the adapter table only. No other linkage is required nor recommended.
adapter.h contains the definition of the opae_api_adapter_table. An abbreviated version is depicted below, along with supporting type opae_plugin:
typedef struct _opae_plugin {\nchar *path;\nvoid *dl_handle;\n} opae_plugin;\ntypedef struct _opae_api_adapter_table {\nstruct _opae_api_adapater_table *next;\nopae_plugin plugin;\nfpga_result (*fpgaOpen)(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result (*fpgaClose)(fpga_handle handle);\n...\nfpga_result (*fpgaEnumerate)(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n// configuration functions\nint (*initialize)(void);\nint (*finalize)(void);\n// first-level query\nbool (*supports_device)(const char *device_type);\nbool (*supports_host)(const char *hostname);\n} opae_api_adapter_table;\n
Some points worth noting are that the adapter tables are organized in memory by adding them to a linked list data structure. This is the use of the next structure member. (The list management is handled by the plugin manager.) The plugin structure member contains the handle to the shared object instance, as created by dlopen. This handle is used in the plugin's opae_plugin_configure to load plugin entry points. A plugin need only implement the portion of the OPAE C API that a target application needs. Any API entry points that are not supported should be left as NULL pointers (the default) in the adapter table. When an OPAE API that has no associated entry point in the adapter table is called, the result for objects associated with that plugin will be FPGA_NOT_SUPPORTED.
The following code illustrates a portion of the opae_plugin_configure for a theoretical OPAE C API plugin libfoo.so:
/* foo_plugin.c */\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\nadapter->fpgaOpen = dlsym(adapter->plugin.dl_handle, \"foo_fpgaOpen\");\nadapter->fpgaClose =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaClose\");\n...\nadapter->fpgaEnumerate =\ndlsym(adapter->plugin.dl_handle, \"foo_fpgaEnumerate\");\n...\nreturn 0;\n}\n
Notice that the implementations of the API entry points for plugin libfoo.so are prefixed with foo_. This is the recommended practice to avoid name collisions and to enhance the debugability of the application. Upon successful configuration, opae_plugin_configure returns 0 to indicate success. A non-zero return value indicates failure and causes the plugin manager to reject the plugin from futher consideration.
"},{"location":"sw/fpga_api/plug_guide/readme/#plugin-optional-functions","title":"Plugin Optional Functions","text":"Once the plugin manager loads and configures each plugin, it uses the adapter table to call back into the plugin so that it can be made ready for runtime. This is the job of the opae_plugin_initialize entry point, whose signature is defined as:
int opae_plugin_initialize(void);\n
The function takes no parameters, as the configuration data was already given to the plugin by opae_plugin_configure. opae_plugin_initialize returns 0 if no errors were encountered during initialization. A non-zero return code indicates that plugin initialization failed. A plugin makes its opae_plugin_initialize available to the plugin manager by populating the adapter table's initialize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_initialize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->initialize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_initialize\");\n...\nreturn 0;\n}\n
If a plugin does not implement an opae_plugin_initialize entry point, then the initialize member of the adapter table should be left uninitialized. During plugin initialization, if a plugin has no opae_plugin_initialize entry in its adapter table, the plugin initialization step will be skipped, and the plugin will be considered to have initialized successfully.
Once plugin initialization is complete for all loaded plugins, the system is considered to be running and fully functional.
During teardown, the plugin manager uses the adapter table to call into each plugin's opae_plugin_finalize entry point, whose signature is defined as:
int opae_plugin_finalize(void);\n
opae_plugin_finalize returns 0 if no errors were encountered during teardown. A non-zero return code indicates that plugin teardown failed. A plugin makes its opae_plugin_finalize available to the plugin manager by populating the adapter table's finalize entry point as shown:
/* foo_plugin.c */\nint foo_plugin_finalize(void)\n{\n...\nreturn 0; /* success */\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->finalize =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_finalize\");\n...\nreturn 0;\n}\n
If a plugin does not implement an opae_plugin_finalize entry point, then the finalize member of the adapter table should be left uninitialized. During plugin cleanup, if a plugin has no opae_plugin_finalize entry point in its adapter table, the plugin finalize step will be skipped, and the plugin will be considered to have finalized successfully.
In addition to initialize and finalize, an OPAE C API plugin has two further optional entry points that relate to device enumeration. During enumeration, when a plugin is being considered for a type of device, the plugin may provide input on that decision by exporting an opae_plugin_supports_device entry point in the adapter table:
bool opae_plugin_supports_device(const char *device_type);\n
opae_plugin_supports_device returns true if the given device type is supported and false if it is not. A false return value from opae_plugin_supports_device causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_device is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_device(const char *device_type)\n{\nif (/* device_type is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_device =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_device\");\n...\nreturn 0;\n}\n
.. note::\n The `opae_plugin_supports_device` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\n
Similarly to determining whether a plugin supports a type of device, a plugin may also answer questions about network host support by populating an opae_plugin_supports_host entry point in the adapter table:
bool opae_plugin_supports_host(const char *hostname);\n
opae_plugin_supports_host returns true if the given hostname is supported and false if it is not. A false return value from opae_plugin_supports_host causes device enumeration to skip the plugin.
Populating the opae_plugin_supports_host is done as:
/* foo_plugin.c */\nbool foo_plugin_supports_host(const char *hostname)\n{\nif (/* hostname is supported */)\nreturn true;\n...\nreturn false;\n}\nint opae_plugin_configure(opae_api_adapter_table *table, const char *config)\n{\n... adapter->supports_host =\ndlsym(adapter->plugin.dl_handle, \"foo_plugin_supports_host\");\n...\nreturn 0;\n}\n
.. note::\n The `opae_plugin_supports_host` mechanism serves as a placeholder only.\n It is not implemented in the current version of the OPAE C API.\n
"},{"location":"sw/fpga_api/plug_guide/readme/#plugin-construction","title":"Plugin Construction","text":"The steps required to implement an OPAE C API plugin, libfoo.so, are:
- Create foo_plugin.c: implements
opae_plugin_configure, opae_plugin_initialize, opae_plugin_finalize, opae_plugin_supports_device, and opae_plugin_supports_host as described in the previous sections. - Create foo_plugin.h: implements function prototypes for each of the plugin-specific OPAE C APIs.
/* foo_plugin.h */\nfpga_result foo_fpgaOpen(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result foo_fpgaClose(fpga_handle handle);\n...\nfpga_result foo_fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens,\nuint32_t *num_matches);\n...\n
- Create foo_types.h: implements plugin-specific types for opaque data structures.
/* foo_types.h */\nstruct _foo_token {\n...\n};\nstruct _foo_handle {\n...\n};\nstruct _foo_event_handle {\n...\n};\nstruct _foo_object {\n...\n};\n
- Create foo_enum.c: implements
foo_fpgaEnumerate, foo_fpgaCloneToken, and foo_fpgaDestroyToken. - Create foo_open.c: implements
foo_fpgaOpen. - Create foo_close.c: implements
foo_fpgaClose. - Create foo_props.c: implements
foo_fpgaGetProperties, foo_fpgaGetPropertiesFromHandle, foo_fpgaUpdateProperties - Create foo_mmio.c: implements
foo_fpgaMapMMIO, foo_fpgaUnmapMMIO foo_fpgaWriteMMIO64, foo_fpgaReadMMIO64, foo_fpgaWriteMMIO32, foo_fpgaReadMMIO32. - Create foo_buff.c: implements
foo_fpgaPrepareBuffer, foo_fpgaReleaseBuffer, foo_fpgaGetIOAddress. - Create foo_error.c: implements
foo_fpgaReadError, foo_fpgaClearError, foo_fpgaClearAllErrors, foo_fpgaGetErrorInfo. - Create foo_event.c: implements
foo_fpgaCreateEventHandle, foo_fpgaDestroyEventHandle, foo_fpgaGetOSObjectFromEventHandle, foo_fpgaRegisterEvent, foo_fpgaUnregisterEvent. - Create foo_reconf.c: implements
foo_fpgaReconfigureSlot. - Create foo_obj.c: implements
foo_fpgaTokenGetObject, foo_fpgaHandleGetObject, foo_fpgaObjectGetObject, foo_fpgaDestroyObject, foo_fpgaObjectGetSize, foo_fpgaObjectRead, foo_fpgaObjectRead64, foo_fpgaObjectWrite64. - Create foo_clk.c: implements
foo_fpgaSetUserClock, foo_fpgaGetUserClock.
"},{"location":"sw/fpga_api/prog_guide/readme/","title":"OPAE C API Programming Guide","text":""},{"location":"sw/fpga_api/prog_guide/readme/#overview","title":"Overview","text":"The OPAE C library (libopae-c) is a lightweight user-space library that provides abstractions for FPGA resources in a compute environment. The OPAE C library builds on the driver stack that supports the FPGA device, abstracting hardware- and OS-specific details. It provides access to the underlying FPGA resources as a set of features available to software programs running on the host. These features include the acceleration logic preconfigured on the FPGA and functions to manage and reconfigure the FPGA. The library enables your applications to transparently and seamlessly benefit from FPGA-based acceleration.
By providing a unified C API, the library supports different FPGA integration and deployment models, ranging from single-node systems with one or a few FPGA devices to large-scale FPGA deployments in a data center. At one end of the spectrum, the API supports a simple application using a PCIe link to reconfigure the FPGA with different accelerator functions. At the other end of the spectrum, resource management and orchestration services in a data center can use this API to discover and select FPGA resources and then allocate them for use by acceleration workloads.
"},{"location":"sw/fpga_api/prog_guide/readme/#opae-role","title":"OPAE Role","text":"The OPAE provides a common base layer for a wide range of applications without sacrificing performance or efficiency. The abstraction layer limits the details of the FPGA hardware that software applications must handle.
The OPAE provides consistent interfaces to crucial components of the platform. The OPAE does not constrain frameworks and applications by making optimizations with limited applicability. When the OPAE does provide convenience functions or optimizations, they are optional.
For example, the OPAE provides an interface to allocate physically contiguous buffers in system memory that user-space software and an accelerator can share. This interface enables the most basic feature set of allocating and sharing a large page of memory in one API call. However, it does not provide a malloc()-like interface backed by a memory pool or slab allocator. Higher layers of the software stack can make such domain-specific optimizations.
"},{"location":"sw/fpga_api/prog_guide/readme/#intel-accelerator-stack-hardware-terminology","title":"Intel Accelerator Stack Hardware Terminology","text":"The following terms define the hardware and hardware processes involved in creating an accelerator function.
- FPGA: Field Programmable Gate Array is a discrete or integrated device connecting to a host CPU via PCIe or other type of interconnects.
- Accelerator Function Unit (AFU): The AFU is the supplied implementation of an accelerator, typically in HDL. AFUs implement a function such as compression, encryption, or mathematical operations. The Quartus Prime Pro software synthesizes the RTL logic into a bitstream.
- Accelerator Function (AF): The AF is the compiled binary for an AFU. An AF is a raw binary file (.rbf) bitstream. A tool (fpgaconf) reconfigures the FPGA using an AF bitstream.
- Reconfiguration: The process of reprogramming the FPGA with a different AF.
"},{"location":"sw/fpga_api/prog_guide/readme/#opae-software-concepts-reflected-in-the-c-api","title":"OPAE Software Concepts Reflected in the C API","text":"The following OPAE data structures and functions integrate AFUs into the OPAE environment. The OPAE C API models these data structures and functions. For more information on the object models refer to the Object model section.
- Accelerator: An accelerator is an allocable accelerator function implemented in an FPGA. An accelerator tracks the ownership of an AFU (or part of it) for a process that uses it. Multiple processes can share an accelerator.
- Device: The OPAE enumerates and models two device types: the FPGA and the AFU.
- Events: Events are asynchronous notifications. The FPGA driver triggers particular events to indicate error conditions. Accelerator logic can also define its own events. User applications can choose to be notified when particular events occur and respond appropriately.
- Shared memory buffers: Software allocates shared memory buffers in user process memory on the host. Shared memory buffers facilitate data transfers between the user process and the accelerator that it owns.
"},{"location":"sw/fpga_api/prog_guide/readme/#opae-library","title":"OPAE Library","text":"Linking with this library is straightforward. Code using the OPAE library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, here is the simplest compile and link command:
gcc myprog.c -I</path/to/fpga.h> -L</path/to/libopae-c.so> -lopae-c -luuid -ljson-c -lpthread
.. note::
The OPAE library uses the third-party `libuuid` and `libjson-c` libraries that are not distributed with \nthe OPAE library. Make sure to install these libraries.\n
"},{"location":"sw/fpga_api/prog_guide/readme/#sample-code","title":"Sample Code","text":"The library source includes two code samples. Use these samples to learn how to call functions in the library. Build and run these samples to determine if your installation and environment are set up properly.
Refer to the Running the Hello FPGA Example chapter in the Intel\u00ae Acceleration Stack Quick Start Guide for for Intel Programmable Acceleration Card with Intel Arria\u00ae 10 GX FPGA for more information about using the sample code.
"},{"location":"sw/fpga_api/prog_guide/readme/#high-level-directory-structure","title":"High-Level Directory Structure","text":"Building and installing the OPAE library results in the following directory structure on the Linux OS. Windows and MacOS have similar directories and files.
Directory & Files Contents include/opae Directory containing all header files include/opae/fpga.h Top-level header for user code to include include/opae/access.h Header file for accelerator acquire/release, MMIO, memory management, event handling, and so on include/opae/bitstream.h Header file for bitstream manipulation functions include/opae/common.h Header file for error reporting functions include/opae/enum.h Header file for AFU enumeration functions include/opae/manage.h Header file for FPGA management functions include/opae/types.h Various type definitions lib Directory containing shared library files lib/libopae-c.so The shared dynamic library for linking with the user application doc Directory containing API documentation doc/html Directory for documentation of HTML format doc/latex Directory for documentation of LaTex format doc/man Directory for documentation of Unix man page format"},{"location":"sw/fpga_api/prog_guide/readme/#basic-application-flow","title":"Basic Application Flow","text":"The figure below shows the basic application flow from the viewpoint of a user-process.
"},{"location":"sw/fpga_api/prog_guide/readme/#api-components","title":"API Components","text":"The API object model abstracts the physical FPGA device and available functions. It is a generalized model and extends to describe any FPGA type.
"},{"location":"sw/fpga_api/prog_guide/readme/#object-models","title":"Object Models","text":" fpga_objtype: An enum type that represents the type of an FPGA resource, either FPGA_DEVICE or FPGA_ACCELERATOR. An FPGA_DEVICE object corresponds to a physical FPGA device. Only FPGA_DEVICE objects can invoke management functions. The FPGA_ACCELERATOR represents an instance of an AFU. fpga_token: An opaque type that represents a resource known to, but not necessarily owned by, the calling process. The calling process must own a resource before it can invoke functions of the resource.fpga_handle: An opaque type that represents a resource owned by the calling process. The API functions fpgaOpen() and fpgaClose() acquire and release ownership of a resource that an fpga_handle represents. (Refer to the Functions section for more information.)fpga_properties: An opaque type for a properties object. Your applications use these properties to query and search for appropriate resources. The FPGA Resource Properties section documents properties visible to your applications.fpga_event_handle: An opaque handle the FPGA driver uses to notify your application about an event. fpga_event_type: An enum type that represents the types of events. The following are valid values: FPGA_EVENT_INTERRUPT, FPGA_EVENT_ERROR, and FPGA_EVENT_POWER_THERMAL. (The Intel Programmable Acceleration Card (PAC) with Intel Arria 10 GX FPGA does not handle thermal and power events.)fpga_result: An enum type to represent the result of an API function. If the function returns successfully the result is FPGA_OK. Otherwise, the result is the appropriate error codes. Function fpgaErrStr() translates an error code into human-readable strings.
"},{"location":"sw/fpga_api/prog_guide/readme/#functions","title":"Functions","text":"The table below groups important API calls by their functionality. For more information about each of the functions, refer to the OPAE C API reference manual.
Functionality API Call FPGA Accelerator Description Enumeration fpgaEnumerate() Yes Yes Query FPGA resources that match certain properties Enumeration: Properties fpga[Get, Update, Clear, Clone, Destroy Properties]() Yes Yes Manage fpga_properties life cycle fpgaPropertiesGet[Prop]() Yes Yes Get the specified property Prop, from the FPGA Resource Properties table fpgaPropertiesSet[Prop]() Yes Yes Set the specified property Prop, from the FPGA Resource Properties table Access: Ownership fpga[Open, Close]() Yes Yes Acquire/release ownership Access: Reset fpgaReset() Yes Yes Reset an accelerator Access: Event handling fpga[Register, Unregister]Event() Yes Yes Register/unregister an event to be notified about fpga[Create, Destroy]EventHandle() Yes Yes Manage fpga_event_handle life cycle Access: MMIO fpgaMapMMIO(), fpgaUnMapMMIO() Yes Yes Map/unmap MMIO space fpgaGetMMIOInfo() Yes Yes Get information about the specified MMIO space fpgaReadMMIO[32, 64]() Yes Yes Read a 32-bit or 64-bit value from MMIO space fpgaWriteMMIO[32, 64]() Yes Yes Write a 32-bit or 64-bit value to MMIO space Memory management: Shared memory fpga[Prepare, Release]Buffer() Yes Yes Manage memory buffer shared between the calling process and an accelerator fpgaGetIOAddress() Yes Yes Return the device I/O address of a shared memory buffer fpgaBindSVA() Yes Yes Bind IOMMU shared virtual addressing Management: Reconfiguration fpgaReconfigureSlot() Yes No Replace an existing AFU with a new one Error report fpgaErrStr() Yes Yes Map an error code to a human readable string .. note::
The UMsg APIs are not supported for the Intel(R) PAC cards. They will be deprecated in a future release.\n
"},{"location":"sw/fpga_api/prog_guide/readme/#fpga-resource-properties","title":"FPGA Resource Properties","text":"Applications query resource properties by specifying the property name for Prop in the fpgaPropertiesGet[Prop]() and fpgaPropertiesSet[Prop]() functions. The FPGA and Accelerator columns state whether or not the Property is available for the FPGA or Accelerator objects.
Property FPGA Accelerator Description Parent No Yes fpga_token of the parent object ObjectType Yes Yes The type of the resource: either FPGA_DEVICE or FPGA_ACCELERATOR Bus Yes Yes The bus number Device Yes Yes The PCI device number Function Yes Yes The PCI function number SocketId Yes Yes The socket ID DeviceId Yes Yes The device ID NumSlots Yes No Number of AFU slots available on an FPGA_DEVICE resource BBSID Yes No The FPGA Interface Manager (FIM) ID of an FPGA_DEVICE resource BBSVersion Yes No The FIM version of an FPGA_DEVICE resource VendorId Yes No The vendor ID of an FPGA_DEVICE resource GUID Yes Yes The GUID of an FPGA_DEVICE or FPGA_ACCELERATOR resource NumMMIO No Yes The number of MMIO space of an FPGA_ACCELERATOR resource NumInterrupts No Yes The number of interrupts of an FPGA_ACCELERATOR resource AcceleratorState No Yes The state of an FPGA_ACCELERATOR resource: either FPGA_ACCELERATOR_ASSIGNED or FPGA_ACCELERATOR_UNASSIGNED"},{"location":"sw/fpga_api/prog_guide/readme/#opae-c-api-return-codes","title":"OPAE C API Return Codes","text":"The OPAE C library returns a code for every exported public API function. FPGA_OK indicates successful completion of the requested operation. Any return code other than FPGA_OK indicates an error or unexpected behavior. When using the OPAE C API, always check the API return codes.
Error Code Description FPGA_OK Operation completed successfully FPGA_INVALID_PARAM Invalid parameter supplied FPGA_BUSY Resource is busy FPGA_EXCEPTION An exception occurred FPGA_NOT_FOUND A required resource was not found FPGA_NO_MEMORY Not enough memory to complete operation FPGA_NOT_SUPPORTED Requested operation is not supported FPGA_NO_DRIVER Driver is not loaded FPGA_NO_DAEMON FPGA Daemon (fpgad) is not running FPGA_NO_ACCESS Insufficient privileges or permissions FPGA_RECONF_ERROR Error while reconfiguring FPGA"},{"location":"sw/fpga_api/prog_guide/readme/#usage-models","title":"Usage Models","text":""},{"location":"sw/fpga_api/prog_guide/readme/#query-and-search-for-a-resource","title":"Query and Search for a Resource","text":"The user-code first populates an fpga_properties object with the required properties. Then, fpgaEnumerate() searches for matching resources. fpgaEnumerate() may return more than one matching resource.
#include \"fpga/fpga.h\"\n\nfpga_guid guid;\nfpga_properties filter = NULL;\nfpga_result res;\nfpga_token tokens[MAX_NUM_TOKENS];\nuint32_t num_matches = 0;\n\n/* Start with an empty properties object */\nres = fpgaGetProperties(NULL, &filter);\n\n/* Populate the properties object with required values.\n In this case, search for accelerators that matches \n the specified GUID.\n*/\nuuid_parse(GUID, guid);\nres = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nres = fpgaPropertiesSetGuid(filter, guid);\n\n/* Query the number of matching resources */\nres = fpgaEnumerate(&filter, 1, NULL, 1, &num_matches);\n\n/* Return tokens for all matching resources */\nres = fpgaEnumerate(&filter, 1, tokens, num_matches, &num_matches);\n\n/* Destroy the properties object */\nres = fpgaDestroyProperties(&filter);\n\n/* More code */\n......\n\n/* Destroy tokens */\nfor (uint32_t i = 0; i < num_matches; ++i) {\n res = fpgaDestroyToken(tokens[i]);\n}\n
The fpgaEnumerate() function can take multiple fpga_propertiesobjects in an array. In such cases, the function performs a logical OR of the properties object and returns resources that match any of the multiple properties. The fpga_token objects that fpgaEnumerate() returns, do not signify ownership. To acquire ownership of a resource represented by a token, pass the token to fpgaOpen().
"},{"location":"sw/fpga_api/prog_guide/readme/#acquire-and-release-a-resource","title":"Acquire and Release a Resource","text":"Use fpgaOpen() and fpgaClose() to acquire and release ownership of a resource. The calling process must own the resource before it can initiate MMIO, access share memory buffers, and use functions offered by the resource.
#include \"fpga/fpga.h\"\nfpga_handle handle;\nfpga_result res;\n/* Acquire ownership of a resource that \n `fpgaEnumerate()` previously returned as a token */\nres = fpgaOpen(token, &handle);\n/* More code */\n......\n/* Release the ownership */\nres = fpgaClose(handle);\n
"},{"location":"sw/fpga_api/prog_guide/readme/#shared-memory-buffer","title":"Shared Memory Buffer","text":"This code snippet shows how to prepare a memory buffer to be shared between the calling process and an accelerator.
#include \"fpga/fpga.h\"\nfpga_handle handle;\nfpga_result res;\n/* Hint for the virtual address of the buffer */\nvolatile uint64_t *addr_hint;\n/* An ID we can use to reference the buffer later */\nuint32_t bufid;\n/* Flag to indicate whether or not the buffer is preallocated */\nint flag = 0;\n/* Allocate (if necessary), pin, and map a buffer to be accessible\n by an accelerator\n */\nres = fpgaPrepareBuffer(handle, BUF_SIZE, (void **) &addr_hint,\n&bufid, flag);\n/* The actual address mapped to the buffer */\nuint64_t iova;\n/* Get the IO virtual address for the buffer */\nres = fpgaGetIOAddress(handle, bufid, &iova);\n/* Inform the accelerator about the virtual address by writing to its mapped\n register file\n */\n......\n/* More code */\n......\n/* Release the shared buffer */\nres = fpgaReleaseBuffer(handle, bufid);\n
.. note::
The `flag` variable can take a constant `FPGA_BUF_PREALLOCATED` to\nindicate that the calling process has already allocated the address space\nthat `addr_hint` points to.\n
"},{"location":"sw/fpga_api/prog_guide/readme/#mmio","title":"MMIO","text":"This code snippet shows how to map and unmap the register file of an accelerator into the calling process's virtual memory space.
#include \"fpga/fpga.h\"\nfpga_handle handle;\nfpga_result res;\n/* Index of the MMIO space. There might be multiple spaces on an accelerator */\nuint32_t mmio_num = 0;\n/* Mapped address */\nuint64_t mmio_addr;\n/* Map MMIO */\nres = fpgaMapMMIO(handle, mmio_num, &mmio_addr);\n/* Write to a 32-bit value to the mapped register file at a certain byte\n offset.\n CSR_CTL is the offset in the mapped space to where the value will be\n written. It's defined elsewhere.\n */\nres = fpgaWriteMMIO32(handle, mmio_num, CSR_CTL, value);\n/* More code */\n......\n/* Unmap MMIO */\nres = fpgaUnmapMMIO(handle, mmio_num);\n
.. Note::
Every AFU has its own register adress space and its own protocol to control operation through \nthe registers. \n
"},{"location":"sw/fpga_api/quick_start/readme/","title":"Quick Start Guide","text":""},{"location":"sw/fpga_api/quick_start/readme/#overview","title":"Overview","text":"The OPAE C library is a lightweight user-space library that provides an abstraction for FPGA resources in a compute environment. Built on top of the OPAE Intel\u00ae FPGA driver stack that supports Intel\u00ae FPGA platforms, the library abstracts away hardware-specific and OS-specific details and exposes the underlying FPGA resources as a set of features accessible from within software programs running on the host.
These features include the acceleration logic preconfigured on the device, as well as functions to manage and reconfigure the device. Hence, the library can enable user applications to transparently and seamlessly leverage FPGA-based acceleration.
In this document, we will explore the initial steps on how to set up the required libraries and utilities to use the FPGA devices.
If you do not have access to an Intel\u00ae Xeon\u00ae processor with integrated FPGA, or a programmable FPGA acceleration card for Intel\u00ae Xeon\u00ae processors, you will not be able to run the examples below. However, you can still make use of the AFU simulation environment (ASE) to develop and test accelerator RTL with OPAE applications.
The source for the OPAE SDK Linux device drivers is available at the OPAE Linux DFL drivers repository.
"},{"location":"sw/fpga_api/quick_start/readme/#build-the-opae-linux-device-drivers-from-the-source","title":"Build the OPAE Linux device drivers from the source","text":"For building the OPAE kernel and kernel driver, the kernel development environment is required. So before you build the kernel, you must install the required packages. Run the following commands (we are using Fedora 32 as an example):
sudo yum install gcc gcc-c++ make kernel-headers kernel-devel elfutils-libelf-devel ncurses-devel openssl-devel bison flex\n
Download the OPAE upstream kernel tree from GitHub:
git clone https://github.com/OPAE/linux-dfl.git -b fpga-upstream-dev-5.8.0\n
Configure the kernel:
cd linux-dfl\ncp /boot/config-`uname -r` .config\ncat configs/n3000_d5005_defconfig >> .config echo 'CONFIG_LOCALVERSION=\"-dfl\"' >> .config\necho 'CONFIG_LOCALVERSION_AUTO=y' >> .config\nmake olddefconfig\n
Compile and install the new kernel:
make -j\nsudo make modules_install\nsudo make install\n
After the installation finishes, reboot your system. Log back into the system, and confirm the correct version for the kernel:
$ uname -a\nLinux localhost.localdomain 5.8.0-rc1-dfl-g73e16386cda0 #6 SMP Wed Aug 19 08:38:32 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux\n
"},{"location":"sw/fpga_api/quick_start/readme/#building-and-installing-the-opae-sdk-from-the-source","title":"Building and installing the OPAE SDK from the source","text":"Download the OPAE SDK source package:
- Go to the section corresponding to the desired release on GitHub:
- Click the
Source code (tar.gz) link under the section's Assets. -
On the command line, go through the following steps to install it:
# Unpack\ntar xfvz opae-sdk-<release>.tar.gz\n# Configure\ncd opae-sdk-<release>\nmkdir build\ncd build\n# The default installation prefix is `/usr/local`;\n# You have the option to configure for a different location\ncmake [-DCMAKE_INSTALL_PREFIX=<prefix>] ..\n# Compile\nmake\n# Install: you need system administration privileges (`sudo`)\n# if you have elected to install in the default location\n[sudo] make install\n
The remainder of this guide assumes you installed into /usr/local.
"},{"location":"sw/fpga_api/quick_start/readme/#configuring-the-fpga-loading-an-fpga-afu","title":"Configuring the FPGA (loading an FPGA AFU)","text":"The fpgaconf tool exercises the AFU reconfiguration functionality. It shows how to read a bitstream from a disk file, check its validity and compatibility, and then injects it into FPGA to be configured as a new AFU, which can then be discovered and used by user applications.
For this step, you require a valid green bitstream (GBS) file. To reconfigure the FPGA slot, you can issue the following command as system administrator:
sudo fpgaconf -b 0x5e <filename>.gbs\n
The -b option to fpgaconf indicates the target bus number of the FPGA slot to be reconfigured. Alternatively, you can also specify the target socket number of the FPGA using the -s option.
$ sudo fpgaconf --help\nUsage:\n fpgaconf [-hvn] [-b <bus>] [-d <device>] [-f <function>] [-s <socket>] <gbs>\n\n-h,--help Print this help\n-v,--verbose Increase verbosity\n -n,--dry-run Don't actually perform actions\n -b,--bus Set target bus number\n -d,--device Set target device number\n -f,--function Set target function number\n -s,--socket Set target socket number\n
The sample application on the Building a Sample Application\nsection requires loading of an AFU called \"Native Loopback\nAdapter\" (NLB) on the FPGA. Please refer to the NLB documentation\nfor the location of the NLB's green bitstream. You also can verify\nif the NLB green bitstream has already been loaded into the FPGA\nslot by typing the following command and checking the output\nmatches the following:\n\n```bash\n$ cat /sys/class/fpga_region/region0/dfl-port.0/afu_id\nd8424dc4a4a3c413f89e433683f9040b\n```\n
The fpgaconf tool is not available for the Intel PAC N3000. The NLB is\nalready included in the AFU.\n
"},{"location":"sw/fpga_api/quick_start/readme/#building-a-sample-application","title":"Building a sample application","text":"The library source includes code samples. Use these samples to learn how to call functions in the library. Build and run these samples as quick sanity checks to determine if your installation and environment are set up properly.
In this guide, we will build hello_fpga.c. This is the \"Hello World!\" example of using the library. This code searches for a predefined and known AFU called \"Native Loopback Adapter\" on the FPGA. If found, it acquires ownership and then interacts with the AFU by sending it a 2MB message and waiting for the message to be echoed back. This code exercises all major components of the API except for AFU reconfiguration: AFU search, enumeration, access, MMIO, and memory management.
You can also find the source for hello_fpga in the samples directory of the OPAE SDK repository on GitHub.
int main(int argc, char *argv[])\n{\nfpga_properties filter = NULL;\nfpga_token afu_token;\nfpga_handle afu_handle;\nfpga_guid guid;\nuint32_t num_matches;\nvolatile uint64_t *dsm_ptr = NULL;\nvolatile uint64_t *status_ptr = NULL;\nvolatile uint64_t *input_ptr = NULL;\nvolatile uint64_t *output_ptr = NULL;\nuint64_t dsm_wsid;\nuint64_t input_wsid;\nuint64_t output_wsid;\nfpga_result res = FPGA_OK;\nif (uuid_parse(NLB0_AFUID, guid) < 0) {\nfprintf(stderr, \"Error parsing guid '%s'\\n\", NLB0_AFUID);\ngoto out_exit;\n}\n/* Look for accelerator by its \"afu_id\" */\nres = fpgaGetProperties(NULL, &filter);\nON_ERR_GOTO(res, out_exit, \"creating properties object\");\nres = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nON_ERR_GOTO(res, out_destroy_prop, \"setting object type\");\nres = fpgaPropertiesSetGuid(filter, guid);\nON_ERR_GOTO(res, out_destroy_prop, \"setting GUID\");\n/* TODO: Add selection via BDF / device ID */\nres = fpgaEnumerate(&filter, 1, &afu_token, 1, &num_matches);\nON_ERR_GOTO(res, out_destroy_prop, \"enumerating accelerators\");\nif (num_matches < 1) {\nfprintf(stderr, \"accelerator not found.\\n\");\nres = fpgaDestroyProperties(&filter);\nreturn FPGA_INVALID_PARAM;\n}\n/* Open accelerator and map MMIO */\nres = fpgaOpen(afu_token, &afu_handle, 0);\nON_ERR_GOTO(res, out_destroy_tok, \"opening accelerator\");\nres = fpgaMapMMIO(afu_handle, 0, NULL);\nON_ERR_GOTO(res, out_close, \"mapping MMIO space\");\n/* Allocate buffers */\nres = fpgaPrepareBuffer(afu_handle, LPBK1_DSM_SIZE,\n(void **)&dsm_ptr, &dsm_wsid, 0);\nON_ERR_GOTO(res, out_close, \"allocating DSM buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&input_ptr, &input_wsid, 0);\nON_ERR_GOTO(res, out_free_dsm, \"allocating input buffer\");\nres = fpgaPrepareBuffer(afu_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&output_ptr, &output_wsid, 0);\nON_ERR_GOTO(res, out_free_input, \"allocating output buffer\");\nprintf(\"Running Test\\n\");\n/* Initialize buffers */\nmemset((void *)dsm_ptr, 0, LPBK1_DSM_SIZE);\nmemset((void *)input_ptr, 0xAF, LPBK1_BUFFER_SIZE);\nmemset((void *)output_ptr, 0xBE, LPBK1_BUFFER_SIZE);\ncache_line *cl_ptr = (cache_line *)input_ptr;\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE / CL(1); ++i) {\ncl_ptr[i].uint[15] = i+1; /* set the last uint in every cacheline */\n}\n/* Reset accelerator */\nres = fpgaReset(afu_handle);\nON_ERR_GOTO(res, out_free_output, \"resetting accelerator\");\n/* Program DMA addresses */\nuint64_t iova;\nres = fpgaGetIOAddress(afu_handle, dsm_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting DSM IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_AFU_DSM_BASEL, iova);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_AFU_DSM_BASEL\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 0);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 1);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nres = fpgaGetIOAddress(afu_handle, input_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting input IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_SRC_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_SRC_ADDR\");\nres = fpgaGetIOAddress(afu_handle, output_wsid, &iova);\nON_ERR_GOTO(res, out_free_output, \"getting output IOVA\");\nres = fpgaWriteMMIO64(afu_handle, 0, CSR_DST_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_DST_ADDR\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_NUM_LINES, LPBK1_BUFFER_SIZE / CL(1));\nON_ERR_GOTO(res, out_free_output, \"writing CSR_NUM_LINES\");\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CFG, 0x42000);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\nstatus_ptr = dsm_ptr + DSM_STATUS_TEST_COMPLETE/8;\n/* Start the test */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 3);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Wait for test completion */\nwhile (0 == ((*status_ptr) & 0x1)) {\nusleep(100);\n}\n/* Stop the device */\nres = fpgaWriteMMIO32(afu_handle, 0, CSR_CTL, 7);\nON_ERR_GOTO(res, out_free_output, \"writing CSR_CFG\");\n/* Check output buffer contents */\nfor (uint32_t i = 0; i < LPBK1_BUFFER_SIZE; i++) {\nif (((uint8_t*)output_ptr)[i] != ((uint8_t*)input_ptr)[i]) {\nfprintf(stderr, \"Output does NOT match input \"\n\"at offset %i!\\n\", i);\nbreak;\n}\n}\nprintf(\"Done Running Test\\n\");\n/* Release buffers */\nout_free_output:\nres = fpgaReleaseBuffer(afu_handle, output_wsid);\nON_ERR_GOTO(res, out_free_input, \"releasing output buffer\");\nout_free_input:\nres = fpgaReleaseBuffer(afu_handle, input_wsid);\nON_ERR_GOTO(res, out_free_dsm, \"releasing input buffer\");\nout_free_dsm:\nres = fpgaReleaseBuffer(afu_handle, dsm_wsid);\nON_ERR_GOTO(res, out_unmap, \"releasing DSM buffer\");\n/* Unmap MMIO space */\nout_unmap:\nres = fpgaUnmapMMIO(afu_handle, 0);\nON_ERR_GOTO(res, out_close, \"unmapping MMIO space\");\n/* Release accelerator */\nout_close:\nres = fpgaClose(afu_handle);\nON_ERR_GOTO(res, out_destroy_tok, \"closing accelerator\");\n/* Destroy token */\nout_destroy_tok:\nres = fpgaDestroyToken(&afu_token);\nON_ERR_GOTO(res, out_destroy_prop, \"destroying token\");\n/* Destroy properties object */\nout_destroy_prop:\nres = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(res, out_exit, \"destroying properties object\");\nout_exit:\nreturn res;\n}\n
Linking with the OPAE library is straightforward. Code using this library should include the header file fpga.h. Taking the GCC compiler on Linux as an example, the minimalist compile and link line should look like:
gcc -std=c99 hello_fpga.c -I/usr/local/include -L/usr/local/lib -lopae-c -luuid -ljson-c -lpthread -o hello_fpga\n
The API uses some features from the C99 language standard. The\n`-std=c99` switch is required if the compiler does not support C99 by\ndefault.\n
Third-party library dependency: The library internally uses\n`libuuid` and `libjson-c`. But they are not distributed as part of the\nlibrary. Make sure you have these libraries properly installed.\n
The layout of AFU is different between the N3000 card and Rush Creek/Darby Creek.\nIn the N3000 card, the NLB and DMA are contained in the AFU, so we need to do\nenumeration again in AFU to discover the NLB.\nTo run the hello_fpga application on the N3000 card, it should use the `-c`\noption to support the N3000 card:\n\n```bash\n$ sudo ./hello_fpga -c\nRunning Test\nRunning on bus 0x08.\nAFU NLB0 found @ 28000\nDone Running Test\n```\n
To run the hello_fpga application; just issue:
$ sudo ./hello_fpga\nRunning Test\nDone\n
"},{"location":"sw/fpga_api/quick_start/readme/#setup-iofs-release1-bitstream-on-fpga-pcie-card","title":"Setup IOFS Release1 Bitstream on FPGA PCIe card","text":"Program IOFS Release1 bitstream on the FPGA D5005 or N6000 cards and reboot the system.
Run this command:
$ lspci | grep acc\n3b:00.0 Processing accelerators: Intel Corporation Device af00 (rev 01)\n
Number of virtual functions supported by bitstream:
$ cat /sys/bus/pci/devices/0000:3b:00.0/sriov_totalvfs output: 3\n
Enable FPGA virtual functions:
sudo sh -c \"echo 3 > /sys/bus/pci/devices/0000:3b:00.0/sriov_numvfs\"\n
List of FPGA PF and VF's:
Physical Functions (PFs):\n 3b:00.0 Processing accelerators: Intel Corporation Device af00 (rev 01)\n\nVirtual Functions (VFs).\n 3b:00.1 Processing accelerators: Intel Corporation Device af01 (rev 01)\n 3b:00.2 Processing accelerators: Intel Corporation Device af01 (rev 01)\n 3b:00.3 Processing accelerators: Intel Corporation Device af01 (rev 01)\n
Bind vfio-pcie driver to FPGA virtual functions:
sudo opaevfio -i 0000:3b:00.1 -u userid -g groupid\nsudo opaevfio -i 0000:3b:00.2 -u userid -g groupid\nsudo opaevfio -i 0000:3b:00.3 -u userid -g groupid\n
List of fpga accelerators:
$ fpgainfo port\n\n//****** PORT ******//\n Object Id : 0x600D000000000000\n PCIe s:b:d.f : 0000:3b:00.3\n Device Id : 0xAF00\n Socket Id : 0xFF\n Accelerator Id : 43425ee6-92b2-4742-b03a-bd8d4a533812\n Accelerator GUID : 43425ee6-92b2-4742-b03a-bd8d4a533812\n //****** PORT ******//\n Object Id : 0x400D000000000000\n PCIe s:b:d.f : 0000:3b:00.2\n Device Id : 0xAF00\n Socket Id : 0xFF\n Accelerator Id : 8568AB4E-6bA5-4616-BB65-2A578330A8EB\n Accelerator GUID : 8568AB4E-6bA5-4616-BB65-2A578330A8EB\n //****** PORT ******//\n Object Id : 0x200D000000000000\n PCIe s:b:d.f : 0000:3b:00.1\n Device Id : 0xAF00\n Socket Id : 0xFF\n Accelerator Id : 56e203e9-864f-49a7-b94b-12284c31e02b\n Accelerator GUID : 56e203e9-864f-49a7-b94b-12284c31e02b\n\nFPGA VF1/3b:00.1/Host Exerciser Loopback Accelerator GUID: 56E203E9-864F-49A7-B94B-12284C31E02B\nFPGA VF2/3b:00.2/Host Exerciser Memory Accelerator GUID: 8568AB4E-6bA5-4616-BB65-2A578330A8EB\nFPGA VF3/3b:00.3/Host Exerciser HSSI Accelerator GUID: 43425ee6-92b2-4742-b03a-bd8d4a533812\n
Unbind pcie-vfio dirver to FPGA virtual functions:
sudo opaevfio -r 0000:3b:00.1\n
Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA:
$ host_exerciser lpbk\n\n[lpbk] [info] starting test run, count of 1\nInput Config:0\n Allocate SRC Buffer\n Allocate DST Buffer\n Allocate DSM Buffer\n Start Test\n Test Completed\n Host Exerciser swtest msg:0\n Host Exerciser numReads:32\n Host Exerciser numWrites:32\n Host Exerciser numPendReads:0\n Host Exerciser numPendWrites:0\n [lpbk] [info] Test lpbk(1): PASS\n
In order to successfully run hello\\_fpga, the user needs to configure\n system hugepage to reserve 2M-hugepages.\n For example, the command below reserves 20 2M-hugepages:\n\n ```bash\n echo 20 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages\n ```\n\n For x86_64 architecture CPU, user can use the following command to find out available huge page sizes:\n\n ```bash\n $ grep pse /proc/cpuinfo | uniq\n flags : ... pse ...\n ```\n\n If this command returns a non-empty string, 2MB pages are supported:\n\n ```bash\n $ grep pse /proc/cpuinfo | uniq\n flags : ... pdpe1gb ...\n ```\n\n If this commands returns a non-empty string, 1GB pages are supported.\n ````\n\n````{note}\nThe default configuration for many Linux distributions currently sets a\nrelatively low limit for pinned memory allocations per process \n(RLIMIT_MEMLOCK, often set to a default of 64kiB).\n\nTo run an OPAE application that attempts to share more memory than specified\nby this limit between software and an accelerator, you can either:\n\n* Run the application as root, or\n* Increase the limit for locked memory via `ulimit`:\n\n```bash\nulimit -l unlimited\n```\n\nSee the Installation Guide for how to permanently adjust the memlock limit.\n
"},{"location":"sw/fpga_dfl_drv/fpga_dfl_drv/","title":"Enable OPAE on FPGA PCIe drivers","text":".. toctree::\n.. highlight:: c\n.. highlight:: console\n
FPGA PCIe driver for PCIe-based Field-Programmable Gate Array (FPGA) solutions which implement the Device Feature List (DFL). This driver provides interfaces for user space applications to configure, enumerate, open and access FPGA accelerators on the FPGA DFL devices. additionally, it also enables system level management functions such as FPGA partial reconfiguration, power management, virtualization with DFL framework and DFL feature device drivers.
OPAE 1.4.0 release supports both FPGA Intel Linux driver as well as Linux FPGA DFL driver patch set2. Linux PCIe FPGA DFL driver supports Intel FPGA devices.
FPGA DFL Linux driver source code patchset2 available https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers?h=linux-5.4.y
FPGA DFL Linux driver source code patchset1 available https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/fpga?h=v4.19.14
"},{"location":"sw/fpga_tools/readme/","title":"fpga_tools","text":""},{"location":"sw/fpga_tools/readme/#fpgainfo","title":"fpgainfo","text":""},{"location":"sw/fpga_tools/readme/#name","title":"NAME","text":"fpgainfo - FPGA information tool
"},{"location":"sw/fpga_tools/readme/#synopsis","title":"SYNOPSIS","text":"fpgainfo [-h | --help] [-s | --socket-id] <command> [<args>]\n
"},{"location":"sw/fpga_tools/readme/#description","title":"DESCRIPTION","text":"fpgainfo is a tool to show FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp and is used to specify what type of information to report. Some commands may also have other arguments/options that can be used to control the behavior of that command.
"},{"location":"sw/fpga_tools/readme/#common-options","title":"COMMON OPTIONS","text":"--help, -h
Print help information and exit.\n
--socket-id, -s
Socket ID encoded in BBS. Default=0\n
"},{"location":"sw/fpga_tools/readme/#fpgainfo-commands","title":"FPGAINFO COMMANDS","text":"errors
Show/clear errors of an FPGA resource as specified by the first argument.\nError information is parsed to display in human readable form.\n
power
Show total power consumed by the FPGA hardware in watts\n
temp
Show FPGA temperature values in degrees Farenheit\n
"},{"location":"sw/fpga_tools/readme/#errors-options","title":"ERRORS OPTIONS","text":"--clear, -c
Clear errors for the given FPGA resource\n
"},{"location":"sw/fpga_tools/readme/#errors-arguments","title":"ERRORS ARGUMENTS","text":"The first argument to the errors command is used to specify what kind of resource to act on. It must be one of the following: fme,port,first_error,pcie0,pcie1,bbs,gbs,all More details on the errors reported for the resource can be found below:
"},{"location":"sw/fpga_tools/readme/#errors-resources","title":"ERRORS RESOURCES","text":"fme
Show/clear errors pertaining to the FME\n
port
Show/clear errors pertaining to the PORT\n
first_error
Show/clear first errors encountered by the FPGA\n
pcie0
Show/clear errors pertaining to the PCIE0 lane\n
pcie1
Show/clear errors pertaining to the PCIE1 lane\n
bbs
Show/clear errors pertaining to the BBS (blue bitstream)\n
gbs
Show/clear errors pertaining to the GBS (green bitstream)\n
all
Show/clear errors for all resources\n
"},{"location":"sw/fpga_tools/readme/#examples","title":"EXAMPLES","text":"This command shows the current power consumtion:
./fpgainfo power\n
This command shows the current temperature reading:
./fpgainfo temp\n
This command shows the errors for the FME resource:
./fpgainfo errors fme\n
This command clears all the errors on all resources: ./fpgainfo errors all -c\n
"},{"location":"sw/fpga_tools/readme/#fpgaconf","title":"fpgaconf","text":""},{"location":"sw/fpga_tools/readme/#name_1","title":"NAME","text":"fpgadiag - Configure a green bitstream to an FPGA
"},{"location":"sw/fpga_tools/readme/#synopsis_1","title":"SYNOPSIS","text":"fpgaconf [-hvn] [-b <bus>] [-d <device>] [-f <function>] [-s <socket>] <gbs>
"},{"location":"sw/fpga_tools/readme/#description_1","title":"DESCRIPTION","text":"fpgaconf writes accelerator configuration bitstreams (also referred to as \"green bitstreams\" to an FPGA device recognized by OPAE. In the process, it also checks the green bitstream file for compatibility with the targeted FPGA and its current infrastructure bitstream (the \"blue bistream\"). fpgaconf takes the following arguments:
-h, --help
Print usage information\n
-v, --verbose
Print more verbose messages while enumerating and configuring. Can be\ngiven more than once\n
-n, --dry-run
Perform enumeration, but skip any operations with side-effects (like the\nactual configuration of the bitstream\n
-b, --bus
PCI bus number of the FPGA to target\n
-d, --device
PCI device number of the FPGA to target\n
-f, --function
PCI function number of the FPGA to target\n
-s, --socket
Socket number of the FPGA to target\n
fpgaconf will enumerate available FPGA devices in the system and select compatible FPGAs for configuration. If there are more than one candidate FPGAs that are compatible with the given green bitstream, fpgaconf will exit and ask you to be more specific in selecting the target FPGAs (e.g. by specifying a socket number, or a PCIe bus/device/function).
"},{"location":"sw/fpga_tools/readme/#examples_1","title":"EXAMPLES","text":"fpgaconf my_green_bitstream.gbs
Program \"my_green_bitstream.gbs\" to a compatible FPGA\n
fpgaconf -v -s 0 my_green_bitstream.gbs
Program \"my_green_bitstream.gbs\" to the FPGA in socket 0, if compatible,\nwhile printing out slightly more verbose information\n
"},{"location":"sw/fpga_tools/readme/#fpgad","title":"fpgad","text":""},{"location":"sw/fpga_tools/readme/#name_2","title":"NAME","text":"fpgad - log errors and generate events
"},{"location":"sw/fpga_tools/readme/#synopsis_2","title":"SYNOPSIS","text":"fpgad --daemon [--directory=<dir>] [--logfile=<file>] [--pidfile=<file>] [--umask=<mode>] [--socket=<sock>] [--null-bitstream=<file>] fpgad [--socket=<sock>] [--null-bitstream=<file>]
"},{"location":"sw/fpga_tools/readme/#description_2","title":"DESCRIPTION","text":"Periodically monitors/reports the error status reflected in the device driver's error status sysfs files. Establishes the channel by which events are communicated to the OPAE application. Programs a NULL bitstream in response to AP6 event.
fpgad is required to be running before API calls fpgaRegisterEvent and fpgaUnregisterEvent will succeed.
Use SIGINT to stop fpgad.
-d, --daemon
When given, fpgad executes as a system demon process.\n
-D, --directory <dir>
When running in daemon mode, execute from the given directory.\nIf omitted when daemonizing, /tmp is used.\n
-l, --logfile <file>
When running in daemon mode, send output to file. When not in daemon mode, the output is sent to stdout.\nIf omitted when daemonizaing, /tmp/fpgad.log is used.\n
-p, --pidfile <file>
When running in daemon mode, write the daemon's process id to file.\nIf omitted when daemonizing, /tmp/fpgad.pid is used.\n
-m, --umask <mode>
When running in daemon mode, use the mode value as the file mode creation mask passed to umask.\nIf omitted when daemonizing, 0 is used.\n
-s, --socket <sock>
Listen for event API registration requests on sock. The default socket value used by the API is\n/tmp/fpga_event_socket.\n
-n, --null-bitstream <file>
Specify the NULL bitstream to program when an AP6 event occurs. This option may be given multiple\ntimes. The bitstream, if any, that matches the FPGA's PR interface id will be programmed when AP6\nis detected.\n
"},{"location":"sw/fpga_tools/readme/#troubleshooting","title":"TROUBLESHOOTING","text":"If any issues are encountered, try the following for additional debug information:
- Examine the log file when in daemon mode.
- Run in non-daemon mode and view stdout.
"},{"location":"sw/fpga_tools/readme/#examples_2","title":"EXAMPLES","text":"fpgad --daemon --null-bitstream=my_null_bits.gbs
"},{"location":"sw/fpga_tools/readme/#see-also","title":"SEE ALSO","text":"umask
"},{"location":"sw/fpga_tools/readme/#fpgadiag","title":"fpgadiag","text":""},{"location":"sw/fpga_tools/readme/#name_3","title":"NAME","text":"fpgadiag - FPGA diagnosis and testing tool.
"},{"location":"sw/fpga_tools/readme/#synopsis_3","title":"SYNOPSIS","text":"fpgadiag [-m | --mode=] <mode> [-t | --target=] <target> [options]\n
"},{"location":"sw/fpga_tools/readme/#description_3","title":"DESCRIPTION","text":"fpgadiag includes several tests to diagnose, test and report on the FPGA hardware.
<mode> chooses which test to run. <target> specifies on what platform to run the test. <target> can be either fpga or ase, where ase stands for \"AFU Simulation Environment\".
The tests that can be selected by <mode> include:
lpbk1
The test performs loopback test on the number of cachelines specified with \nthe `BEGIN` option. _fpgadiag_ sets up source and destination buffers in \nmain memory. The FPGA then performs a memcpy from a source buffer to the \ndestination buffer, one cacheline at a time.\n\nA cacheline is 64 bytes. When `BEGIN = END`, you perform one iteration. When \n`BEGIN = END + x`, you perform `x` iterations. The first iteration consists \nof copying `BEGIN` cachelines; the second iteration consists of copying \n`BEGIN+1` cache lines; the third iteration consists of copying `BEGIN+3` \ncache lines, etc.\n\nThe latency is shown as the number of clock ticks.\n\nWhen you specify `MULTI-CL`, you copy `MULTI-CL` cache lines at a time.\nThere is always a WrFence. `WR-FENCE` chooses what virtual channel the \nWrFence occurs on.\n\nIf you specify continuous mode with `--cont`, the program runs an iteration\nuntil the timeout specified in `TIMEOUT` completes.\n
read
This test performs only a read, not a memcpy. It is used to measure read \nbandwidth.\n
write
This test is used to measure write bandwidth.\n
trput
This test measures both read and write bandwidth by performing 50% read and \n50% write tests.\n
sw
This is a send-and-respond (ping-pong) test where one side sends data and \nwaits for answer.\n
Each test requires presence of one of these bitstreams, as documented below. Before running a test, make sure its required bitstream is properly configured on the platform.
- nlb mode 0 for the
lpbk1 test. - nlb mode 3 for the
trput, read, and write tests. - nlb mode 7 for the
sw test.
"},{"location":"sw/fpga_tools/readme/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/readme/#common-options_1","title":"Common options","text":"--help, -h
Print help information and exit.\n
--target=, -t
Values accepted for this switch are fpga or ase. Default=fpga\n
--mode=, -m
The test to run. Values accepted for this switch are `lpbk1`, `read`,\n`write`, `trput`, `sw`\n
--config=, -c
A configuration file in the JSON format that specifies options for a test.\nIf an option is specified both in the configuration file and on the command \nline, the value in the configuration file prevails\n
--socket-id=, -s
Socket ID encoded in BBS. Default=0\n
--bus-number=, -B
Bus number of the PCIe device. Default=0\n
--device=, -D
Device number of the PCIe device. Default=0\n
--function=, -F
Function number of the PCIe device. Default=0\n
--freq=, -T
Clock frequency in Hz. Default=400 MHz\n
--suppress-hdr, -S
Suppress column headers for text output. Default=off\n
--csv, -V
Comma separated value format. Default=off\n
"},{"location":"sw/fpga_tools/readme/#lpbk1-test-options","title":"lpbk1 test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=D8424DC4-A4A3-C413-F89E-433683F9040B\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -U
M can equal 1, 2, or 4. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I Default=wrline-M\n
--cache-hint=, -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. Default=auto\n
"},{"location":"sw/fpga_tools/readme/#read-test-options","title":"read test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-hint=, -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Attempt to prime the cache with hits. Default=off, Attempt to prime the \ncache with misses. Default=off\n
--cool-cpu-cache, -C
Attempt to prime the cpu cache with misses. Default=off\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. Default=auto\n
"},{"location":"sw/fpga_tools/readme/#write-test-options","title":"write test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I Default=wrline-M\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Attempt to prime the cache with hits. Default=off, Attempt to prime the \ncache with misses. Default=off\n
--cool-cpu-cache, -C
Attempt to prime the cpu cache with misses. Default=off\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1, random. Default=`WRITE-VC`\n
--alt-wr-pattern, -l
Alternate Write Pattern. Default=off\n
"},{"location":"sw/fpga_tools/readme/#trput-test-options","title":"trput test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cont, -L
Continuous mode. Default=off\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode (microseconds portion default=0; milliseconds \nportion default=0; seconds portion default=1; minutes portion default=0;\nhours portion default=0)\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I Default=wrline-M\n
--cache-hint=, -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. Default=`WRITE-VC`\n
"},{"location":"sw/fpga_tools/readme/#sw-test-options","title":"sw test options","text":"--guid=, -g
Accelerator ID to enumerate. Default=7BAF4DEA-A57C-E91E-168A-455D9BDA88A3\n
--begin=B, -b
1 <= B <= 65535. Default=1, B = number of cache lines\n
--end=E, -e
1 <= E <= 65535. Default=B, B and E designate number of cache lines\n
--multi-cl=M, -u
M can equal 1, 2, or 4. Default=1\n
--strided-access=S, -a
1<= S <= 64. Default=1\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I. Default=wrline-M\n
--cache-hint= -i
Can be rdline-I or rdline-S. Default=rdline-I\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random Default=auto\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random Default=auto\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. Default=`WRITE-VC`\n
--notice=, -N
Can be poll, csr-write, umsg-data, or umsg-hint. Default=poll\n
"},{"location":"sw/fpga_tools/readme/#examples_3","title":"EXAMPLES","text":"This command starts an lpbk1 test on the FPGA on bus 0x5e. The test copies 57535, 57536, 57537, ..., up to 65535 cache lines, one line at a time. The test output is printed in the CSV format with header suppressed.
./fpgadiag --mode=lpbk1 --target=fpga -SV --bus-number=0x5e --begin=57535\n--end=65535 --cache-hint=rdline-I --cache-policy=wrpush-I --multi-cl=1\n--write-vc=vl0 --read-vc=vh1 --wrfence-vc=auto\n
This command starts a read test on the FPGA located on bus 0xbe. The test reads 2045 cache lines in the continuous mode with a 15-second timeout period. Data is accessed with a strided pattern with a 10-byte stride length.
./fpgadiag --mode=read --target=fpga -SV --bus-number=0xbe --begin=2045 --cont\n--timeout-sec=15 --cache-hint=rdline-I --multi-cl=1 -a=10 --write-vc=vh1\n--read-vc=auto --wrfence-vc=auto\n
This command starts an sw test on the FPGA located on bus 0xbe. The test notifies completion using a CSR write.
./fpgadiag --mode=sw --target=fpga -SV --bus-number=0xbe --begin=4 --end=8192\n--cache-hint=rdline-I --cache-policy=wrline-I --notice=csr-write --write-vc=vl0\n--wrfence-vc=auto --read-vc=random \n
"},{"location":"sw/fpga_tools/readme/#troubleshooting_1","title":"TROUBLESHOOTING","text":"When a test fails to run or gives errors, check the following:
- Is Intel FPGA driver properly installed? See Installation Guide for driver installation instructions.
- Are FPGA port permissions set properly? Check the permission bits of the port, for example,
/dev/intel-fpga-port-0. Users need READ and WRITE permissions to run fpgadiag tests. - Is hugepage properly configured on the system? See Installation Guide for hugepage configuration steps.
- Is the required bitstream loaded? See DESCRIPTION for information about what bitstream is required by which test.
- Are
--begin and --end values set properly? --end must be no smaller than the --begin. Also, --begin must be a multiple of the --multi-cl value. - The
--warm-fpga-cache and --cool-fpga-cache options in the read and write tests are mutually exclusive. - The timeout options are only meaningful for the continuous mode (with the
--cont option).
"},{"location":"sw/fpga_tools/readme/#mmlink","title":"mmlink","text":""},{"location":"sw/fpga_tools/readme/#name_4","title":"NAME","text":"MMLink - Debugging RTL.
"},{"location":"sw/fpga_tools/readme/#synopsis_4","title":"SYNOPSIS","text":"mmlink [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-P <TCP port>] [-I <IP Address>]
"},{"location":"sw/fpga_tools/readme/#description_4","title":"DESCRIPTION","text":"Remote signaltap is software tool used for debug RTL (AFU), effectively a signal trace capability that Quartus places into a green bitstream. Remote Signal Tap provides access the RST part of the Port MMIO space, and then runs the remote protocol on top.
"},{"location":"sw/fpga_tools/readme/#examples_4","title":"EXAMPLES","text":"./mmlink -B 0x5e -P 3333
MMLink app starts and listens for connection.
"},{"location":"sw/fpga_tools/readme/#options_1","title":"OPTIONS","text":"-B,--bus FPGA Bus number.
-D,--device FPGA Device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-P,--port TCP port number.
-I,--ip IP address of FPGA system.
"},{"location":"sw/fpga_tools/readme/#notes","title":"NOTES","text":"Driver privilege:
Change AFU driver privilege to user .
command: chmod 777 /dev/intel-fpga-port.0
set memlock:
command: ulimit -l 10000
"},{"location":"sw/fpga_tools/readme/#coreidle","title":"coreidle","text":""},{"location":"sw/fpga_tools/readme/#name_5","title":"NAME","text":"coreidle - idles cores for shared TDP sockets to run online cores at maximum capacity.
"},{"location":"sw/fpga_tools/readme/#synopsis_5","title":"SYNOPSIS","text":"coreidle [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-G <GBS path>]
"},{"location":"sw/fpga_tools/readme/#description_5","title":"DESCRIPTION","text":"This tools parses input GBS, extracts power from metadata ,calculates fpga power, number of online and idle cores. It moves threads from idle cores to online cores.
"},{"location":"sw/fpga_tools/readme/#examples_5","title":"EXAMPLES","text":"./coreidle -B 0x5e -G /home/lab/gbs/mode0.gbs
Idle cores to run online cores at maximum capacity.
"},{"location":"sw/fpga_tools/readme/#options_2","title":"OPTIONS","text":"-B,--bus FPGA Bus number.
-D,--device FPGA Device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-G,--gbs Green bitstream file path.
"},{"location":"sw/fpga_tools/readme/#fpgamux","title":"fpgamux","text":""},{"location":"sw/fpga_tools/readme/#name_6","title":"NAME","text":"fpgamux - Software MUX for running multiple AFU (accelerator functional unit) tests in one GBS (green bitsream)\n
"},{"location":"sw/fpga_tools/readme/#synopsis_6","title":"SYNOPSIS","text":"fpgamux [-h] [-S|--socket-id SOCKET_ID] [-B|--bus-number BUS] [-D|--device DEVICE] [-F|--function FUNCTION]\n [-G|--guid GUID] -m|--muxfile MUXFILE.json\n
"},{"location":"sw/fpga_tools/readme/#description_6","title":"DESCRIPTION","text":"fpgamux is a testing tool to interact with multiple AFUs that have been synthesized into one GBS along with the CCIP-MUX BBB (basic building block). The CCIP-MUX uses upper bits in the MMIO addresses to route MMIO reads/writes to the AFU running on the corresponding CCIP-MUX port. fpgamux uses a configuration file that lists the software components and configuration to use.
.. note::
Only one (the first) AFU is discoverable by the OPAE driver. Enumerating acceleration on an FPGA will find\n the accelerator associated with the first AFU only. The first software component in the configuration will\n be used to determine the GUID to use for enumeration. This can be overridden with the -G|--guid option.\n
"},{"location":"sw/fpga_tools/readme/#options_3","title":"OPTIONS","text":"-S SOCKET_ID, --socket-id SOCKET_ID\n socket id of FPGA resource\n\n-B BUS, --bus BUS\n bus id of FPGA resource\n\n-D DEVICE, --device DEVICE\n device id of FPGA resource\n\n\n-F FUNCTION, --function FUNCTION\n function id of FPGA resource\n\n-G, --guid\n specify what guid to use for the accelerator enumeration\n
"},{"location":"sw/fpga_tools/readme/#configuration","title":"CONFIGURATION","text":"fpgamux uses a configuration file (in JSON format) to determine what software components to instantiate and how to configure them for interacting with the AFUs in the GBS. This schema for this is listed below:
[\n {\n \"app\" : \"fpga_app\",\n \"name\" : \"String\",\n \"config\" : \"Object\"\n }\n]\n
"},{"location":"sw/fpga_tools/readme/#examples_6","title":"EXAMPLES","text":"An example configuration with two components is listed below:
[\n {\n \"app\" : \"nlb0\",\n \"name\" : \"nlb0\",\n \"config\" :\n {\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"cont\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n },\n {\n \"app\" : \"nlb3\",\n \"name\" : \"nlb3\",\n \"config\" :\n {\n \"mode\" : \"read\",\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"strided-access\" : 1,\n \"cont\" : false,\n \"warm-fpga-cache\" : false,\n \"cool-fpga-cache\" : false,\n \"cool-cpu-cache\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"alt-wr-pattern\" : false,\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n }\n]\n
"},{"location":"sw/fpga_tools/readme/#userclk","title":"userclk","text":""},{"location":"sw/fpga_tools/readme/#name_7","title":"NAME","text":"userclk - to set afu high and low clock frequency.
"},{"location":"sw/fpga_tools/readme/#synopsis_7","title":"SYNOPSIS","text":"userclk [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-P <Port id>] [-H <User clock high frequency>] -L <User clock low frequency>]
"},{"location":"sw/fpga_tools/readme/#description_7","title":"DESCRIPTION","text":"userclk tool used to set high and low clock frequency to acceleration function unit.
"},{"location":"sw/fpga_tools/readme/#examples_7","title":"EXAMPLES","text":"./userclk -B 0x5e -H 400 -L 200
Sets AFU frequency.
"},{"location":"sw/fpga_tools/readme/#options_4","title":"OPTIONS","text":"-B,--bus FPGA Bus number.
-D,--device FPGA Device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-P,--port Port id.
-H,--freq-high User clock high frequency.
-L,--freq-low User clock low frequency.
"},{"location":"sw/fpga_tools/coreidle/coreidle/","title":"coreidle","text":""},{"location":"sw/fpga_tools/coreidle/coreidle/#synopsis","title":"SYNOPSIS","text":"coreidle [-v] [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-G <GBS path>]
"},{"location":"sw/fpga_tools/coreidle/coreidle/#description","title":"DESCRIPTION","text":"coreidle parses the Accelerator Function Unit (AFU) metadata and extracts power information. coreidle calculates the FPGA power and calculates the number of online and idle cores. It moves threads from idle cores to online cores. coreidle is only available the Integrated FPGA Platform. You cannot run coreidle on the PCIe Accelerator Card (PAC).
"},{"location":"sw/fpga_tools/coreidle/coreidle/#examples","title":"EXAMPLES","text":"./coreidle -B 0x5e -G /home/lab/gbs/mode0.gbs
Idle cores to run online cores at maximum capacity.
"},{"location":"sw/fpga_tools/coreidle/coreidle/#options","title":"OPTIONS","text":"-v,--version Prints version information and exit.
-B,--bus FPGA bus number.
-D,--device FPGA device number.
-F,--functio FPGA function number.
-S,--socket FPGA socket number.
-G,--gbs Green bitstream file path.
"},{"location":"sw/fpga_tools/fecmode/fecmode/","title":"fecmode (N3000 specific tool)","text":""},{"location":"sw/fpga_tools/fecmode/fecmode/#synopsis","title":"SYNOPSIS","text":"fecmode [<mode>][<args>]\n
"},{"location":"sw/fpga_tools/fecmode/fecmode/#description","title":"DESCRIPTION","text":"Fecmode changes FEC mode of external ethernet PHY, this tool only support on N3000 Card.
"},{"location":"sw/fpga_tools/fecmode/fecmode/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"--segment, -S
segment number of the PCIe device.
--bus, -B
bus number of the PCIe device.
--device, -D
device number of the PCIe device.
--function, -F function number of the PCIe device
--rsu, -r reboot card only if mode is not configured
--debug, -d output debug information
"},{"location":"sw/fpga_tools/fecmode/fecmode/#fec-mode","title":"FEC Mode","text":"no no FEC.
kr BaseR FEC (Fire-Code) correction \u2013 4 orders
rs Reed-Solomon FEC correction \u2013 7 orders
"},{"location":"sw/fpga_tools/fecmode/fecmode/#example","title":"EXAMPLE","text":"This command change FEC mode to \u201ckr\u201d:
# fecmode -B 0x25 kr\n
This command reboot card (no need to specify bus number if there is only one card):
# fecmode -r\n
This command display the current FEC mode:
# fecmode\n
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/","title":"fpgabist","text":""},{"location":"sw/fpga_tools/fpgabist/fpgabist/#synopsis","title":"SYNOPSIS","text":"fpgabist [-h] [-i device_id] [-b bus] [-d device] [-f function] [path_to_gbs1 path_to_gbs2 ...]\n
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#description","title":"DESCRIPTION","text":"The fpgabist tool performs self-diagnostic tests on supported FPGA platforms.
The tool accepts one or more Accelerator Function (AF) binaries from a predetermined set of AFs. Depending on the available binaries, the tool runs appropriate tests and reports hardware issues.
fpgabist always uses fpgainfo to report system information before running any hardware tests.
Currently, fpgabist accepts the following AFs: 1. nlb_mode_3: The native loopback (NLB) test implements a loopback from TX to RX. Use it to verify basic functionality and to measure bandwidth. 2. dma_afu: The direct memory access (DMA) AFU test transfers data from host memory to FPGA-attached local memory.
The installation includes the AF files, but you can also compile the AFs from the source.
If there are multiple PCIe\u00ae devices, use -b, -d, -f to specify the BDF for the specific PCIe\u00ae device.
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"[path_to_gbs1 path_to_gbs2 ...]
Paths to Accelerator Function (AF) files.
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"You can use the single letter or the full parameter name for the command line arguments.
-h, --help
Prints usage information
-i device_id, --device-id device_id
Device ID for Intel FPGA. Default is: 0x09c4
-B bus, --bus bus
Bus number for specific FPGA
-D device, --device device
Device number for specific FPGA
-F function, --function function
Function number for specific FPGA
"},{"location":"sw/fpga_tools/fpgabist/fpgabist/#examples","title":"EXAMPLES","text":"fpgabist <path_to_gbs_files>/dma_afu.gbs <path_to_gbs_files>/nlb_3.gbs
Runs fpgabist on any platform in the system that matches the default device ID. This command runs both the DMA and NLB_MODE_3 tests.
fpgabist -i 09c4 -b 5 <path to gbs>/dma_afu.gbs
Runs fpgabist the DMA test on the PCIe\u00ae Endpoint with device_id 09c4 on bus 5.
"},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/","title":"fpgaconf","text":""},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/#synopsis","title":"SYNOPSIS","text":"fpgaconf [-hvVn] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR] <gbs>
"},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/#description","title":"DESCRIPTION","text":"fpgaconf configures the FPGA with the accelerator function (AF). It also checks the AF for compatibility with the targeted FPGA and the FPGA Interface Manager (FIM). fpgaconf takes the following arguments:
-h, --help
Prints usage information.\n
-v, --version
Prints version information and exits.\n
-V, --verbose
Prints more verbose messages while enumerating and configuring. Can be\nrequested more than once.\n
-n, --dry-run
Performs enumeration. Skips any operations with side-effects such as the\nactual AF configuration.\n
-S, --segment
PCIe segment number of the target FPGA.\n
-B, --bus
PCIe bus number of the target FPGA.\n
-D, --device
PCIe device number of the target FPGA.\n
-F, --function
PCIe function number of the target FPGA.\n
--force
Reconfigure the AFU even if it is in use.\n
fpgaconf enumerates available FPGA devices in the system and selects compatible FPGAs for configuration. If more than one FPGA is compatible with the AF, fpgaconf exits and asks you to be more specific in selecting the target FPGAs by specifying a a PCIe BDF.
"},{"location":"sw/fpga_tools/fpgaconf/fpgaconf/#examples","title":"EXAMPLES","text":"fpgaconf my_af.gbs
Program \"my_af.gbs\" to a compatible FPGA.\n
fpgaconf -V -B 0x3b my_af.gbs
Program \"my_af.gbs\" to the FPGA in bus 0x3b, if compatible,\nwhile printing out slightly more verbose information.\n
fpgaconf 0000:3b:00.0 my_af.gbs
Program \"my_af.gbs\" to the FPGA at address 0000:3b:00.0.\n
"},{"location":"sw/fpga_tools/fpgad/fpgad/","title":"fpgad","text":""},{"location":"sw/fpga_tools/fpgad/fpgad/#synopsis","title":"SYNOPSIS","text":"fpgad --daemon [--version] [--directory=<dir>] [--logfile=<file>] [--pidfile=<file>] [--umask=<mode>] [--socket=<sock>] [--null-bitstream=<file>] fpgad [--socket=<sock>] [--null-bitstream=<file>]
"},{"location":"sw/fpga_tools/fpgad/fpgad/#description","title":"DESCRIPTION","text":"fpgad monitors the device sensors, checking for sensor values that are out of the prescribed range.
When any of the sensors is detected to be out of bounds, fpgad will focus on keeping the server from rebooting by masking PCIE AER, and send a message to system administrator. System administrator can take further actions like stop the application and stop the FPGA, but fpgad just focus on monitor the sensors and will not take any cooling actions.
Note: fpgad must be running (as root) and actively monitoring devices when a sensor anomaly occurs in order to initiate Graceful Shutdown. If fpgad is not loaded during such a sensor anomaly, the out-of-bounds scenario will not be detected, and the resulting effect on the hardware is undefined.
"},{"location":"sw/fpga_tools/fpgad/fpgad/#arguments","title":"ARGUMENTS","text":"-v, --version
Prints version information and exits.\n
-d, --daemon
When specified, fpgad executes as a system daemon process.\n
-D, --directory <dir>
When running in daemon mode, run from the specified directory.\nIf omitted when daemonizing, `fpgad` uses /tmp.\n
-l, --logfile <file>
When running in daemon mode, send output to file. When not in daemon mode, the output goes to stdout.\nIf omitted when daemonizaing, fpgad uses /tmp/fpgad.log.\n
-p, --pidfile <file>
When running in daemon mode, write the daemon's process id to a file.\nIf omitted when daemonizing, fpgad uses /tmp/fpgad.pid.\n
-m, --umask <mode>
When running in daemon mode, use the mode value as the file mode creation mask passed to umask.\nIf omitted when daemonizing, fpgad uses 0.\n
-s, --socket <sock>
Listen for event API registration requests on the UNIX domain socket on the specified path. \nThe default=/tmp/fpga_event_socket.\n
-n, --null-bitstream <file>
Specify the NULL bitstream to program when an AP6 event occurs. This option may be specified multiple\ntimes. The AF, if any, that matches the FPGA's PR interface ID is programmed when an AP6\nevent occurs.\n
"},{"location":"sw/fpga_tools/fpgad/fpgad/#troubleshooting","title":"TROUBLESHOOTING","text":"If you encounter any issues, you can get debug information in two ways:
- By examining the log file when in daemon mode.
- By running in non-daemon mode and viewing stdout.
"},{"location":"sw/fpga_tools/fpgad/fpgad/#examples","title":"EXAMPLES","text":"fpgad --daemon --null-bitstream=my_null_bits.gbs
This command starts fpgad as a system daemon process:
sudo systemctl start fpgad\n
"},{"location":"sw/fpga_tools/fpgadiag/","title":"fpgadiag","text":""},{"location":"sw/fpga_tools/fpgadiag/#synopsis","title":"SYNOPSIS","text":"fpgadiag [-m | --mode=] <mode> [-t | --target=] <target> [options]\n
"},{"location":"sw/fpga_tools/fpgadiag/#description","title":"DESCRIPTION","text":"Includes several tests to diagnose, test, and report on the FPGA hardware.
<mode> chooses which test to run. <target> specifies the platform that runs the test. <target> can be either fpga or ase where ase. <ase> is the abbreviation for Accelerator Simulation Environment.
The <mode> selects from the following tests:
lpbk1
This test runs a loopback test on the number of cachelines specified with the BEGIN option. fpgadiag sets up source and destination buffers in main memory. The FPGA then performs a memcpy from a source buffer to the destination buffer, one cacheline at a time.
A cacheline is 64 bytes. When BEGIN = END, the test performs one iteration. When BEGIN = END + x, the test performs x iterations. The first iteration consists of copying BEGIN cachelines; the second iteration consists of copying BEGIN+1 cache lines. The third iteration consists of copying BEGIN+2 cache lines, and so on.
The latency is shown as the number of clock cycles.
When you specify MULTI-CL, you copy MULTI-CL cache lines at a time. The WR-FENCE chooses on which virtual channel the WrFence occurs.
If you specify continuous mode with --cont, the program iterates until the timeout specified in TIMEOUT completes.
read
This test performs reads. Use this test to measure read bandwidth.
write
This test performs writes. Use it to measure write bandwidth.
trput
This test measures both read and write bandwidth by performing 50% read and 50% write tests.
sw
This is a send-and-respond (ping-pong) test. One side sends data and waits for response.
Each test requires a particular AF. Before running a test, make sure the required AF is properly configured on the platform.
- The lpbk1 test requires the nlb mode 0 AF.
- The trput test requires the nlb mode 3 AF.
- The sw test requires the nlb mode 7 AF. This AF is only available for the integrated FPGA platform. You cannot run it on the PCIe accelerator card (PAC).
fpgalpbk
This enable/disable FPGA loopback.
fpgastats
This get fpga mac statistics.
mactest
This compare mac addresses that read from MAC ROM with mac addresses read from Host side.
"},{"location":"sw/fpga_tools/fpgadiag/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/fpgadiag/#common-options","title":"Common options","text":"--help, -h
Print help information and exit.\n
--target=, -t
This switch specifies fpga (hardware) or ase (simulation). The default=fpga.\n
--mode=, -m
The test to run. The valid values are `lpbk1`, `read`,\n`write`, `trput`, and `sw`.\n
--config=, -c
A configuration file in the JSON format that specifies options for a test.\nIf an option is specified both in the configuration file and on the command \nline, the value in the configuration file takes precedence.\n
--dsm-timeout-usec
Timeout in microseconds for test completion. The test fails if not completed by \nspecified timeout. The default=1000000.\n
--socket-id=, -s
Socket ID encoded in FPGA Interface Manager (FIM). The default=0.\n
--bus=, -B
Bus number of the PCIe device. The default=0.\n
--device=, -D
Device number of the PCIe device. The default=0.\n
--function=, -F
Function number of the PCIe device. The default=0.\n
--freq=, -T
Clock frequency (in Hz) used for bandwidth calculation. The default=400000000 Hz (400 MHz).\n
eval_rst .. note:: This frequency is used only when the software cannot infer the frequency from the accelerator.
--suppress-hdr, -S
Suppress column headers for text output. The default=off.\n
--csv, -V
Comma separated value format. The default=off.\n
--suppress-stats
Suppress statistics output at the end of test. The default=off.\n
"},{"location":"sw/fpga_tools/fpgadiag/#lpbk1-test-options","title":"lpbk1 test options","text":"--guid=, -g
AFU ID to enumerate. The default=D8424DC4-A4A3-C413-F89E-433683F9040B.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I The default=wrline-M.\n
--cache-hint=, -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. The default=auto.\n
"},{"location":"sw/fpga_tools/fpgadiag/#read-test-options","title":"read test options","text":"--guid=, -g
AFU ID to enumerate. The default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--strided-access=S, -a
1<= S <= 64. The default=1.\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-hint=, -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Try to prime the cache with hits. The default=off. Try to prime the \ncache with misses. The default=off.\n
--cool-cpu-cache, -C
Try to prime the cpu cache with misses. The default=off.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. The default=auto\n
"},{"location":"sw/fpga_tools/fpgadiag/#write-test-options","title":"write test options","text":"--guid=, -g
AFU ID to enumerate. The default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18\n
--begin=B, -b
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--strided-access=S, -a
1<= S <= 64. The default=1.\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I The default=wrline-M\n
--warm-fpga-cache -H; --cool-fpga-cache -M
Try to prime the cache with hits. The default=off. Try to prime the \ncache with misses. The default=off.\n
--cool-cpu-cache, -C
Try to prime the cpu cache with misses. The default=off.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1, random. The default=`WRITE-VC`.\n
--alt-wr-pattern, -l
Alternate Write Pattern. The default=off.\n
"},{"location":"sw/fpga_tools/fpgadiag/#trput-test-options","title":"trput test options","text":"--guid=, -g
AFU ID to enumerate. The default=F7DF405C-BD7A-CF72-22F1-44B0B93ACD18.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--multi-cl=M, -u
M can equal 1, 2, or 4. The default=1.\n
--strided-access=S, -a
1<= S <= 64. The default=1\n
--cont, -L
Continuous mode. The default=off.\n
--timeout-usec=, --timeout-msec=, --timeout-sec=, --timeout-min=, --timeout-hour=
timeout for --cont mode. The default for all options is 0.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I The default=wrline-M.\n
--cache-hint=, -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random. The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. The default=`WRITE-VC`.\n
"},{"location":"sw/fpga_tools/fpgadiag/#sw-test-options","title":"sw test options","text":"--guid=, -g
AFU ID to enumerate. The default=7BAF4DEA-A57C-E91E-168A-455D9BDA88A3.\n
--begin=B, -b
1 <= B <= 65535. The default=1, B = number of cache lines.\n
--end=E, -e
1 <= E <= 65535. The default=B, B and E designate number of cache lines.\n
--cache-policy=, -p
Can be wrline-I, wrline-M, or wrpush-I. The default=wrline-M.\n
--cache-hint= -i
Can be rdline-I or rdline-S. The default=rdline-I.\n
--read-vc=, -r
Can be auto, vl0, vh0, vh1, random The default=auto.\n
--write-vc=, -w
Can be auto, vl0, vh0, vh1, random The default=auto.\n
--wrfence-vc=, -f
Can be auto, vl0, vh0, vh1. The default=`WRITE-VC`.\n
--notice=, -N
Can be poll or csr-write. The default=poll.\n
"},{"location":"sw/fpga_tools/fpgadiag/#enable-fpga-n3000-ethernet-group-vfio-mdev","title":"Enable FPGA N3000 Ethernet group VFIO mdev","text":"FPGA DFL driver does not support any ioctls to read/write ethernet group info and registers. Users can read/write eth group registers by enabling VFIO mdev. Unbind the dfl_eth_group driver and bind vfio-mdev-dfl driver for ethernet group dfl-device; then userspace can take full control of ethernet group feature id 10.
Ethernet group must be enabled before running fpgalpbk, mactest tools.
"},{"location":"sw/fpga_tools/fpgadiag/#steps-to-enablecreate-vfio-mdev","title":"Steps to enable/create vfio mdev","text":"unbind eth group feature id 10:\n echo dfl-fme.0.8 > /sys/bus/dfl/drivers/dfl-eth-group/unbind\n echo dfl-fme.0.7 > /sys/bus/dfl/drivers/dfl-eth-group/unbind\nbind to vfio-mdev-dfl:\n echo vfio-mdev-dfl > /sys/bus/dfl/devices/dfl-fme.0.7/driver_override\n echo vfio-mdev-dfl > /sys/bus/dfl/devices/dfl-fme.0.8/driver_override\nload vfio driver:\n modprobe vfio_pci\n modprobe vfio_iommu_type1\n modprobe vfio_mdev\n modprobe vfio_mdev_dfl\ntrigger mdev:\n echo dfl-fme.0.7 >/sys/bus/dfl/drivers_probe\n echo dfl-fme.0.8 >/sys/bus/dfl/drivers_probe\n echo 83b8f4f2-509f-382f-3c1e-e6bfe0fa1001 > /sys/bus/dfl/devices/dfl-fme.0.7/mdev_supported_types/vfio-mdev-dfl-1/create\n echo 83b8f4f2-509f-382f-3c1e-e6bfe0fa1002 > /sys/bus/dfl/devices/dfl-fme.0.8/mdev_supported_types/vfio-mdev-dfl-1/create\n\nlinux kerenl msg after enabling mdev:\n i40e 0000:b3:00.0 eth1: NIC Link is Down\n i40e 0000:b1:00.1 eth0: NIC Link is Down\n vfio-mdev-dfl dfl-fme.2.7: MDEV: Registered\n vfio-mdev-dfl dfl-fme.2.8: MDEV: Registered\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1005: Adding to iommu group 140\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1005: MDEV: group_id = 140\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1006: Adding to iommu group 141\n vfio_mdev 83b8f4f2-509f-382f-3c1e-e6bfe0fa1006: MDEV: group_id = 141\n
"},{"location":"sw/fpga_tools/fpgadiag/#remove-vfio-mdev","title":"Remove vfio mdev","text":" echo 1 | sudo tee /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1002/remove\n echo 1 | sudo tee /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove\n\n rmmod vfio_mdev_dfl\n modprobe dfl_eth_group\n\n echo dfl-fme.0.7 >/sys/bus/dfl/drivers_probe\n echo dfl-fme.0.8 >/sys/bus/dfl/drivers_probe\n\n echo dfl-eth-group > /sys/bus/dfl/devices/dfl-fme.0.7/driver_override\n echo dfl-eth-group > /sys/bus/dfl/devices/dfl-fme.0.8/driver_override\n
"},{"location":"sw/fpga_tools/fpgadiag/#fpgalpbk-test-options","title":"fpgalpbk test options","text":"--enable
Enable fpga phy loopback.\n
--disable
Disable fpga phy loopback.\n
--direction
Can be local, remote.\n
--type
Can be serial, precdr, postcdr.\n
--side
Can be line, host.\n
--port
0 <= port <= 7, the default is all.\n
"},{"location":"sw/fpga_tools/fpgadiag/#mactest-test-options","title":"mactest test options","text":"--offset
Read mac addresses from an offset, The default=0.\n
"},{"location":"sw/fpga_tools/fpgadiag/#examples","title":"EXAMPLES","text":"This command starts a lpbk1 test for the FPGA on bus 0x5e. The test copies 57535, 57536, 57537 ... up to 65535 cache lines, one line at a time. The test prints output in the comma separated values (CSV) format with the header suppressed.
./fpgadiag --mode=lpbk1 --target=fpga -V --bus=0x5e --begin=57535\n--end=65535 --cache-hint=rdline-I --cache-policy=wrpush-I --multi-cl=1\n--write-vc=vl0 --read-vc=vh1 --wrfence-vc=auto\n
This command starts a read test on the FPGA located on bus 0xbe. The test reads 2045 cache lines in the continuous mode with a 15-second timeout period. The reads use a strided pattern with a 10-byte stride length.
./fpgadiag --mode=read --target=fpga -V --bus=0xbe --begin=2045 --cont\n--timeout-sec=15 --cache-hint=rdline-I --multi-cl=1 -a=10 \n--read-vc=auto --wrfence-vc=auto\n
This command starts a sw test on the FPGA located on bus 0xbe. The test signals completion using a CSR write.
./fpgadiag --mode=sw --target=fpga -V --bus=0xbe --begin=4 --end=8192\n--cache-hint=rdline-I --cache-policy=wrline-I --notice=csr-write --write-vc=vl0\n--wrfence-vc=auto --read-vc=random \n
This command enable a fpgalpbk on the FPGA located on bus 0xbe.
./fpgadiag -m fpgalpbk --bus 0xbe --enable --direction local --type postcdr\n--side host\n
This command show fpgastats on the FPGA located on bus 0xbe.
./fpgadiag -m fpgastats --bus 0xbe\n
"},{"location":"sw/fpga_tools/fpgadiag/#troubleshooting","title":"TROUBLESHOOTING","text":"When a test fails to run or gives errors, check the following:
- Is the Intel FPGA driver properly installed? See Installation Guide for driver installation instructions.
- Are FPGA port permissions set properly? Check the permission bits of the port, for example,
/dev/intel-fpga-port-0. You need READ and WRITE permissions to run fpgadiag tests. - Is hugepage properly configured on the system? See Installation Guide for hugepage configuration steps. In particular,
fpgadiag requires a few 1 GB pages. - Is the required AFU loaded? See DESCRIPTION for information about what AFU the test requires.
- Are
--begin and --end values set properly? --end must be larger than the --begin. Also, --begin must be a multiple of the --multi-cl value. - The
--warm-fpga-cache and --cool-fpga-cache options in the read and write tests are mutually exclusive. - The timeout options are only meaningful for the continuous mode (with the
--cont option).
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/","title":"fpgaflash","text":""},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#synopsis","title":"SYNOPSIS","text":"fpgaflash [-h] {user,factory} file [bdf]\n
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#description","title":"DESCRIPTION","text":"fpgaflash updates the static FIM image loaded from flash at power-on.
If there are multiple devices in the system, fpgaflash must specify a BDF to select the correct device. If no BDF is specified, fpgaflash prints out the BDFs of any compatible devices.
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"{user, factory}
Specifies the type of flash programming.
user
Only reprograms the user image in flash.
factory
Reprograms the entire flash. A catastrophic failure during a factory update such as a power outage requires a USB cable and quartus_pgm to recover.
file
Specifies the Raw Programming Data File (rpd) to program into flash.
bdf
Specifies the bus, device and function (BDF) of device to program such as 04:00.0 or 0000:04:00.0. This flag is optional when there is a single device in the system.
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Print usage information.
"},{"location":"sw/fpga_tools/fpgaflash/fpgaflash/#example","title":"EXAMPLE","text":"fpgaflash user new_image.rpd 0000:04:00.0
Programs new_image.rpd to flash of device with BDF 0000:04:00.0.
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/","title":"super-rsu","text":""},{"location":"sw/fpga_tools/fpgaflash/superrsu/#synopsis","title":"SYNOPSIS","text":"super-rsu [-h] [-n] [--verify] | [ [--log-level {trace,debug,error,warn,info,notset}]\n [--log-file <filename>] [--rsu-only] [--with-rsu] [--force-flash] ]\n rsu_config\n
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#description","title":"DESCRIPTION","text":"super-rsu is a tool that can be used for flashing image files and commanding an Intel PAC device to perform RSU (remote system update - or a board reboot). Performing an RSU on an Intel PAC device will cause it to reload any firmware or programmable logic and restart its execution, a requirement for any updated firmware or programmable logic to take effect.
At the core of super-rsu is its configuration file (referred to in this document as 'rsu_config') which is essentially a manifest file for identifying both the target device and the binary images (and their versions) to be flashed.
At a high level, the flow of super-rsu should be: 1. Read and parse rsu_config file 2. Use product identifiers (like vendor, device and any additional vendor, device pairs that may be present in the PCIe bus) to locate all compatible devices on the PCIe bus. 3. For every device found on the system, update the device using the flash images defined in the \"flash\" section in the rsu_config data (or nvmupdate section). Each item in the \"flash\" section is a \"flash spec\" that contains: * The flash type (\"user\", \"bmc_fw\", \"bmc_img\", ...) * The filename of the image to flash. super-rsu will look for this file first in the same directory of the rsu_config file, and then look in the current working directory. * The version of the image. * An optional \"force\" indicator * An optional \"requires\" indicator The \"nvmupdate\" section is used to describe an Ethernet firmware file and its version. 4. Using the data in the \"nvmupdate\" and \"flash\" sections, the update routine involves: * If an \"nvmupdate\" section is present: 1. Locate the file on the file system to use to flash the Ethernet device. 2. Call nvmupdate to get an \"inventory\" of devices matching the vendor and device id in this section. 3. Use this data to dynamically generate an nvmupdate compatible configuration file. 4. Call nvmupdate with the generated configuration file to flash the Ethernet interfaces in the Vista Creek card (if version reported by system does not match the version in this section). * For each spec in the \"flash\" section: 1. Locate the file on the file system to use to flash. 2. Compare the version listed in the \"flash spec\" to version reported by the target component. 3. Create a task to call fpgaflash if either of the following conditions is met (and the revision specified is compatible): * The \"force\" indicator is present and set to true. * The version in the spec does not match the version reported by the system OR the flash type is factory type. * For each task created from the \"flash\" section: 1. Call fpgaflash with the command line arguments that correspond to the flash type and the file name in the spec used to create the task. This opens and controls the execution of fpgaflash in another process.
NOTE: If the system reports a revision for one of the components being flashed, this revision must be in the set of revisions listed in the manifest. Example: if the system reports 'a' for bmc_img and the manifest includes 'ab', then the image will be flashed.
NOTE: Each update routine is run in a thread specific to a device located on the PCIe bus. Every task in an update routine involves opening a new process that is controlled and managed by its update routine thread. If a task includes a timeout and the timeout is reached, a termination request will be sent to its process and it will be counted as a failure. If a global timeout is reached in the main thread, a termination request will be sent to each thread performing the update. Consequently, the update routine will give the current task extra time before terminating the process. The RSU operation will only be performed if requested with either --with-rsu command line argument or with the --rsu-only command line argument. The former will perform the RSU command upon successful completion of flash operations. The latter will skip the process of version matching and flashing images and will only perform the RSU command. It is recommended that super-rsu be executed again if any flash operation is interrupted.
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"rsu config
Specifies the name of the file containing the RSU configuration (in JSON format)
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Print usage information.
--verify
Compare versions of flashable components on the system against the manifest. Return non-zero exit if compatible components are not up to date.
-n, --dry-run
Don't perform any updates, just a dry run. This will print out commands that can be executed in a Linux shell.
--log-level {trace,debug,error,warn,info,notset}
Log level to use. Default is 'info'.
`--log-file (default: /tmp/super-rsu.log)
Emit log messages (with DEBUG level) to filename NOTE: The default log file (/tmp/super-rsu.log) is set to rollover every time super-rsu is executed. This will create numbered backups before truncating the log file. The maximum number of backups is 50.
--rsu-only
Only perform the RSU command.
--with-rsu
Perform RSU after updating flash components(experimental)
--force-flash
Flash all images regardless of versions matching or not.
"},{"location":"sw/fpga_tools/fpgaflash/superrsu/#configuration","title":"CONFIGURATION","text":"The following is the JSON schema expected by super-rsu. Any deviance from this schema may result in errors executing super-rsu.
{\n\"definitions\": {},\n\"$schema\": \"http://json-schema.org/draft-07/schema#\",\n\"$id\": \"http://example.com/root.json\",\n\"type\": \"object\",\n\"title\": \"The Root Schema\",\n\"required\": [\n\"product\",\n\"vendor\",\n\"device\",\n\"flash\"\n],\n\"optional\": [\n\"nvmupdate\",\n],\n\"properties\": {\n\"product\": {\n\"$id\": \"#/properties/product\",\n\"type\": \"string\",\n\"title\": \"The Product Schema\",\n\"default\": \"\",\n\"examples\": [\n\"n3000\"\n],\n\"pattern\": \"^(.*)$\"\n},\n\"vendor\": {\n\"$id\": \"#/properties/vendor\",\n\"type\": \"string\",\n\"title\": \"The Vendor Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x8086\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"device\": {\n\"$id\": \"#/properties/device\",\n\"type\": \"string\",\n\"title\": \"The Device Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x0b30\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"nvmupdate\": {\n\"$id\": \"#/properties/nvmupdate\",\n\"type\": \"object\",\n\"title\": \"The nvmupdate Schema\",\n\"required\": [\n\"vendor\",\n\"device\",\n\"filename\",\n\"version\"\n],\n\"optional\": [\n\"interfaces\"\n],\n\"properties\": {\n\"vendor\": {\n\"$id\": \"#/properties/nvmupdate/vendor\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Vendor Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x8086\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"device\": {\n\"$id\": \"#/properties/nvmupdate/device\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Device Schema\",\n\"default\": \"\",\n\"examples\": [\n\"0x0d58\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{4})$\"\n},\n\"interfaces\": {\n\"$id\": \"#/properties/nvmupdate/interfaces\",\n\"type\": \"number\",\n\"title\": \"The nvmupdate Interfaces Schema\",\n\"default\": \"1\",\n\"examples\": [\n2, 4\n]\n},\n\"filename\": {\n\"$id\": \"#/properties/nvmupdate/filename\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Filename Schema\",\n\"default\": \"\",\n\"examples\": [\n\"PSG_XL710_6p80_XLAUI_NCSI_CFGID2p61_Dual_DID_0D58_800049C6.bin\"\n],\n\"pattern\": \"^(.*)$\"\n},\n\"version\": {\n\"$id\": \"#/properties/nvmupdate/version\",\n\"type\": \"string\",\n\"title\": \"The nvmupdate Version Schema\",\n\"default\": \"\",\n\"examples\": [\n\"800049C6\"\n],\n\"pattern\": \"^((0x)?[A-Fa-f0-9]{8})$\"\n},\n\"timeout\": {\n\"$id\": \"#/properties/nvmupdate/timeout\",\n\"type\": \"string\",\n\"title\": \"The Timeout Schema\",\n\"default\": \"\",\n\"examples\": [\n\"10m\"\n],\n\"pattern\": \"^([0-9]+(\\\\.[0-9]+)?([dhms]))+$\"\n}\n}\n},\n\"flash\": {\n\"$id\": \"#/properties/flash\",\n\"type\": \"array\",\n\"title\": \"The Flash Schema\",\n\"items\": {\n\"$id\": \"#/properties/flash/items\",\n\"type\": \"object\",\n\"title\": \"The Items Schema\",\n\"required\": [\n\"filename\",\n\"type\",\n\"version\",\n\"revision\"\n],\n\"optional\": [\n\"enabled\",\n\"force\",\n\"timeout\",\n\"requires\"\n],\n\"properties\": {\n\"enabled\": {\n\"$id\": \"#/properties/flash/items/properties/enabled\",\n\"type\": \"boolean\",\n\"title\": \"The Enabled Schema\",\n\"default\": \"true\"\n},\n\"filename\": {\n\"$id\": \"#/properties/flash/items/properties/filename\",\n\"type\": \"string\",\n\"title\": \"The Filename Schema\",\n\"default\": \"\",\n\"examples\": [\n\"vista_creek_qspi_xip_v1.0.6.ihex\"\n],\n\"pattern\": \"^(.*)$\"\n},\n\"type\": {\n\"$id\": \"#/properties/flash/items/properties/type\",\n\"type\": \"string\",\n\"title\": \"The Type Schema\",\n\"default\": \"\",\n\"examples\": [\n\"bmc_fw\"\n],\n\"enum\": [\"user\", \"bmc_fw\", \"bmc_img\", \"dtb\", \"factory_only\",\n\"phy_eeprom\"]\n},\n\"version\": {\n\"$id\": \"#/properties/flash/items/properties/version\",\n\"type\": \"string\",\n\"title\": \"The Version Schema\",\n\"default\": \"\",\n\"examples\": [\n\"1.0.6\"\n],\n\"pattern\": \"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+$\"\n},\n\"force\": {\n\"$id\": \"#/properties/flash/items/properties/force\",\n\"type\": \"boolean\",\n\"title\": \"The Force Schema\",\n\"default\": false,\n\"examples\": [\ntrue\n]\n},\n\"revision\": {\n\"$id\": \"#/properties/flash/items/properties/revision\",\n\"type\": \"string\",\n\"title\": \"The Revision Schema\",\n\"default\": \"\",\n\"examples\": [\n\"C\"\n],\n\"pattern\": \"^([A-Za-z])$\"\n},\n\"timeout\": {\n\"$id\": \"#/properties/nvmupdate/timeout\",\n\"type\": \"string\",\n\"title\": \"The Timeout Schema\",\n\"default\": \"\",\n\"examples\": [\n\"10m\"\n],\n\"pattern\": \"^([0-9]+(\\.[0-9]+)?([dhms]))+$\"\n},\n\"requires\": {\n\"$id\": \"#/properties/flash/items/properties/requires\",\n\"type\": \"string\",\n\"title\": \"The Requires Schema\",\n\"default\": \"\",\n\"examples\": [\n\"bmc_img >= 1.0.12\"\n],\n\"pattern\": \"^(([a-z_]+) ((<>!=)?=) ([0-9a-z\\\\.]+)$\"\n}\n}\n}\n}\n}\n}\n
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/","title":"fpgainfo","text":""},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#synopsis","title":"SYNOPSIS","text":" fpgainfo [-h] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\n {errors,power,temp,fme,port,bmc,mac,phy,security}\n
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#description","title":"DESCRIPTION","text":"fpgainfo displays FPGA information derived from sysfs files. The command argument is one of the following: errors, power, temp, port, fme, bmc, phy or mac,security,events. Some commands may also have other arguments or options that control their behavior.
For systems with multiple FPGA devices, you can specify the BDF to limit the output to the FPGA resource with the corresponding PCIe configuration. If not specified, information displays for all resources for the given command.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#fpgainfo-commands","title":"FPGAINFO COMMANDS","text":"errors
Show/clear errors of an FPGA resource that the first argument specifies. fpgainfo displays information in human readable form.
Error Description Catfatal Errors Bit 8 indicates an Injected Fatal error, bit 11 indicates an Injected Catastrophic Error. Inject Errors [2:0] are mainly writeable bits. Can read back values. (FME) Next Error [59:0] 60 LSBs are taken from the given error register that was triggered second, [60:61] 0 = FME0 Error, 1 = PCIe0 Error. (FME) First Error [59:0] 60 LSBs are taken from the given error register that was triggered first, [60:61] 0 = FME0 Error, 1 = PCIe0 Error. FME Errors Error from Partial Reconfiguration Block reporting a FIFO Parity Error has occurred. Non-fatal Errors Bit 6 is used to advertise an Injected Warning Error. power
Show total the power in watts that the FPGA hardware consumes.
temp
Show FPGA temperature values in degrees Celcius.
port
Show information about the port such as the AFU ID of currently loaded AFU.
fme
Show information about the FPGA platform including the partial reconfiguration (PR) Interface ID, the OPAE version, and the FPGA Interface Manager (FIM) ID.
The User1/User2/Factory Image Info lines reflect the information of the image that is present in the Flash.
The Bitstream Id line reflects the information of the image that is programmed in the FPGA.
bmc
Show all Board Management Controller sensor values for the FPGA resource, if available.
phy
Show information about the PHY integrated in the FPGA, if available.
mac
Show information about the MAC address in ROM attached to the FPGA, if available.
security
Show information about the security keys, hashs and flash count, if available.
events
Show information about events and sensors, if available.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"--help, -h
Prints help information and exit.
--version, -v
Prints version information and exit.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#common-arguments","title":"COMMON ARGUMENTS","text":"The following arguments are common to all commands and are optional.
-S, --segment
PCIe segment number of resource.
-B, --bus
PCIe bus number of resource.
-D, --device
PCIe device number of resource.
-F, --function
PCIe function number of resource.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#errors-arguments","title":"ERRORS ARGUMENTS","text":"The first argument to the errors command specifies the resource type. It must be one of the following: fme,port,all
fme
Show/clear FME errors.
port
Show/clear PORT errors.
all
Show/clear errors for all resources.
The optional <command-args> arguments are:
--clear, -c
Clear errors for the given FPGA resource.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#phy-arguments","title":"PHY ARGUMENTS","text":"The optional <command-args> argument is:
--group, -G
Select which PHY group(s) information to show.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#events-arguments","title":"EVENTS ARGUMENTS","text":"The optional <command-args> argument is:
--list,-l
List boots (implies --all).
--boot,-b
Boot index to use, i.e: \u00a0\u00a0\u00a0\u00a00 for current boot (default). \u00a0\u00a0\u00a0\u00a01 for previous boot, etc.
--count,-c
Number of events to print.
--all,-a
Print all events.
--sensors,-s
Print sensor data too.
--bits,-i
Print bit values too.
--help,-h
Print this help.
"},{"location":"sw/fpga_tools/fpgainfo/fpgainfo/#examples","title":"EXAMPLES","text":"This command shows the current power telemetry:
./fpgainfo power\n
This command shows the current temperature readings:
./fpgainfo temp\n
This command shows FME resource errors:
./fpgainfo errors fme\n
This command clears all errors on all resources: ./fpgainfo errors all -c\n
This command shows information of the FME on bus 0x5e ./fpgainfo fme -B 0x5e\n
This command shows information of the FPGA security on bus 0x5e ./fpgainfo security -B 0x5e\n
This command shows all events and sensors information including sensor bits: ./fpgainfo events -asi\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/","title":"fpgamux","text":""},{"location":"sw/fpga_tools/fpgamux/fpgamux/#synopsis","title":"SYNOPSIS","text":"fpgamux [-h] [-S|--socket-id SOCKET_ID] [-B|--bus-number BUS] [-D|--device DEVICE] [-F|--function FUNCTION]\n [-G|--guid GUID] -m|--muxfile <filepath.json>\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#description","title":"DESCRIPTION","text":"fpgamux tests multiple AFUs that are synthesized into a single AFU along with the CCIP-MUX basic building block (BBB). The CCIP-MUX uses the upper bits in the MMIO addresses to route MMIO reads and writes to the AFU running on the corresponding CCIP-MUX port. fpgamux uses a configuration file that lists the software components and correct configuration. fpgamux only runs on the Integrated FPGA Platform. You cannot run it on the PCIe accelerator card (PAC).
.. note::
The OPAE driver discovers only the first AFU. The first software component in the configuration \n determines the GUID to use for enumeration. Use the -G|--guid option to override the GUID\n for the first software component.\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#options","title":"OPTIONS","text":"-S SOCKET_ID, --socket-id SOCKET_ID
socket id of FPGA resource.
-B BUS, --bus BUS
bus id of FPGA resource.
-D DEVICE, --device DEVICE
The device id of FPGA resource.
-F FUNCTION, --function FUNCTION
The function id of FPGA resource.
-G, --guid
Specifies the GUID to use for the resource enumeration.
-m, --muxfile <filepath.json>
The path to the fpgamux configuration file. This file must be in JSON format following the schema described below.
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#configuration","title":"CONFIGURATION","text":"fpgamux uses a configuration file (in JSON format) to determine what software components to instantiate and how to configure them to work with the AFUs. The schema includes the following elements:
[\n {\n \"app\" : \"fpga_app\",\n \"name\" : \"String\",\n \"config\" : \"Object\"\n }\n ]\n
"},{"location":"sw/fpga_tools/fpgamux/fpgamux/#examples","title":"EXAMPLES","text":"The following example shows a configuration with two components:
[\n {\n \"app\" : \"nlb0\",\n \"name\" : \"nlb0\",\n \"config\" :\n {\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"cont\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n },\n {\n \"app\" : \"nlb3\",\n \"name\" : \"nlb3\",\n \"config\" :\n {\n \"mode\" : \"read\",\n \"begin\" : 1,\n \"end\" : 1,\n \"multi-cl\" : 1,\n \"strided-access\" : 1,\n \"cont\" : false,\n \"warm-fpga-cache\" : false,\n \"cool-fpga-cache\" : false,\n \"cool-cpu-cache\" : false,\n \"cache-policy\" : \"wrline-M\",\n \"cache-hint\" : \"rdline-I\",\n \"read-vc\" : \"vh0\",\n \"write-vc\" : \"vh1\",\n \"wrfence-vc\" : \"write-vc\",\n \"alt-wr-pattern\" : false,\n \"timeout-usec\" : 0,\n \"timeout-msec\" : 0,\n \"timeout-sec\" : 1,\n \"timeout-min\" : 0,\n \"timeout-hour\" : 0,\n \"freq\" : 400000000\n }\n }\n ]\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/","title":"fpgaport","text":""},{"location":"sw/fpga_tools/fpgaport/fpgaport/#synopsis","title":"SYNOPSIS","text":"fpgaport [-h] [-N NUMVFS] [-X] [--debug] {assign,release} device [port]\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#description","title":"DESCRIPTION","text":"The fpgaport enables and disables virtualization. It assigns and releases control of the port to the virtual function (VF). By default, the driver assigns the port to the physical function (PF) in the non-virtualization use case.
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"{assign, release}
Action to perform.\n
device
The FPGA device being targeted with this action.\n
port
The number of the port.\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-N NUMVFS, --numvfs NUMVFS
Create NUMVFS virtual functions. The typical value is 1.\n
-X, --destroy-vfs
Destroy all virtual functions prior to assigning.\n
--debug
Display additional log information.\n
-h, --help
Print usage information.\n
"},{"location":"sw/fpga_tools/fpgaport/fpgaport/#example","title":"EXAMPLE","text":"fpgaport release /dev/dfl-fme.0 0
Release port 0 from physical function control.\n
fpgaport assign /dev/dfl-fme.0 0
Assign port 0 to physical function control.\n
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/","title":"fpgasupdate","text":""},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#synopsis","title":"SYNOPSIS","text":"fpgasupdate [--log-level=<level>] file [bdf]
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#description","title":"DESCRIPTION","text":"The fpgasupdate command implements a secure firmware update for the following programmable accelerator cards (PACs): * Intel\u00ae PAC with Intel Arria\u00ae 10 GX FPGA * Intel\u00ae FPGA PAC D5005 * Intel\u00ae PAC N3000 * Intel\u00ae FPGA SmartNIC N6001-PL with Intel® Agilex® FPGA * Intel\u00ae FPGA IPU F2000X-PL
--log-level <level>
Specifies the `log-level` which is the level of information output to your command tool.\nThe following seven levels are available: `state`, `ioctl`, `debug`, `info`, `warning`,\n`error`, `critical`. Setting `--log-level=state` provides the most verbose output.\nSetting `--log-level=ioctl` provides the second most information, and so on. The default\nlevel is `info`.\n
file
Specifies the secure update firmware file to be programmed. This file may be to program a\nstatic region (SR), programmable region (PR), root entry hash, key cancellation, or other\ndevice-specific firmware.\n
bdf
The PCIe® address of the PAC to program. `bdf` is of the form `[ssss:]bb:dd:f`,\ncorresponding to PCIe segment, bus, device, function. The segment is optional. If\nyou do not specify a segment, the segment defaults to `0000`. If the system has only\none PAC you can omit the `bdf` and let `fpgasupdate` determine the address\nautomatically.\n
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#troubleshooting","title":"TROUBLESHOOTING","text":"To gather more debug output, decrease the --log-level parameter.
"},{"location":"sw/fpga_tools/fpgasupdate/fpgasupdate/#examples","title":"EXAMPLES","text":"fpgasupdate firmware.bin fpgasupdate firmware.bin 05:00.0 fpgasupdate firmware.bin 0001:04:02.0 --log-level=ioctl
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/","title":"host_exerciser","text":""},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#synopsis","title":"SYNOPSIS","text":"Usage: host_exerciser [OPTIONS] SUBCOMMAND\nOptions:\n -h,--help Print this help message and exit\n -p,--pci-address TEXT [<domain>:]<bus>:<device>.<function>\n -l,--log-level TEXT:{trace,debug,info,warning,error,critical,off}=warning\n stdout logging level\n -s,--shared open in shared mode, default is off\n -t,--timeout UINT=60000 test timeout (msec)\n -m,--mode UINT:value in {lpbk->0,read->1,trput->3,write->2} OR {0,1,3,2}=lpbk\n host exerciser mode {lpbk,read, write, trput}\n --cls UINT:value in {cl_1->0,cl_2->1,cl_4->2,cl_8->3} OR {0,1,2,3}=cl_1\n number of CLs per request{cl_1, cl_2, cl_4, cl_8}\n --continuousmode BOOLEAN=false\n test rollover or test termination\n --atomic UINT:value in {cas_4->9,cas_8->11,fadd_4->1,fadd_8->3,off->0,swap_4->5,swap_8->7} OR {9,11,1,3,0,5,7}=off\n atomic requests (only permitted in combination with lpbk/cl_1)\n --encoding UINT:value in {default->0,dm->1,pu->2,random->3} OR {0,1,2,3}=default\n data mover or power user encoding -- random interleaves both in the same stream\n -d,--delay BOOLEAN=false Enables random delay insertion between requests\n --interleave UINT=0 Interleave requests pattern to use in throughput mode {0, 1, 2}\n indicating one of the following series of read/write requests:\n 0: rd-wr-rd-wr\n 1: rd-rd-wr-wr\n 2: rd-rd-rd-rd-wr-wr-wr-wr\n --interrupt UINT:INT in [0 - 3]\n The Interrupt Vector Number for the device\n --contmodetime UINT=1 Continuous mode time in seconds\n --testall BOOLEAN=false Run all tests\n --clock-mhz UINT=0 Clock frequency (MHz) -- when zero, read the frequency from the AFU\nSubcommands:\n lpbk run simple loopback test\n mem run simple mem test\n
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#description","title":"DESCRIPTION","text":"The host exerciser used to exercise and characterize the various host-FPGA interactions eg. MMIO, Data transfer from host to FPGA , PR, host to FPGA memory etc.
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#host-exerciser-loopback-he-lbk","title":"Host Exerciser Loopback (HE-LBK)","text":"HE-LB is responsible for generating traffic with the intention of exercising the path from the AFU to the Host at full bandwidth. Host Exerciser Loopback (HE-LBK) AFU can move data between host memory and FPGA.
HE-LBK supports: 1. Latency (AFU to Host memory read) 2. MMIO latency (Write+Read) 3. MMIO BW (64B MMIO writes) 4. BW (Read/Write, Read only, Wr only)
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#host-exerciser-memory-he-mem","title":"Host Exerciser Memory (HE-MEM)","text":"HE-MEM is used to exercise use of FPGA connected DDR; data read from the host is written to DDR, and the same data is read from DDR before sending it back to the host. HE-MEM uses external DDR memory (i.e. EMIF) to store data. It has a customized version of the AVMM interface to communicate with the EMIF memory controller.
Execution of these exercisors requires the user to bind specific VF endpoint to vfio-pci Bind the correct endpoint for a device with B/D/F 0000:b1:00.0
[user@localhost]: sudo opae.io init -d 0000:b1:00.2 user:user
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#host-exerciser-sub-commands","title":"HOST EXERCISER SUB COMMANDS","text":"lpbk
run host exerciser loopback test
mem
run host exerciser memory test
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"--help, -h
Prints help information and exit.
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#common-arguments-options","title":"COMMON ARGUMENTS / OPTIONS","text":"The following arguments are common to all commands and are optional.
-p,--pci-address
PCIe domain, bus, device, function number of fpga resource.
-l,--log-level
set host exerciser tool log level, trace, debug, info, warning, error, critical, off
-s,--shared
open FPGA PCIe resource in shared mode
-t,--timeout
host exerciser tool time out, by default time out 60000
-m,--mode
host exerciser test modes are lpbk, read, write, trput
--cls
Number of cachelines per request 1, 2, 3, 4.
--continuousmode
Configures test rollover or test termination mode.
--atomic
atomic requests.
--encoding
select data mover mode or power user mode or random.
-d,--delay
Enables random delay insertion between requests.
--interleave
Enables interleave requests in throughput mode. Value:3'b000-Rd,Wr,Rd,Wr Value:3'b001-Rd,Rd,Wr,Wr Value:3'b010-Rd,Rd,Rd,Rd,Wr,Wr,Wr,Wr Value:3'b011-Not supported
--interrupt
Accelerator interrupt vector Number.
--contmodetime
Continuous mode time in seconds.
--testall
Run all host exerciser tests.
--clock-mhz
pcie clock frequency, default value 350Mhz. When specified by the user, will not check value against AFU clock before calculating performance metrics.
"},{"location":"sw/fpga_tools/host_exerciser/host_exerciser/#examples","title":"EXAMPLES","text":"This command exerciser Loopback afu:
host_exerciser lpbk\n
This command exerciser memory afu:
host_exerciser mem\n
This command exerciser Loopback afu on pcie 000:3b:00.0:
host_exerciser --pci-address 000:3b:00.0 lpbk\n
This command exerciser Loopback afu on pcie 000:3b:00.0 and run in write mode:
host_exerciser --pci-address 000:3b:00.0 --mode write lpbl\n
This command exerciser Loopback afu on pcie 000:3b:00.0 and run 2 cache lines per request:
host_exerciser --pci-address 000:3b:00.0 --cls cl_2 lpbk\n
This command exerciser Loopback afu on pcie 000:3b:00.0 and run continuous mode for 10 seconds:
host_exerciser --pci-address 000:3b:00.0 -cls cl_1 -m 0 --continuousmode true --contmodetime 10 lpbk\n
"},{"location":"sw/fpga_tools/hssi/hssi/","title":"hssi","text":""},{"location":"sw/fpga_tools/hssi/hssi/#synopsis","title":"SYNOPSIS","text":"hssi COMMON_OPTIONS MODE MODE_OPTIONS
"},{"location":"sw/fpga_tools/hssi/hssi/#description","title":"DESCRIPTION","text":"The hssi application provides a means of interacting with the 10G, 100G, and 200G/400F HE-HSSI AFUs. In all operating modes, the application initializes the AFU and completes the desired transfer as described by the mode- specific options.
COMMON_OPTIONS - application options common to the 10G, 100g, and 200G/400G modes.
-h, --help
Display common command-line help and exit.\n
-p, --pci-address ADDR
The PCIe address of the desired accelerator in ssss:bb:dd.f format.\n
-s, --shared on|off
Whether to open the accelerator in shared mode. The default is off.\n
-t, --timeout VALUE
The application timeout value in milliseconds. The default timeout is 60000 msec.\n
MODE - select AFU. Valid values are hssi_10g, hssi_100g, hssi_200g_400g.
MODE_OPTIONS [hssi_10g] - application options specific to the 10G AFU.
-h, --help
Display 10G AFU specific command-line help and exit.\n
--port PORT
Select the QSFP port in the range 0-7. The default is port 0.\n
--num-packets PACKETS
The number of packets to transfer. The default is 1 packet.\n
--random-length fixed|random
Specify packet length randomization. Valid values are fixed and\nrandom. The default is fixed (no randomization).\n
--random-payload incremental|random
Specify payload randomization. Valid values are incremental and\nrandom. The default is incremental.\n
--packet-length LENGTH
Specify packet length. The default is 64 bytes.\n
--src-addr ADDR
Specify the source MAC address. The default value is 11:22:33:44:55:66.\n
--dest-addr ADDR
Specify the destination MAC address. The default value is 77:88:99:aa:bb:cc.\n
--rnd-seed0 SEED0
Specify the prbs generator bits [31:0]. The default is 1592590336.\n
--rnd-seed1 SEED1
Specify the prbs generator bits [47:32]. The default is 1592590337.\n
--rnd-seed2 SEED2
Specify the prbs generator bits [91:64]. The default is 155373.\n
MODE_OPTIONS [hssi_100g] - application options specific to the 100G AFU.
--port PORT
Select the QSFP port in the range 0-7. The default is port 0.\n
--eth-loopback on|off
Whether to enable loopback on the ethernet interface. Valid values are\non and off. The default is on.\n
--num-packets PACKETS
The number of packets to transfer. The default is 1 packet.\n
--gap random|none
Inter-packet gap. Valid values are random and none. The default is none.\n
--pattern random|fixed|increment
Pattern mode. Valid values are random, fixed, or increment. The default\nis random.\n
--src-addr ADDR
Specify the source MAC address. The default value is 11:22:33:44:55:66.\n
--dest-addr ADDR
Specify the destination MAC address. The default value is 77:88:99:aa:bb:cc.\n
--start-size SIZE
Specify the packet size in bytes, or the first packet size for --pattern increment.\n
--end-size SIZE
Specify the end packet size in bytes.\n
--end-select pkt_num|gen_idle
Specify packet generation end mode.\n
MODE_OPTIONS [pkt_filt_10g] - application options specific to the Packet Filter 10G AFU.
--dfl-dev DFL_DEV
Packet Filter DFL device, eg --dfl-dev dfl_dev.0\n
MODE_OPTIONS [pkt_filt_100g] - application options specific to the Packet Filter 100G AFU.
--dfl-dev DFL_DEV
Packet Filter DFL device, eg --dfl-dev dfl_dev.1\n
MODE_OPTIONS [hssi_200g_400g] - application options specific to the 200G/400G AFU.
--num-packets PACKETS
The number of packets to transfer. Must be a multiple of 32. Default value is 32. Increasing the timeout (--timeout) may be necessary if specifying a large number of packets.\n
"},{"location":"sw/fpga_tools/hssi/hssi/#examples","title":"EXAMPLES","text":"hssi -h hssi hssi_10g -h sudo hssi --pci-address=0000:3b:00.0 hssi_10g --eth-loopback=on --num-packets=500 sudo hssi --pci-address=0000:3b:00.0 hssi_100g --pattern=increment sudo hssi --pci-address=0000:0d:00.6 hssi_200g_400g --num-packets=640000
"},{"location":"sw/fpga_tools/hssi_config/readme/","title":"hssi_config","text":""},{"location":"sw/fpga_tools/hssi_config/readme/#synopsis","title":"Synopsis","text":"hssi_config reads or writes HSSI registers on either on an Intel\u00ae FPGA using the FPGA Interface Manager (FIM) or on an HSSI retimer card attached to the board. hssi_config is only available for the Integrated FPGA Platform. You cannot run it on the PCIe accelerator card (PAC).
"},{"location":"sw/fpga_tools/hssi_config/readme/#usage","title":"Usage","text":"hssi_config [--resource|-r <sysfs resource>] [--socket-id|s 0|1] command [command options]
Where command is one of the following:
dump [outfile.csv] [--input-file inputfile.csv]\niread instance (0,1) device-addr byte-address byte-count\n iwrite instance (0,1) device-addr byte-address byte1 [byte2 [byte3...]]\nload [inputfile.csv] [--c-header]\nread lane(0-15) reg-address\n rread device(0x30, 0x32, 0x34, 0x36) channel(0-3) address\n rwrite device(0x30, 0x32, 0x34, 0x36) channel(0-3) address value\n test (rd|rw) inputfile.csv [--acktimes] [--repeat N]\nwrite lane(0-15) reg-address value\n
The first argument is the command and any additional arguments are command arguments. The following options and commands are available:
"},{"location":"sw/fpga_tools/hssi_config/readme/#options","title":"Options","text":"[--resource|-r <sysfs resource path>
The resource path in the sysfs pseudo-filesystem. Example: /sys/devices/pci0000\\:5e/0000\\:5e\\:00.0/resource0
[--socket-id 0|1]
The socket id of the target FPGA. Required on two-socket systems to differentiate between the two possible target FPGAs.
"},{"location":"sw/fpga_tools/hssi_config/readme/#commands","title":"Commands","text":"dump [outfile.csv] [--input-file inputfile.csv]
Dump registers to stdout or to a file, if provided. hssi_config has a built-in set of registers to dump. The first argument is the path to a file to write. The command dumps to stdout if you do not specify a file name. Use the --input-file option to specify a different set of registers.
load [inputfile.csv] [--c-header]
Load a set of register values from either stdin or an input file, if provided. The first argument is the path to a file containing the registers to load. Loads from stdin if omitted.
Use --c-header to generate a C header file with an array of 64-bit numbers to write to the HSSI_CTRL register. This header file can substitute for the input file. NOTE: You must perform the acknowledge routine after each write.
read lane(0-15) reg-address
Read from a single XCVR (transceiver) register. The first command argument is the XCVR lane. Use -1 to specify a read from all lanes. The second argument is the XCVR address (offset).
write lane(0-15) reg-address value
Write to a single XCVR register. The first argument is the XCVR lane. The second argument is the XCVR address(offset). The third argument is the value to write to the register.
rread device(0x30, 0x32, 0x34, 0x36) channel(0-3) address
Read from a single retimer register. The first argument is the I2C device address. The second argument is the channel. The third argument is the register address (or I2C byte address).
rwrite device(0x30, 0x32, 0x34, 0x36) channel(0-3) address value
Write to a single retimer register. The first argument is the I2C device address. The second argument is the channel. The third argument is the register address (or I2C byte address). The fourth argument is the value to write.
iread instance (0,1) device-addr byte-address byte-count
Read from a device on the I2C bus. The first argument is the I2C controller instance (0 or 1). The second argument is the device address to read from. The third argument is the byte address of the register to read from the device. The fourth argument is the number of bytes to read.
iwrite instance (0,1) device-addr byte-address byte1 [byte2 [byte3...]]
Write to a device on the I2C bus. The first argument is the I2C controller instance (0 or 1). The second argument is the device address to read from. The third argument is the byte address of the register to read from the device. All subsequent arguments are the bytes to write to the device.
test (rd|rw) inputfile.csv [--acktimes]
Perform built-in test for reading or writing XCVR registers. The first argument is the path to a file containing the registers to test.
"},{"location":"sw/fpga_tools/hssi_config/readme/#overview","title":"Overview","text":"The hssi_config utility reads or writes hssi equalization parameters stored in either the transceiver (XCVR) registers or the registers of the retimer on the I2C bus. To access registers, the hssi controller writes to the HSSI_CTRL register and reads from the HSSI_STAT register in the FPGA Management Engine (FME). These two registers implement the HSSI AUX bus mailbox protocol to access devices on other buses. Because hssi_config maps the FME MMIO space directly, the FPGA driver is not required.
"},{"location":"sw/fpga_tools/hssi_config/readme/#locating-the-fme-device","title":"Locating the FME Device","text":"The FME reads and writes the HSSI_CTRL and HSSI_STAT registers on the NIOS device. The FME maps the MMIO address space of the FME identified by its resource in the sysfs psuedo-filesystem in Linux operating systems. To identify resource paths, use the lspci utility to query for Intel devices with device id of bcc0.
Example:
lspci -d 8086:bcc0
This command should print out at least one line like the following example:
5e:00.0 Processing accelerators: Intel Corporation Device bcc0
Use the first three numbers (bus:device.function) to locate the device resource in the sysfs filesystem:
/sys/devices/pci0000\\:<bus>/0000\\:<bus>\\:<device>.<function>/resource0
For example, the example above with bus of 5e, device of 00 and function of 0 would use a resource path as follows:
/sys/devices/pci0000\\:5e/0000\\:5e\\:00.0/resource0
"},{"location":"sw/fpga_tools/hssi_config/readme/#csv-file-format","title":"CSV File Format","text":"Any CSV file parsed by hssi_config must meet have at least four columns. The following table provides the column specifications:
Column Name Description 1 Register type. Can be either FPGA_RX, FPGA_TX, RTMR_RX, RTMR_TX (or their corresponding numeric values, 1-4). 2 Lane or Channel 0-15 for XCVR lanes on FPGA, 0-3 for retimer channels. -1 to designate all lanes or channels. 3 Device address (on I2C bus) Only applies to retimer registers. 0 - 3, -1 to designate all devices. 4 Register address (or offset) Examples: 0x213, 0x100. 5 Register value to write Examples: 0x1, 1, 0. Applies only when loading or writing registers."},{"location":"sw/fpga_tools/hssi_config/readme/#examples-of-commands","title":"Examples of Commands","text":""},{"location":"sw/fpga_tools/hssi_config/readme/#dumping-registers","title":"Dumping Registers","text":"Dump default register set to stdout:
>hssi_config dump
Dump default registers to a file, data.csv:
>hssi_config dump data.csv
Dump to an output file, data.csv, Registers specified in an input file, regspec.csv:
>hssi_config dump data.csv --input-file regspec.csv
"},{"location":"sw/fpga_tools/hssi_config/readme/#reading-single-registers","title":"Reading Single Registers","text":"Read register from XCVR at 0x2e1 on lane 0:
>hssi_config read 0 0x2e1
Read register 0x109 from retimer on channel 0, device 0x30, channel 1:
>hssi_config rread 0 0x30 0x109
"},{"location":"sw/fpga_tools/hssi_config/readme/#loading-registers","title":"Loading Registers","text":"Load registers specified in an input file called data.csv:
>hssi_config load data.csv
Load registers specified from stdin:
>hssi_config load
FPGA_RX,1,-1,0x213,0\nFPGA_RX,2,-1,0x213,0\nFPGA_RX,3,-1,0x213,0\n<CTRL-D>\n
"},{"location":"sw/fpga_tools/hssi_config/readme/#writing-single-registers","title":"Writing Single Registers","text":"Write 1 to XCVR register at 0x2e1 on lane 0:
>hssi_config write 0 0x2e1 1
Read register 0x109 from retimer on channel 0, device 0x30, channel 1:
>hssi_config rread 0 0x30 0x109
"},{"location":"sw/fpga_tools/hssi_config/readme/#testing-hssi-read-and-write","title":"Testing HSSI Read and Write","text":"> test (rd|rw) register-file.csv [--acktimes]
rd|wr
Specifies either a rd or wr of transceiver registers. For writes, every register in the file is read from and written to in the following sequence:
1. Read the register, save the value 2. Write 1 to the register 3. Read the register, verify that the register value is 1 4. Write 0 to the register 5. Read the register, verify that the register value is 0 6. Write the original value to the register 7. Read the register, assert it is the original value
register=file.csv
Specifies the path to a file containing the set of registers to test.
--acktimes
Specifies the time spent in the ack routine. When measured, a summary of ack times prints to stdout. This argument is optional.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/","title":"HSSI ethernet loopback","text":""},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#synopsis","title":"SYNOPSIS","text":"hssiloopback [-h] [--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS] --loopback [{enable,disable}]\n
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#description","title":"DESCRIPTION","text":"The hssiloopback tool enables and disable ethernet loopback.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Prints usage information
--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS
The PCIe address of the desired fpga in ssss:bb:dd.f format. sbdf of device to program (e.g. 04:00.0 or 0000:04:00.0). Optional when one device in system.
--loopback [{enable,disable}]
Ethernet enable or disable loopback.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssiloopback/#examples","title":"EXAMPLES","text":"hssiloopback --pcie-address 0000:04:00.0 --loopback enable
Enables ethernet loopback
hssiloopback --pcie-address 0000:04:00.0 --loopback disable
Disable ethernet loopback
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/","title":"HSSI ethernet mac","text":""},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#synopsis","title":"SYNOPSIS","text":"hssimac [-h] --pcie-address PCIE_ADDRESS [--port PORT]\n
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#description","title":"DESCRIPTION","text":"The hssimac tool provides Maximum TX and RX frame size.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Prints usage information
--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS
The PCIe address of the desired fpga in ssss:bb:dd.f format. sbdf of device to program (e.g. 04:00.0 or 0000:04:00.0).
--port PORT
hssi port number.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssimac/#examples","title":"EXAMPLES","text":"hssimac --pcie-address 0000:04:00.0 --port 1
prints Maximum TX and RX frame size for port 1.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/","title":"HSSI ethernet statistics","text":""},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#synopsis","title":"SYNOPSIS","text":"hssistats [-h] [--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS]\n
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#description","title":"DESCRIPTION","text":"The hssistats tool provides the MAC statistics.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help
Prints usage information
--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS
The PCIe address of the desired fpga in ssss:bb:dd.f format. sbdf of device to program (e.g. 04:00.0 or 0000:04:00.0). Optional when one device in system.
"},{"location":"sw/fpga_tools/hssi_ethernet/hssistats/#examples","title":"EXAMPLES","text":"hssistats --pcie-address 0000:04:00.0
prints the MAC statistics
"},{"location":"sw/fpga_tools/hssi_loopback/readme/","title":"hssi_loopback","text":""},{"location":"sw/fpga_tools/hssi_loopback/readme/#name","title":"NAME","text":"hssi_loopback - Software utility to run HSSI loopback tests on FPGA
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#synopsis","title":"SYNOPSIS","text":"hssi_loopback [[--bus|-b <bus number>] [--device | -d <device number>] [--function | -f <function number>]]|[--socket-id <socket-id>] [--mode|-m auto|e40|e10] [send [<source port> [<destination port>] [--packet-count|-c <count>] [--packet-delay|-d <delay>] [--packet-length|-l <length>]] |status [clear] | stop | readmacs
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#description","title":"DESCRIPTION","text":"The hssi_loopback utility works in conjunction with a packet generator accelerator function unit (AFU) to test high-speed serial interface (HSSI) cards. The hssi_loopback utility tests both external and internal loopbacks. hssi_loopback runs an external loopback test when the command line arguments include both source and destination ports. hssi_loopback runs an internal loopback test when command line arguments include a single port. hssi_loopback only runs on the Intel Xeon with Arria 10 FPGA. You cannot run it on the Intel PAC (programmable accelerator card).
NOTE: The following limitations apply to the current version of hssi_loopback:
- For the external loopback the two port arguments can be the same. For the e10 design, the ports should be the same.
- The
hssi_loopback test supports only the e40 and e10 E2E AFUs. The e10 E2E AFU tests HSSI with a retimer card. - The
hssi_loopback test uses the control and status registers (CSRs) defined in the AFU.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#options","title":"OPTIONS","text":"-S SOCKET_ID, --socket-id SOCKET_ID
Socket ID FPGA resource.
-B BUS, --bus BUS
Bus ID of FPGA resource.
-D DEVICE, --device DEVICE
Device ID of FPGA resource.
-F FUNCTION, --function FUNCTION
Function ID of FPGA resource.
-G, --guid
Specifies guid for the resource enumeration.
-m, --mode
One of the following: [auto, e40, e10] auto is the default and indicates that the software runs the mode based on the first accelerator functional unit it identifies.
-t, --timeout
Timeout (in seconds) before the application terminates in continuous mode. Continuous mode is the default when you do not specify the number of packets.
-y, --delay
Delay (in seconds) between printing out a simple status line. Default is 0.100 seconds (100 milliseconds).
-c, --packet-count
The number of packets to send.
-d, --packet-delay
The delay in between packets. This delay is the number of 100 MHz clock cycles, roughly 10 nanoseconds.
-s, --packet-size
The packet size to send. The minimum is 46 bytes and the maximum is 1500 bytes. The default is 46 bytes.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#commands","title":"COMMANDS","text":"send <source port> [<destination port>] [--packet-count|-c <count>] [--packet-delay|-d <delay>] [--packet-length|-l <length>]
Send packets from one port to the other. If the command line does not specify a destination port, the test runs an internal loopback. Otherwise, the test runs an external loopback from the source port to the destination port.
status [clear]
Read and interpret the status registers and print to the screen. clear clears the status registers.
stop
Issue a stop command to all Ethernet controllers in the AFU.
readmacs
Read and display the port MAC addresses. An EEPROM stores the MAC addresses.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#exit-codes","title":"EXIT CODES","text":"0 Success - Number of packets received are equal to the number of packets sent and no errors are reported.
-1 Loopback failure - Either number of packets does not match or the test detected errors.
-2 Errors parsing arguments.
"},{"location":"sw/fpga_tools/hssi_loopback/readme/#examples","title":"EXAMPLES","text":"Read the MAC addresses of the AFU loaded on bus 0x5e:
>sudo hssi_loopback readmacs -B 0x5e\n
Run an external loopback, sending 100 packets from port 0 to port 1. The AFU is on bus 0x5e:
>sudo hssi_loopback -B 0x5e send 0 1 -c 100\n
Run an internal loopback until a timeout of 5 seconds is reached. The AFU is on bus 0x5e:
>sudo hssi_loopback -B 0x5e send 0 -t 5\n
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/","title":"mem_tg","text":""},{"location":"sw/fpga_tools/mem_tg/mem_tg/#synopsis","title":"SYNOPSIS","text":"Usage: mem_tg [OPTIONS] SUBCOMMAND\nOptions:\n -h,--help Print this help message and exit\n -g,--guid TEXT=4DADEA34-2C78-48CB-A3DC-5B831F5CECBB\n GUID\n -p,--pci-address TEXT [<domain>:]<bus>:<device>.<function>\n -l,--log-level TEXT:{trace,debug,info,warning,error,critical,off}=info\n stdout logging level\n -s,--shared open in shared mode, default is off\n -t,--timeout UINT=60000 test timeout (msec)\n -m,--mem-channel UINT=0 Target memory bank for test to run on (0 indexed)\n --loops UINT=1 Number of read/write loops to be run\n -w,--writes UINT=1 Number of unique write transactions per loop\n -r,--reads UINT=1 Number of unique read transactions per loop\n -b,--bls UINT=1 Burst length of each request\n --stride UINT=1 Address stride for each sequential transaction\n --data UINT:value in {fixed->0,prbs15->2,prbs31->3,prbs7->1,rot1->3} OR {0,2,3,1,3}=fixed\n Memory traffic data pattern: fixed, prbs7, prbs15, prbs31, rot1\n -f,--mem-frequency UINT=0 Memory traffic clock frequency in MHz\nSubcommands:\n tg_test configure & run mem traffic generator test\n
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#description","title":"DESCRIPTION","text":"The memory traffic generator (TG) used to exercise and test available memory channels with a configurable traffic pattern.
Execution of this application requires the user to bind the specific VF endpoint containing the mem_tg AFU id to vfio-pci
In the TG, read responses are checked against a specified pattern. If the application is configured to perform a read only test on a region of memory that has not previously been initialized to contain that pattern it will flag a test failure.
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"--help, -h
Prints help information and exit.
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#common-arguments-options","title":"COMMON ARGUMENTS / OPTIONS","text":"The following arguments are common to all commands and are optional.
-p,--pci-address
PCIe domain, bus, device, function number of fpga resource.
-l,--log-level
set application log level, trace, debug, info, warning, error, critical, off
-s,--shared
open FPGA PCIe resource in shared mode
-t,--timeout
mem_tg application time out, by default time out 60000
-m,--mem-channel
Target memory bank for test to run on (0 indexed) default: 0
--loops
Number of read/write loops to be run default: 1
-w,--writes
Number of unique write transactions per loop. default: 1
-r,--reads
Number of unique read transactions per loop default: 1
-b,--bls
AXI4 burst length of each request. Supports 1-256 transfers beginning from 0. default: 0
--stride
Address stride for each sequential transaction (>= burst length) default: 1
--data
Memory traffic data pattern. 0 = fixed {0xFF, 0x00} 1 = prbs7 2 = prbs15 3 = prbs31 4 = rot1
default: fixed
-f, --mem-frequency
Memory traffic clock frequency in MHz default: 300 MHz
"},{"location":"sw/fpga_tools/mem_tg/mem_tg/#examples","title":"EXAMPLES","text":"This command will run a basic read/write test on the channel 0 traffic generator:
mem_tg tg_test\n
This command will run the application for an afu on pcie 000:b1:00.7:
mem_tg --pci-address 000:b1:00.7 tg_test\n
This command will test channel 2 write bandwidth:
mem_tg -loops 1000 -w 1000 -r 0 -m 2 tg_test\n
This command will perform a read bandwidth test with a burst of 16 on channel 1 and perform a data comparison with the prbs7 pattern:
mem_tg -loops 1000 -w 0 -r 1000 -b 0xF --data prbs7 -m 1 tg_test\n
This command will perform a read/write test with 1 MB strided access to channel 0 memory:
mem_tg -loops 10000 --stride 0x100000 tg_test\n
"},{"location":"sw/fpga_tools/mmlink/mmlink/","title":"mmlink","text":""},{"location":"sw/fpga_tools/mmlink/mmlink/#synopsis","title":"Synopsis","text":"mmlink [-v] [-B <bus>] [-D <device>] [-F <function>] [-S <socket>] [-P <TCP port>] [-I <IP Address>]
"},{"location":"sw/fpga_tools/mmlink/mmlink/#description","title":"Description","text":"The Remote Signal Tap logic analyzer provides real-time hardware debugging for the Accelerator Function Unit (AFU). It provides a signal trace capability that the Quartus Prime software adds to the AFU. The Remote Signal Tap logic analyzer provides access to the Remote Signal Tap part of the Port MMIO space and then runs the remote protocol.
"},{"location":"sw/fpga_tools/mmlink/mmlink/#examples","title":"Examples","text":"./mmlink -B 0x5e -P 3333
MMLink app starts and listens for connection.
"},{"location":"sw/fpga_tools/mmlink/mmlink/#options","title":"Options","text":"-v,--version
Prints version information and exits.
-B,--bus
FPGA Bus number.
-D,--device
FPGA Device number.
-F,--function
FPGA function number.
-S,--socket
FPGA socket number.
-P,--port
TCP port number.
-I,--ip
IP address of FPGA system.
"},{"location":"sw/fpga_tools/mmlink/mmlink/#notes","title":"Notes","text":"Driver privilege:
Change AFU driver privilege to user:
$ chmod 777 /dev/intel-fpga-port.0\n
Change locked memory size:
edit the file /etc/security/limits.conf
$ sudo vi /etc/security/limits.conf\n\nuser hard memlock 10000\n\nuser soft memlock 10000\n
Exit terminal and log into a new terminal.
Verify that the locked memory is now set: ``` $ ulimit -l 10000
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/","title":"ofs.uio","text":""},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#synopsis","title":"SYNOPSIS","text":"ofs.uio [-h] [--pcie-address PCIE_ADDRESS] [--uio uiox] [--feature-id FEATURE_ID] [--region-index REGION_INDEX] [--mailbox-cmdcsr offset] [--bit-size {8,16,32,64}] [--peek offset] [--poke offset value] [--mailbox-read offset] [--mailbox-dump address size] [--mailbox-write address value]
ofs.uio [--uio uiox] [--peek offset] ofs.uio [--uio uiox] [--poke offset value] ofs.uio [--uio uiox] [--mailbox-read address] ofs.uio [--uio uiox] [--mailbox-write address value] ofs.uio [--uio uiox] [--mailbox-dump address size] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--peek offset] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--poke offset value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-read address] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-write address value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-dump address size]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#description","title":"DESCRIPTION","text":"ofs.uio is a tool that provides user space access to DFL UIO devices, command line options like peek, poke, mailbox-read, mailbox-write, mailbox-dump to access Configuration and Status Registers (CSRs).
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#peek","title":"Peek","text":"Peek/Read UIO CSR offset ofs.uio [--uio uio] [--peek offset] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#poke","title":"Poke","text":"Poke/Write value to UIO CSR offset ofs.uio [--uio uio] [--poke offset value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--poke offset value]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-read","title":"Mailbox Read","text":"Read CSR address using mailbox ofs.uio [--uio uio] [--mailbox-read address] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-read address]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-write","title":"Mailbox Write","text":"Write value to CSR address using mailbox ofs.uio [--uio uio] [--mailbox-write address value] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-write address value]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-dump","title":"Mailbox Dump","text":"Reads/Dumps block size of CSR address using mailbox ofs.uio [--uio uio] [--mailbox-dump address size] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--mailbox-dump address size]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#bit-size","title":"Bit size","text":"Read/Write bit-field 8,16,32,64 sizes ofs.uio [--uio uio] --bit-size 8 [--peek offset] ofs.uio [--uio uio] --bit-size 32 [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#pcie-address","title":"PCIe Address","text":"PCIE_ADDR PCIe address of FPGA device ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#uio-region-index","title":"UIO region index","text":"UIO region index, default region index is 0 ofs.uio [--uio uio] --region-index 0 [--peek offset] ofs.uio [--uio uio] --region-index 1 [--peek offset]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#mailbox-command-status-csr-offset","title":"Mailbox command status csr offset","text":"Mailbox command status csr offset, default value set to dfl pcie subsystem system feature mailbox command status register offset 0x28 ofs.uio [--uio uio] --mailbox-cmdcsr 0xa8 [--mailbox-read address] ofs.uio [--pcie-address PCIE_ADDRESS] [--feature-id FEATURE_ID] --mailbox-cmdcsr 0xa8 [--mailbox-read address]
"},{"location":"sw/fpga_tools/ofs.uio/ofs.uio/#examples","title":"EXAMPLES","text":"Peek/Read
ofs.uio --uio uio0 --peek 0x0\npeek(0x0): 0x3000000010002015\n\nofs.uio --uio uio6 --peek 0x0\npeek(0x0): 0x3000000100000020\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x15 --peek 0x0\npeek(0x0): 0x3000000010002015\n\nofs.uio --uio uio0 --peek 0x0 --bit-size 32\npeek(0x0): 0x10002015\n
Poke/Write
ofs.uio --uio uio6 --peek 0x8\npeek(0x8): 0x0\nofs.uio --uio uio6 --poke 0x8 0xabcdd12345\npoke(0x8):0xabcdd12345\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x15 --peek 0x0\npeek(0x8): 0x0\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x15 --poke 0x8 0x1234\npoke(0x8):0x1234\n
Mailbox Read
ofs.uio --uio uio6 --mailbox-read 0x0\nMailboxRead(0x0): 0x1000000\nofs.uio --uio uio6 --mailbox-read 0x8\nMailboxRead(0x8): 0x110c000\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-read 0x0\nMailboxRead(0x0): 0x1000000\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-read 0x8 \nMailboxRead(0x8): 0x110c000\n
Mailbox Write
ofs.uio --uio uio6 --mailbox-write 0x0 0x1234\nMailboxWrite(0x0):0x1234\nofs.uio --uio uio6 --mailbox-read 0x0\nMailboxRead(0x0):0x1234\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-write 0x0 0x1234\nMailboxWrite(0x0):0x1234\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-read 0x0 \nMailboxRead(0x0):0x1234\n
Mailbox Dump
ofs.uio --uio uio6 --mailbox-dump 0x0 0x10\nMailboxDump(0x0): 0x1000000\nMailboxDump(0x4): 0x1000000\nMailboxDump(0x8): 0x110c000\nMailboxDump(0xc): 0x110c000\nMailboxDump(0x10): 0x0\nMailboxDump(0x14): 0x0\nMailboxDump(0x18): 0x0\nMailboxDump(0x1c): 0x0\nMailboxDump(0x20): 0x0\nMailboxDump(0x24): 0x0\nMailboxDump(0x28): 0x0\nMailboxDump(0x2c): 0x0\nMailboxDump(0x30): 0x0\nMailboxDump(0x34): 0x0\nMailboxDump(0x38): 0x0\nMailboxDump(0x3c): 0x0\n\nofs.uio --pcie-address 0000:b1:00.0 --feature-id 0x20 --mailbox-dump 0x0 0x10\nMailboxDump(0x0): 0x1000000\nMailboxDump(0x4): 0x1000000\nMailboxDump(0x8): 0x110c000\nMailboxDump(0xc): 0x110c000\nMailboxDump(0x10): 0x0\nMailboxDump(0x14): 0x0\nMailboxDump(0x18): 0x0\nMailboxDump(0x1c): 0x0\nMailboxDump(0x20): 0x0\nMailboxDump(0x24): 0x0\nMailboxDump(0x28): 0x0\nMailboxDump(0x2c): 0x0\nMailboxDump(0x30): 0x0\nMailboxDump(0x34): 0x0\nMailboxDump(0x38): 0x0\nMailboxDump(0x3c): 0x0\n
"},{"location":"sw/fpga_tools/opae-mem/opae-mem/","title":"opae-mem","text":""},{"location":"sw/fpga_tools/opae-mem/opae-mem/#synopsis","title":"SYNOPSIS","text":"opae-mem ls [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-t,--token-type {device,accel}] [-i,--interface {dfl,vfio,dfl_ase,vfio_ase,uio}] opae-mem peek [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-c,--count COUNT] [-o,--output OUTPUT] [-F,--format {simple,hex,json}] offset opae-mem poke [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-j,--json JSON_FILE] [offset] [value] opae-mem mb-read [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-b,--mailbox-base BASE] [-c,--count COUNT] [-t,--timeout TIMEOUT] [-s,--sleep SLEEP] [-o,--output OUTPUT] [-F,--format {simple,hex,json}] address opae-mem mb-write [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] [-b,--mailbox-base BASE] [-t,--timeout TIMEOUT] [-s,--sleep SLEEP] [-j,--json JSON_FILE] [address] [value] opae-mem lock [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID] opae-mem unlock [-d,--device <PCIE_ADDR>|<PCIE_ID>] [-f,--feature-id ID]
"},{"location":"sw/fpga_tools/opae-mem/opae-mem/#description","title":"DESCRIPTION","text":"opae-mem is a utility that provides access to FPGA accelerator MMIO. It is written on top of the OPAE Python bindings; so it works with any FPGA accelerator, whether the accelerator is connected via DFL, VFIO, UIO, or ASE.
Accelerators are enumerated using the opae-mem ls command:
$ sudo chmod 666 /dev/uio* $ opae-mem ls [0000:d8:00.0] (8086:bcce 8086:0000) UIO 0x14 00000000-0000-0000-0000-000000000000 UIO 0x20 00000000-0000-0001-0000-000000000000 [0000:3b:00.0] (8086:bcce 8086:0000) UIO 0x15 00042415-0004-2415-0000-001100010000 UIO 0x20 00000000-0000-0001-0000-000000000000 UIO 0x14 00000000-0000-0000-0000-000000000000
This output shows two FPGA cards with a total of five FPGA MMIO regions.
The first card at address 0000:d8:00.0 has two MMIO regions accessible via UIO:
Feature ID 0x14: s10 IOPLL Feature ID 0x20: PCIe Subsystem
The second card at address 0000:3b:00.0 has three MMIO regions accessible via UIO:
Feature ID 0x15: HSSI Subsystem Feature ID 0x20: PCIe Subsystem Feature ID 0x14: s10 IOPLL
The peek command provides a way to view a range of MMIO addresses:
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 0x0 00000000: 3000000010000020 0000000000000000 | ......0........| 00000010: 0000000000000001 0000000000000000 |................|
Here, --count 4 causes the command to display four 64-bit qwords, from addresses 0x0 through 0x18.
The default format is hex display, which is modeled after the output of hexdump -C. The format can be controlled using the -F option. Valid values for -F are simple, hex, and json.
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 -F simple 0x0 00000000: 3000000010000020 00000008: 0000000000000000 00000010: 0000000000000001 00000018: 0000000000000000
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 -F json 0x0 { \"0x00000000\": \"0x3000000010000020\", \"0x00000008\": \"0x0000000000000000\", \"0x00000010\": \"0x0000000000000001\", \"0x00000018\": \"0x0000000000000000\" }
The output of opae-mem peek can be sent to a file using the -o option. This is useful for capture/playback. Playback is available with the opae-mem poke command.
$ opae-mem peek -d 0000:3b:00.0 -f 0x20 --count 4 -F json -o file.json 0x0 $ cat file.json { \"0x00000000\": \"0x3000000010000020\", \"0x00000008\": \"0x0000000000000000\", \"0x00000010\": \"0x0000000000000001\", \"0x00000018\": \"0x0000000000000000\" }
The poke command provides a way to write MMIO addresses:
$ opae-mem poke -d 0000:3b:00.0 -f 0x20 0x0 0xc0cac01a
In the above poke command, 0x0 is the MMIO offset to write and 0xc0cac01a is the value.
The output of the opae-mem peek command with -F json can be played back as a series of writes to an MMIO region using the opae-mem poke --json option:
$ opae-mem poke -d 0000:3b:00.0 -f 0x20 --json file.json
The opae-mem mb-read command is used to issue read requests to an FPGA mailbox interface:
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 0x0 mb-read [0000:3b:00.0] 0x20 This command needs -b mailbox_base. For feature_id 0x20, try -b 0x0028
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 0x0 00000000: 01000000 00000001 01104000 00000000 |.........@......|
Each mailbox address represents a 32-bit data value.
Like peek, the default display format for mb-read is hex. To change the display format, use the -F option, which accepts simple, hex, or json.
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 -F simple 0x0 00000000: 01000000 00000004: 00000001 00000008: 01104000 0000000c: 00000000
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 -F json 0x0 { \"0x00000028\": { \"0x00000000\": \"0x01000000\", \"0x00000004\": \"0x00000001\", \"0x00000008\": \"0x01104000\", \"0x0000000c\": \"0x00000000\" } }
The mb-write command provides a way to issue write commands to an FPGA mailbox interface.
$ opae-mem mb-write -d 0000:3b:00.0 -f 0x20 -b 0x28 0x0 0xc0cac01a
In the above command, 0x0 is the mailbox address and 0xc0cac01a is the 32-bit data value to be written.
The output of the opae-mem mb-read command with -F json can be played back as a series of writes to a mailbox interface using the opae-mem mb-write --json option:
$ opae-mem mb-read -d 0000:3b:00.0 -f 0x20 -c 4 -b 0x28 -F json -o mb.json 0x0 $ opae-mem mb-write -d 0000:3b:00.0 -f 0x20 -b 0x28 --json mb.json
Each of the above commands has explicitly specified the -d PCIE_ADDR and -f FEATURE_ID parameters, making for some long command lines. To shorten the length, opae-mem can be \"locked\" to a (device, feature_id) pair:
$ opae-mem lock -d 0000:3b:00.0 -f 0x20 lock [0000:3b:00.0] 0x20 OK
Once \"locked\" to a device, issuing the command again displays the lock status:
$ opae-mem lock [locked 0000:3b:00.0 0x20] lock currently held by 0000:3b:00.0 0x20.
From now until the time the session is unlocked, opae-mem commands may omit the explicit -d and -f parameters:
$ opae-mem peek 0x0 00000000: 3000000010000020 | ......0 |
\"Locking\" is simply a convenient way to shorten the opae-mem command line. Each of the other commands operates in the same way, as if -d and -f were specified explicitly.
Note: a \"lock\" can be overridden by specifying -d and/or -f:
$ opae-mem -V peek -d 0000:d8:00.0 -f 0x14 -c 4 0x0 [locked 0000:3b:00.0 0x20 [override 0000:d8:00.0 0x14]] peek [0000:d8:00.0] 0x14 offset=0x0 region=0 format=hex 00000000: 3000000010000014 0000000000000000 |.......0........| 00000010: 0000000000000000 1000000000000000 |................|
The preceding command used a lock override by specifying a different device address to -d and a different feature_id to -f. The -V (verbose) option was given to show the lock override status.
The unlock command is used to release a lock:
$ opae-mem unlock [locked 0000:3b:00.0 0x20] unlock Please tell me the device / feature id to unlock. (-d 0000:3b:00.0 -f 0x20)
$ opae-mem unlock -d 0000:3b:00.0 -f 0x20 [locked 0000:3b:00.0 0x20] unlock [0000:3b:00.0] 0x20 OK
$ opae-mem lock lock Give me the device address and feature id.
"},{"location":"sw/fpga_tools/opae.io/opae.io/","title":"opae.io","text":""},{"location":"sw/fpga_tools/opae.io/opae.io/#synopsis","title":"SYNOPSIS","text":"opae.io ls [-v,--viddid VIDDID] [-s,--sub-viddid SUB_VIDDID] [--all] [--system-class] opae.io init [-d,--device PCI_ADDR] [USER[:GROUP]] opae.io release [-d,--device PCI_ADDR] opae.io [-d,--device PCI_ADDR] [-r,--region REGION] walk [--offset [OFFSET]] [-u,--show-uuid] [-D,--dump] [-c,--count COUNT] [-y,--delay DELAY] [-s,--safe] opae.io [-d,--device PCI_ADDR] [-r,--region REGION] dump [--offset [OFFSET]] [-o,--output OUTPUT] [-f,--format {bin,hex}] [-c,--count COUNT] opae.io [-d,--device PCI_ADDR] [-r,--region REGION] peek OFFSET opae.io [-d,--device PCI_ADDR] [-r,--region REGION] poke OFFSET VALUE opae.io [-d,--device PCI_ADDR] [-r,--region REGION] script SCRIPT ARG1 ARG2 ... ARGN opae.io [-d,--device PCI_ADDR] [-r,--region REGION]
"},{"location":"sw/fpga_tools/opae.io/opae.io/#description","title":"DESCRIPTION","text":"opae.io is an interactive Python environment packaged on top of libopaevfio.so, which provides user space access to PCIe devices via the vfio-pci driver. The main feature of opae.io is its built-in Python command interpreter, along with some Python bindings that provide a means to access Configuration and Status Registers (CSRs) that reside on the PCIe device.
Note: opae.io peek/poke will call fpgaEnumerate() internally and so a reset will be asserted on the AFU region. You can avoid initializing these registers by passing opae.io a python script directly.
opae.io has two operating modes: command line mode and interactive mode.
"},{"location":"sw/fpga_tools/opae.io/opae.io/#command-line-mode","title":"COMMAND LINE MODE","text":"To view the accelerator devices that are present on the system, opae.io provides the ls command option.
opae.io ls [-v,--viddid VIDDID] [-s,--sub-viddid SUB_VIDDID] [--all] [--system-class]
Each accelerator device is listed along with the PCIe address, the PCIe vendor/device ID, a brief description of the device, and the driver to which the device is currently bound.
Device filtering is available by providing a Vendor ID:Device ID pair, eg -v 8086:bcce. Further filtering can be done by providing a sub- Vendor ID:sub-Device ID pair, eg -s 8086:1771. The --all option provides a list of all of the PCIe devices in the system, which an be quite verbose. The --system-class option prints the PCIe database class of the accelerator device, rather than the product name.
opae.io provides an option to initialize a PCIe device for use with the vfio-pci driver. In order for the device CSRs to be accessed from user space, the device must first be bound to the vfio-pci driver. This is the job of the init command option.
opae.io init [-d,--device PCI_ADDR] [USER[:GROUP]]
The init command unbinds the specified device from its current driver and binds it to vfio-pci. This creates a new vfio group under /dev/vfio. This group path is then used by the libopaevfio.so library to interact with the device.
To release the PCIe device from vfio-pci and return it to use with its previous driver, the release command option is used.
opae.io release [-d,--device PCI_ADDR]
The release command option reverses the actions of the last init command, releasing the device from vfio-pci and binding it to the driver which was bound at the time the init command was issued.
The walk command option traverses and displays the Device Feature List of the given region.
opae.io walk [--offset [OFFSET]] [-u,--show-uuid] [-D,--dump] [-c,--count COUNT] [-y,--delay DELAY] [-s,--safe]
The various fields of each Device Feature Header are displayed. The --show-uuid option additionally displays the GUID for each feature. OFFSET can be used to specify the beginning of the DFL in the MMIO region. --dump displays the raw DFH contents in hex format. COUNT limits the number of DFH entries traversed. DELAY causes a pause between each printout. --safe examines each DFH offset for proper alignment.
The dump command provides a means to dump the MMIO space in ASCII hex or binary format.
opae.io dump [--offset [OFFSET]] [-o,--output OUTPUT] [-f,--format {bin,hex}] [-c,--count COUNT]
OFFSET specifies the starting MMIO offset. OUTPUT gives the name of a file to capture the dump output, where sys.stdout is used by default. --format allows changing the output format. COUNT specifies the number of qwords to dump.
The peek command option reads and displays a CSR value.
opae.io peek OFFSET
The poke command option writes a given value to a CSR.
opae.io poke OFFSET VALUE
opae.io can also execute Python scripts from the command line. These Python scripts may contain calls to the device built-in functions that are available during an interactive session. Refer to the description of interactive mode for details.
opae.io script myscript.py a b c
In order to enter the interactive mode of opae.io, simply invoke it and optionally pass the desired device address and MMIO region options.
opae.io [-d,--device PCI_ADDR] [-r,--region REGION]
"},{"location":"sw/fpga_tools/opae.io/opae.io/#interactive-mode","title":"INTERACTIVE MODE","text":"Upon entering interactive mode, opae.io begins a Python interpreter session and displays the command prompt shown below:
0000:3f:00.0[0]>>
The first portion of the prompt shows the address of the active PCIe device, here 0000:3f:00.0. The part in square brackets shows the active MMIO region, here [0].
The interpreter waits for a valid Python command, then attempts to execute the given command in the usual way. The only differences between the traditional Python command intepreter and opae.io are that opae.io provides 1) the notion of an active PCIe device and MMIO region and 2) several built-in functions and objects that allow manipulating the active device and MMIO region.
"},{"location":"sw/fpga_tools/opae.io/opae.io/#built-in-functions","title":"BUILT-IN FUNCTIONS","text":"The opae.io built-in functions assume an active device and MMIO region. Attempting to use the built-in functions without first opening a device and region will result in errors.
peek(OFFSET)
The peek built-in function reads and displays a CSR value from the active device and region, at the offset supplied by its argument.
0000:3f:00.0[0]>> peek(0x28) 0xdeadbeef
poke(OFFSET, VALUE)
The poke built-in function writes the given VALUE to the current MMIO region at the given OFFSET.
0000:3f:00.0[0]>> poke(0x28, 0xdeadbeef)
read_csr(OFFSET)
The read_csr built-in function returns the value of the CSR at the active MMIO region and the given OFFSET.
0000:3f:00.0[0]>> print('0x{:0x}'.format(read_csr(0x28))) 0xdeadbeef
write_csr(OFFSET, VALUE)
The write_csr built-in function writes the given VALUE to the current MMIO region at the given OFFSET.
0000:3f:00.0[0]>> write_csr(0x28, 0xdeadbeef)
device(PCI_ADDR)
The device built-in function allows changing the active PCIe device.
0000:3f:00.0[0]>> device('0000:2b:00.0') 0000:2b:00.0>>
region(REGION)
The region built-in function allows changing the active MMIO region.
0000:2b:00.0>> region(0) 0000:2b:00.0[0]>>
allocate_buffer(SIZE)
The allocate_buffer built-in function creates and returns a DMA buffer object. The underlying buffer will be SIZE bytes in length.
0000:2b:00.0[0]>> b1 = allocate_buffer(4096) 0000:2b:00.0[0]>> print(b1.size, '0x{:0x}'.format(b1.address), b1.io_address) 4096 0x7f9361c66000 0
version()
The version built-in function returns a tuple containing the four components used to identify the opae.io version:
0000:2b:00.0[0]>> print(version()) ('opae.io', 0, 2, 0)
"},{"location":"sw/fpga_tools/opae.io/opae.io/#built-in-objects","title":"BUILT-IN OBJECTS","text":"opae.io interactive mode provides two global objects corresponding to the current device and that device's current MMIO region. These objects are referred to by global variables the_device and the_region, respectively.
The device class:
the_device.descriptor() : method that returns the integer file descriptor of the VFIO container.
0000:2b:00.0[0]>> print(the_device.descriptor()) 5
the_device.repr() : method that is invoked when a device object is printed.
0000:2b:00.0[0]>> print(the_device) 0000:2b:00.0
the_device.allocate(SIZE) : method that allocates and returns a system_buffer object. The buffer will be mapped into the DMA space of the_device.
0000:2b:00.0[0]>> b1 = the_device.allocate(4096)
the_device.pci_address() : read-only property that returns the PCIe address of the_device.
0000:2b:00.0[0]>> print(the_device.pci_address) 0000:2b:00.0
the_device.num_regions : read-only property that returns the number of MMIO regions in the_device.
0000:2b:00.0[0]>> print(the_device.num_regions) 2
the_device.regions : read-only property that returns a list of the active MMIO regions of the_device:
0000:2b:00.0[0]>> print(the_device.regions) [0, 2]
The region class:
the_region.write32(OFFSET, VALUE) : method that writes a 32-bit VALUE to the CSR at OFFSET.
the_region.read32(OFFSET) : method that returns a 32-bit CSR at the given OFFSET.
0000:2b:00.0[0]>> the_region.write32(0x28, 0xdeadbeef) 0000:2b:00.0[0]>> print('0x{:0x}'.format(the_region.read32(0x28))) 0xdeadbeef
the_region.write64(OFFSET, VALUE): method that writes a 64-bit VALUE to the CSR at OFFSET.
the_region.read64(OFFSET): method that returns a 64-bit CSR at the given OFFSET.
0000:2b:00.0[0]>> the_region.write64(0x28, 0xbaddecaf) 0000:2b:00.0[0]>> print('0x{:0x}'.format(the_region.read64(0x28))) 0xbaddecaf
the_region.index(): method that returns the MMIO index of the_region.
0000:2b:00.0[0]>> print(the_region.index()) 0
the_region.repr(): method that is invoked when a region object is printed.
0000:2b:00.0[0]>> print(the_region) 0
the_region.len(): method that is invoked to determine the MMIO region size.
0000:2b:00.0[0]>> print(len(the_region)) 524288
The allocate_buffer() built-in function and the device.allocate() method return objects of type system_buffer.
The system_buffer class is as follows:
buf.size: read-only property that gives the buffer size.
0000:2b:00.0[0]>> print(b1.size) 4096
buf.address: read-only property that gives the buffer's user mode virtual address.
0000:2b:00.0[0]>> print('0x{:0x}'.format(b1.address)) 0x7f2c15d8200
buf.io_address: read-only property that gives the buffer's IO address.
0000:2b:00.0[0]>> print('0x{:0x}'.format(b1.io_address)) 0x0
buf.__getitem__ and buf.__setitem__: indexing get/set of 64-bit data item.
0000:2b:00.0[0]>> b1[0] = 0xdecafbad 0000:2b:00.0[0]>> print('0x{:0x}'.format(b1[0])) 0xdecafbad
buf.read8(OFFSET) buf.read16(OFFSET) buf.read32(OFFSET) buf.read64(OFFSET) : methods that read the given size data item from the given buffer OFFSET.
buf.fill8(VALUE) buf.fill16(VALUE) buf.fill32(VALUE) buf.fill64(VALUE) : methods that fill the buffer with the given VALUE, using the given size.
b1.compare(b2): method that compares buffers. The method returns the index of the first byte that miscompares, or the length of b1.
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/","title":"opaeuio","text":""},{"location":"sw/fpga_tools/opaeuio/opaeuio/#synopsis","title":"SYNOPSIS","text":"opaeuio [-h] [-i] [-r] [-d DRIVER] [-u USER] [-g GROUP] [-v] [device]
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/#description","title":"DESCRIPTION","text":"The opaeuio command enables the binding/unbinding of a DFL device to/from the dfl-uio-pdev device driver. See https://kernel.org/doc/html/v4.14/driver-api/uio-howto.html for a description of uio.
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/#options","title":"OPTIONS","text":"device The DFL device name, eg dfl_dev.10
-h, --help
Display command-line help and exit.\n
-i, --init
Specifies binding mode operation - initialize the given device for uio.\nUsed in conjunction with -u, -g, and -d.\n
-r, --release
Specifies unbinding mode operation - release the given device from uio.\n
-d DRIVER, --driver DRIVER
Specifies the device driver to bind to when binding to uio.\nThe default value is dfl-uio-pdev.\n
-u USER, --user USER
The user ID to assign when binding to uio. A new device node is created in\n/dev when the device is bound to uio. Use this option to specify\nthe new device owner.\n
-g GROUP, --group GROUP
The group ID to assign when binding to uio. Use this option to specify the\nnew device group for the device created in /dev.\n
-v, --version
Display script version information and exit.\n
"},{"location":"sw/fpga_tools/opaeuio/opaeuio/#examples","title":"EXAMPLES","text":"opaeuio -h opaeuio -v sudo opaeuio -i -u lab -g labusers dfl_dev.10 sudo opaeuio -r dfl_dev.10
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/","title":"opaevfio","text":""},{"location":"sw/fpga_tools/opaevfio/opaevfio/#synopsis","title":"SYNOPSIS","text":"opaevfio [-h] [-i] [-r] [-d DRIVER] [-u USER] [-g GROUP] [-n] [-v] [addr]
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/#description","title":"DESCRIPTION","text":"The opaevfio command enables the binding/unbinding of a PCIe device to/from the vfio-pci device driver. See https://kernel.org/doc/Documentation/vfio.txt for a description of vfio-pci.
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/#options","title":"OPTIONS","text":"addr The PCIe address of the device in ssss:bb:dd.f format, eg 0000:7f:00.0
-h, --help
Display command-line help and exit.\n
-i, --init
Specifies binding mode operation - initialize the given addr for vfio.\nUsed in conjunction with -u, -g, and -n.\n
-r, --release
Specifies unbinding mode operation - release the given addr from vfio.\nUsed in conjunction with -d.\n
-d DRIVER, --driver DRIVER
Specifies the device driver to bind to when releasing from vfio.\nWhen omitted, the device is not rebound to a driver (default).\n
-u USER, --user USER
The user ID to assign when binding to vfio. A new device node is created in\n/dev/vfio when the device is bound to vfio-pci. Use this option to specify\nthe new device owner.\n
-g GROUP, --group GROUP
The group ID to assign when binding to vfio. Use this option to specify the\nnew device group for the device created in /dev/vfio.\n
-n, --no-sriov
Do not enable SR-IOV when binding to vfio. The default value for this option\nis FALSE, ie the script should specify SR-IOV functionality when binding to\nthe vfio-pci driver. When omitted, the modprobe command which loads the vfio-pci\ndriver will contain the `enable_sriov=1` option. When given, it will not.\n
-v, --version
Display script version information and exit.\n
"},{"location":"sw/fpga_tools/opaevfio/opaevfio/#examples","title":"EXAMPLES","text":"opaevfio -h opaevfio -v sudo opaevfio -i -u lab -g labusers 0000:7f:00.0 sudo opaevfio -r 0000:7f:00.0
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/","title":"Pac hssi config","text":"# pac_hssi_config #\n\n## SYNOPSIS ##\n```console\npac_hssi_config.py [-h] subcommand [subarg] [bdf]\n
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#description","title":"DESCRIPTION","text":"The pac_hssi_config.py tool exercises the Ethernet 10 Gbps (10GbE) and 40GbE transceivers for designs using the Intel\u00ae Programmable Acceleration Card (PAC) with Intel Arria\u00ae 10 GX FPGA. This tool does not support the Intel Xeon\u00ae Processor with Integrated FPGA.
The two required arguments to the pac_hssi_config.py tool specify the subcommand and bus, device, and function (BDF) for the PCIe device under test. You must provide the BDF parameter for systems with more than one PCIe card.
.. note::\n If you do not provide the BDF when required, the command prints a list of valid BDFs for the system. You can also\n determine the BDF using the ``lspci`` command.\n
For usage help, type the following at a command prompt:
pac_hssi_config.py [-h|--help]
To configure the network ports, send data, and read statistics, use the following form of the pac_hssi_config.py script:
pac_hssi_config.py subcommand [subarg] [bdf]
Only a subset of subcommand arguments support subarg.
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#table-1-general-subcommands","title":"Table 1. General Subcommands","text":"Subcommand Subarg Description stat N/A Prints high speed serial interface (HSSI) controller statistics. eeprom N/A Reads the 128-bit unique board ID, MAC address, and board-specific IDs from EEPROM."},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#table-2-1040-gbe-traffic-generation-subcommands","title":"Table 2. 10/40 GbE Traffic Generation Subcommands","text":"Subcommand Subarg Description e10init and e40init N/A Initializes HSSI PHY to 10GbE or 40GbE mode. Clears statistics and enable internal HSSI transceiver loopback. e10loop and e40loop On/Off Turns on or off internal HSSI transceiver loopback. e10reset and e40reset On/Off Asserts or deasserts AFU reset. Clears packet statistics and disables internal HSSI transceiver loopback. e10send and e40send N/A Sends 1,000,000 1500-byte packets. For 10GbE sends packets on all four ports. 40GbE has a single port. e10stat and e40stat N/A Prints packet statistics. e10statclr and e40statclr N/A Clears packet statistics. Use this command after switching loopback modes to clear any transient statistics accumulated during the mode switch. The transceiver equalization eqwrite and eqread subcommands write and read transceiver equalization settings. These subcommands require you to specify the transceiver channel, the equalization setting, and the value (for writes). Use the following form for the eqwrite command:
pac_hssi_config.py eqwrite [transceiver channel number] [equalization setting] [equalization value] [bdf]
Use the following form for the eqreadcommand:
pac_hssi_config.py eqread [transceiver channel number] [equalization setting] [bdf]
"},{"location":"sw/fpga_tools/pac_hssi_config/pac_hssi_config/#table-3-transceiver-equalization-subcommands","title":"Table 3. Transceiver Equalization Subcommands","text":"Subcommand Channel Number Equalization Setting Value eqwrite 0-3 0 = Continuous time-linear equalization (CTLE) 1 = Variable gain amplifier (VGA) 2 = DCGAIN 3 = Pre-emphasis first post-tap 4 = Pre-emphasis second post-tap 5 = Pre-emphasis first pre-tap 6 = Pre-emphasis second pre-tap 7 = Differential output voltage (VOD) Specifies the value for the specified equalization setting. eqread 0-3 0 = Continuous time-linear equalization (CTLE) 1 = Variable gain amplifier (VGA) 2 = DCGAIN 3 = Pre-emphasis first post-tap 4 = Pre-emphasis second post-tap 5 = Pre-emphasis first pre-tap 6 = Pre-emphasis second pre-tap 7 = Differential output voltage (VOD) N/A For more information about reconfiguring transceiver analog parameter settings In Arria\u00ae 10 devices, refer to \"Changing PMA Analog Parameters\" in the Intel\u00ae Arria\u00ae 10 Transceiver PHY User Guide.
"},{"location":"sw/fpga_tools/packager/packager/","title":"packager","text":""},{"location":"sw/fpga_tools/packager/packager/#synopsis","title":"SYNOPSIS","text":"packager <cmd> [arguments]
"},{"location":"sw/fpga_tools/packager/packager/#description","title":"Description","text":"The packager provides tools that Accelerator Functional Unit (AFU) developers use to create Accelerator Function (AF) files. The AF file is the programming file for an AFU on Intel\u00ae FPGA platforms. The packager tool concatenates the metadata from the JSON file to a raw binary file (.rbf) that the Intel Quartus\u00ae Prime software generates.
The packager's only function is to create an AF file. Refer to Packager Command Syntax for more information about invoking the packager. The packager depends on a JSON file to describe AFU metadata. Refer to Accelerator Description File for more information about the JSON file.
The packager requires Python 2.7.1 and Python 2.7.3. The tool indicates if it is being called with a compatible of Python.
"},{"location":"sw/fpga_tools/packager/packager/#packager-command-syntax","title":"Packager Command Syntax","text":"The packager is a command line tool with the following syntax:
$ packager <cmd> [arguments]
The following table describes the <CMD> arguments:
Command Arguments Description create-gbs --rbf=<RBF_PATH> --afu=<AFU_JSON_PATH> --gbs=<GBS_PATH> --set-value=<key>.<value> Creates the AF file. The engineering name for this file is the green bit stream, abbreviated gbs. The --rbf and --afu arguments are required. <RBF_PATH> is the path to the RBF file for the AFU. The Quartus\u00ae Prime software generates this RBF by compiling the AFU design. <AFU_JSON_PATH> is the path to the Accelerator Description file. This is a JSON file that describes the metadata that create-gbs appends to the RBF. <GBS_PATH> is the path to the RBF file for the FPGA Interface Manager (FIM) that contains the FPGA interface unit and other interfaces. If you do not specify the --gbs, the command defaults to <rbf_name>.gbs. You can use the optional --set-value=<key>.<value> argument to set values for JSON metadata. To set more than one JSON value, list a series of <key>.<value> pairs. modify-gbs --gbs=<gbs_PATH> Modifies the AF file. The --input-gbsargument is required. If you do not provide the --output-gbs argument, modify-gbs overwrites the --input-gbs file. Use the --set-value=<key>.<value> argument to set values for JSON metadata. To set more than one JSON value, list a series of <key>.<value> pairs. gbs-info --input-gbs=<gbs_PATH> Prints information about the AF file. The --input-gbs argument is required. get-rbf --gbs=<GBS_PATH> --rbf=<RBF_PATH> Creates the RBF by extracting it from the AF file. The --gbsargument is required. If you do not specify the --rbf argument, the command defaults to <gbs_name.rbf . None, or any <CMD> --help Summarizes the <CMD> options. Typing packager --help gives a list of <CMD> values. Typing packager <CMD> --help provides detailed help for <CMD>"},{"location":"sw/fpga_tools/packager/packager/#examples","title":"Examples","text":"To generate an AF file, run:
$ packager create-gbs --rbf=<RBF_PATH> --afu=<AFU_JSON_PATH> --gbs=<GBS_PATH>
TIP: JSON files are very particular about syntax such as trailing commas. If you are getting errors, use jsonlint.com to validate that your JSON is formatted correctly.
To modify metadata in an existing AF, run the following command:
$ packager modify-gbs --input-gbs=<PATH_TO_GBS_TO_BE_MODIFIED> --outputgbs=<NAME_FOR_NEW_GBS> --set-value <key>:<value>
You can pass in a number of : pairs with --set-value to update values in an AF.
To print the metadata of an existing AF:
$ packager get-info --gbs=<GBS_PATH>
To extract the RBF from the AF:
$ packager get-rbf --gbs=<GBS_PATH> --rbf=<NAME_FOR_RBF>
"},{"location":"sw/fpga_tools/packager/packager/#accelerator-description-file","title":"Accelerator Description File","text":"The Accelerator Description File is a JSON file that describes the metadata associated with an AFU. The Open Progammable Accelerator Engine (OPAE) uses this metadata during reconfiguration. Here is an example file:
{\n \"version\": 1,\n \"platform-name\": \"DCP\",\n \"afu-image\": {\n \"magic-no\": 488605312,\n \"interface-uuid\": \"01234567-89AB-CDEF-0123-456789ABCDEF\",\n \"power\": 0,\n \"accelerator-clusters\": [{\n \"name\": \"dma_test_afu\",\n \"total-contexts\": 1, \n \"accelerator-type-uuid\": \"331DB30C-9885-41EA-9081-F88B8F655CAA\"\n }\n ] \n }\n}\n
The packager stores these parameter values in the resultant AF. After reprogramming the AFU using partial reconfiguration (PR), the software driver reconfigures the PLLs by writing the clock-frequency-high and clock-frequency-low values (if present) over the PCIe\u00ae and CCI interfaces. .. note::
The JSON file format may change as the architecture evolves. Any changes to the current format trigger an update\nto the version number. \n
CATEGORY | NAME | TYPE | DESCRIPTION | MANDATORY ---------|------|------|-------------|:----------:| Per-AFU | version | Integer | Version of the metadata format. | Yes Per-AFU | magic-no (to be deprecated)| Integer | Magic no. Associated with the FPGA Interface Manager. | No Per-AFU | platform-name | String | Name of the platform for which the metadata is intended. The field value is \u201cDCP\u201d for Intel Acceleration Stack for FPGAs. | No Per-AFU | interface-uuid | UUID | Interface id associated with the FPGA Interface Manager. | Yes Per-AFU | power | Integer | Accelerator Function power consumption, in watts. Set to 0 for Intel Acceleration Stack for FPGAs platforms. | Yes Per-AFU | clock-frequency-low | Float | Clock frequency for 1st PLL (Clock network)1 in MHz. | No Per-AFU | clock-frequency-high | Float | Clock frequency for 2nd PLL (0 if absent) in MHz. | No Per-AFC Cluster | total-contexts | Integer | Number of AFCs in this cluster. Always be 1 in current architectures. | Yes Per-AFC Cluster | afc-type-uuid | UUID | AFC type = AFU ID in current architectures. | Yes Per-AFC Cluster | name | string | AFC name = AFU name in current architectures. | Yes
Date Intel Acceleration Stack Version Changes Made 2018.05.21 DCP 1.1 Beta (works with Quartus Prime Pro 17.1.1) Fixed typos."},{"location":"sw/fpga_tools/pci_device/pci_device/","title":"pci_device","text":""},{"location":"sw/fpga_tools/pci_device/pci_device/#synopsis","title":"SYNOPSIS","text":"pci_device [-h] [-E] device-filter [{aer,bind,plug,remove,rescan,topology,unbind,unplug,vf}]
"},{"location":"sw/fpga_tools/pci_device/pci_device/#description","title":"DESCRIPTION","text":"pci_device is a tool to aid in common operations for managing PCIe devices and drivers.
"},{"location":"sw/fpga_tools/pci_device/pci_device/#options","title":"OPTIONS","text":""},{"location":"sw/fpga_tools/pci_device/pci_device/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"`device filter`\n\nPCIe address of a device or vendor/device ID pair.\nThe PCIe address follows the format of [segment:]bus:device.function\nwhile the vendor/device ID pair follows the format [vendor ID]:[device ID]\nwhere at least one of these must be present.\n\n`{aer,bind,plug,remove,rescan,topology,unbind,unplug,vf}`\n\naction to perform on device\n\n`aer`\nPerform AER (Advanced Error Reporting) operations.\nThe aer action has its own sub-commands which are listed below:\n\n* `dump` sub-command will print out the AER error counters as reported\n by the sysfs files for the device.\n* `mask` can either print out the current AER mask bits or set them\n * If `show` or `print` (or nothing) is given after the `mask`\n command, it will show the current mask bits for AER.\nBy default output will be written in stdout but can be written to an\noutput file if `-o|--output FILENAME` argument is given.\n * If `all` is given after the `mask` command, it will mask all bits\n (by setting the values to 0xffffffff and 0xffffffff).\n * If `off` is given after the `mask` command, it will unmask all\n bits (by setting the values to 0x0 and 0x0).\n * If two numbers are present after the `mask` command, those two\n numbers will be used to set the mask bits.\nValues for setting the mask can also be read in from an input file if\n`-i|--input FILENAME` argument is given.\n\n_NOTE_: mask related operations require root privileges\n\n`bind`\n\nAssociate a device with its driver.\n\n`plug`\n\nRestore a device that was previously given to `pci_device <device> unplug`\n\n`remove`\n\nRemove the pci device from the pci bus\n\n`rescan`\n\nRescan the bus as identified by the bus component of the PCIe device address\n\n'topology`\n\nPrint the PCIe topology from the root port to the PCIe device.\nThis shows the PCIe tree rooted at the PCIe root port.\nEach line shows the the PCIe address, vendor ID, and device ID along with\nthe driver bound to the device. The indentation is used to show\nparent/child relationship of devices.\n\nThe line listing the target PCIe device as identified by the given PCIe\naddress will be highlighted in green while the endpoints will be\nhighlighted in cyan.\n\nThe example below shows the topology of an N3000 device with eight virtual\nfunctions created from one of the Ethernet controllers:\n\n```console\n[pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)] (pcieport)\n [pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:3c:09.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:3f:00.0), pci_id(0x8086, 0x0b30)] (dfl-pci)\n [pci_address(0000:3c:11.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:43:00.0), pci_id(0x8086, 0x0b32)] (no driver)\n [pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:3d:02.0), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:00.1), pci_id(0x8086, 0x0d58)] (i40e)\n [pci_address(0000:3d:02.7), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.5), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.3), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.1), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n [pci_address(0000:3d:02.6), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.4), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3d:02.2), pci_id(0x8086, 0x154c)] (iavf)\n [pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)] (pcieport)\n [pci_address(0000:41:00.0), pci_id(0x8086, 0x0d58)] (i40e)\n [pci_address(0000:41:00.1), pci_id(0x8086, 0x0d58)] (i40e)\n\n```\n\n`unbind`\n\nUnbind the driver bound to the device.\n\n`unplug`\n\nRemove device from PCI bus in anticipation of a RSU event by configuring its root port and associated endpoints.\n\n`vf`\n\nCreate/destroy VFs (virtual functions) by setting the number here.\nThe number given here will be written to sriov_numvfs sysfs file triggering\nthe PCIe subsystem to create/destroy VFs so that the current number of VFs\nwill be equal to the given number. If the number given is outside of the total VFs supported, an error message will be displayed to indicate this.\n
"},{"location":"sw/fpga_tools/pci_device/pci_device/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"`-h, --help`\n\nshow this help message and exit\n\n`-E, --other-endpoints`\n\nperform action on peer PCIe devices\n
"},{"location":"sw/fpga_tools/pci_device/pci_device/#examples","title":"EXAMPLES","text":"pci_device 0000:3d:00.0 remove\npci_device 0000:3d:00.0 rescan\npci_device 3d:00.0 topology\npci_device :0b30 topology\npci_device :0b30 aer\npci_device :0b30 aer mask\npci_device :0b30 aer mask all\npci_device :0b30 aer mask -o mask.dat\npci_device :0b30 aer mask -i mask.dat\n
"},{"location":"sw/fpga_tools/rsu/rsu/","title":"rsu","text":""},{"location":"sw/fpga_tools/rsu/rsu/#synopsis","title":"SYNOPSIS","text":"rsu [-h] [-d] {bmc,bmcimg,retimer,fpga,sdm,fpgadefault} [PCIE_ADDR]\n
"},{"location":"sw/fpga_tools/rsu/rsu/#description","title":"DESCRIPTION","text":""},{"location":"sw/fpga_tools/rsu/rsu/#mode-1-rsu","title":"Mode 1: RSU","text":"rsu bmc --page=(user|factory) [PCIE_ADDR]\nrsu retimer [PCIE_ADDR]\nrsu fpga --page=(user1|user2|factory) [PCIE_ADDR]\nrsu sdm --type=(sr|pr|sr_cancel|pr_cancel) [PCIE_ADDR]\n
Perform RSU (remote system update) operation on PAC device given its PCIe address. An RSU operation sends an instruction to the device to trigger a power cycle of the card only. This will force reconfiguration from flash for either BMC, Retimer, SDM, (on devices that support these) or the FPGA.
"},{"location":"sw/fpga_tools/rsu/rsu/#mode-2-default-fpga-image","title":"Mode 2: Default FPGA Image","text":"rsu fpgadefault --page=(user1|user2|factory) --fallback=<csv> [PCIE_ADDR]\n
Set the default FPGA boot sequence. The --page option determines the primary FPGA boot image. The --fallback option allows a comma-separated list of values to specify fallback images.
"},{"location":"sw/fpga_tools/rsu/rsu/#positional-arguments","title":"POSITIONAL ARGUMENTS","text":"{bmc,bmcimg,retimer,fpga,sdm,fpgadefault}
type of RSU operation or set Default FPGA Image operation.
PCIE_ADDR PCIe address of device to do rsu (e.g. 04:00.0 or 0000:04:00.0)
"},{"location":"sw/fpga_tools/rsu/rsu/#optional-arguments","title":"OPTIONAL ARGUMENTS","text":"-h, --help show this help message and exit
-d, --debug log debug statements
--force force rsu operation
"},{"location":"sw/fpga_tools/rsu/rsu/#example","title":"EXAMPLE","text":"# rsu bmc --page=user 25:00.0\n
Triggers a boot of the BMC image (user page) for the device with PCIe address 25:00.0.
# rsu bmc --page=factory 25:00.0\n
Triggers a factory boot of the BMC image for the device with PCIe address 25:00.0.
# rsu fpga --page=user2 25:00.0\n
Triggers a reconfiguration of the FPGA (user2 page) for the device with PCIe address 25:00.0.
# rsu --force fpga --page=user2 25:00.0\n
Forces a reconfiguration of the FPGA (user2 page) for the device with PCIe address 25:00.0. Default behavior is to not perform the rsu operation if DPC (downstream port containment) is not supported and AER (advanced error reporting) is also not supported. Using --force changes this behavior to perform rsu operation regardless but may result in a surprise removal of pci devices which may cause the Linux kernel to panic.
# rsu fpga --page=factory 25:00.0\n
Triggers a factory reconfiguration of the FPGA for the device with PCIe address 25:00.0.
# rsu sdm --type=sr 25:00.0\n
Triggers Static Region key programming for the device with PCIE address 25:00.0.
# rsu fpgadefault --page=factory --fallback=user1,user2 25:00.0\n
Sets the FPGA boot sequence to factory with fallbacks user1, user2.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/","title":"Super Remote System Update User Guide","text":".. toctree::\n.. highlight:: c\n.. highlight:: console\n
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#overview","title":"Overview","text":"Intel Programmable Acceleration Card (PAC) devices are comprised of multiple processors and controllers that execute firmware. Maintaining and updating these firmware images manually is error-prone and does not scale well within the Data Center. The solution described here is derived with the following goals in mind:
- The ability to update one or more (possibly all) firwmare images with a single package.
- The ability to complete all firmware updates within a stipulated time window.
- The ability to update each PAC in the server, all servers in a Data Center, and multiple Data Centers remotely.
- The ability to remotely initiate download of the package and its installation with a single command per server instance.
- The ability to roll back firmware to a previous revision.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#implementation","title":"Implementation","text":"A single package containing firmware images for all programmable parts on a PAC is delivered as an RPM, eg opae-super-rsu-n3000.M.m.p-r.noarch.rpm. The RPM revision will sequentially increase with every update.
Installing or upgrading the RPM invokes the complete update of all programmable parts on all PAC boards in the system.
The standard RPM dependency framework ensures that correct versions of dependecy packages opae-intel-fpga-driver and fpga-tools-extra are installed on the system.
Rolling back is achieved by uninstalling the current version and re-installing a previous version of the RPM.
.. note::
Note: once Secure Update is deployed, roll back restrictions shall be implemented to prevent\nrollback attacks.\n
RPM management on remote systems is standard practice, requiring no new infrastructure/training.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#details","title":"Details","text":"The post-install hook of the opae-super-rsu-n3000 RPM is leveraged to call out to the super-rsu Python script to update all PAC boards. super-rsu uses the manifest file packaged within opae-super-rsu-n3000 to associate a firmware image with its version. Each of the firmware images contained in opae-super-rsu-n3000 is placed on the target system in /usr/share/opae/n3000.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#algorithm","title":"Algorithm","text":" - Acquire the current firmware versions of all programmable parts.
- For each programmable image, if the installed version of firmware does not equal the version provided in the RPM manifest file, then update the firmware image, and set image_updated to True.
- After all updates, if image_updated, then initiate a safe reboot of all boards in the system.
- After safe reboot, verify that the reported firmware versions match those of the RPM manifest. If they do not match, then RPM installation exits with a failing status.
- Run board self test. If the self test fails, then the RPM installation exits with a failing status.
- If all of the above checks is successful, then RPM installation exits with a success status.
"},{"location":"sw/fpga_tools/super-rsu/super-rsu/#dependencies","title":"Dependencies","text":" - The standard Python package for the distro (version 2.7).
- The opae-intel-fpga-driver RPM. (version determined by opae-super-rsu-n3000)
- The opae-tools-extra RPM. (version determined by opae-super-rsu-n3000)
"},{"location":"sw/fpga_tools/userclk/userclk/","title":"userclk","text":""},{"location":"sw/fpga_tools/userclk/userclk/#synopsis","title":"SYNOPSIS","text":"userclk [-hv] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR] [-H <User clock high frequency>] -L <User clock low frequency>]
"},{"location":"sw/fpga_tools/userclk/userclk/#description","title":"DESCRIPTION","text":"userclk sets the frequency range for an AFU.
"},{"location":"sw/fpga_tools/userclk/userclk/#examples","title":"EXAMPLES","text":"./userclk -B 0x5e -H 400 -L 200
Sets AFU frequency.
"},{"location":"sw/fpga_tools/userclk/userclk/#options","title":"OPTIONS","text":"-v,--version
Prints version information and exits.
-S,--segment
FPGA segment number.
-B,--bus
FPGA Bus number.
-D,--device
FPGA Device number.
-F,--function
FPGA function number.
-H,--freq-high
User clock high frequency.
-L,--freq-low
User clock low frequency.
Date Intel Acceleration Stack Version Changes Made 2018.05.21 DCP 1.1 Beta (works with Quartus Prime Pro 17.1.1) Fixed typos."},{"location":"sw/fpga_tools/vabtool/vabtool/","title":"vabtool","text":""},{"location":"sw/fpga_tools/vabtool/vabtool/#synopsis","title":"SYNOPSIS","text":"vabtool [-r RETRIES] [-d] [-y] [-v] <ACTION>
Where ACTION is defined as one of the following:
vabtool sr_key_provision PCIE_ADDRESS SR_RKH_FILE FPGA_IMG_FILE vabtool sr_status PCIE_ADDRESS vabtool pr_key_provision PCIE_ADDRESS PR_AUTH_CERT_FILE PR_RKH_FILE vabtool pr_status PCIE_ADDRESS vabtool sr_key_cancel PCIE_ADDRESS SR_RKH_CANCEL_FILE vabtool sr_cancel_status PCIE_ADDRESS vabtool pr_key_cancel PCIE_ADDRESS PR_RKH_CANCEL_FILE vabtool pr_cancel_status PCIE_ADDRESS
"},{"location":"sw/fpga_tools/vabtool/vabtool/#description","title":"DESCRIPTION","text":"The vabtool command helps perform Vendor Authenticated Boot provisioning of Static Region and Partial Reconfiguration Region key hashes and helps perform SR and PR hash cancellation and status reporting.
"},{"location":"sw/fpga_tools/vabtool/vabtool/#options","title":"OPTIONS","text":"-r RETRIES, --retries RETRIES
Specifies the number of times a failed SR or PR key provision is to be\nretried. The default value for RETRIES is 3.\n
-d, --dry-run
Don't execute the actual fpgasupdate and rsu commands, but only print\nthe commands that would be executed during a normal run of the script.\n
-y, --yes
The tool will respond with an automatic Yes answer to all confirmation\nprompts posed by the sub-tools.\n
-v, --version
Display script version information and exit.\n
"},{"location":"sw/fpga_tools/vabtool/vabtool/#examples","title":"EXAMPLES","text":"sudo vabtool -y sr_key_provision 0000:bc:00.0 my_sr_rkh.bin my_fpga.bin sudo vabtool sr_status 0000:bc:00.0 sudo vabtool -y pr_key_provision 0000:bc:00.0 pr_auth_cert.bin my_pr_rkh.bin sudo vabtool pr_status 0000:bc:00.0 sudo vabtool sr_key_cancel 0000:bc:00.0 my_sr_rhk_cancel.bin sudo vabtool sr_cancel_status 0000:bc:00.0 sudo vabtool pr_key_cancel 0000:bc:00.0 my_pr_rhk_cancel.bin sudo vabtool pr_cancel_status 0000:bc:00.0
"},{"location":"sw/install_guide/installation_guide/","title":"OPAE Installation Guide","text":""},{"location":"sw/install_guide/installation_guide/#how-to-download-the-opae-sdk","title":"How to download the OPAE SDK","text":"OPAE SDK releases are available on GitHub. Source code for the OPAE DFL device driver for Linux is also available on GitHub.
"},{"location":"sw/install_guide/installation_guide/#install-the-fedora","title":"Install the Fedora","text":"Download the Fedora (x86_64 version) installation file in fedora, and install the Fedora in yourserver. You can choose Fedora Workstation or Fedora server.
"},{"location":"sw/install_guide/installation_guide/#build-the-kernel-and-dfl-drivers","title":"Build the kernel and DFL drivers","text":"For building the OPAE kernel and kernel driver, the kernel development environment is required. So before you build the kernel, you must install the required packages. Run the following commands:
$ sudo dnf install gcc gcc-c++ make kernel-headers kernel-devel elfutils-libelf-devel ncurses-devel openssl-devel bison flex\n
Download the OPAE upstream kernel tree from github, for example download from fpga-ofs-dev-5.15-lts branch.
$ git clone https://github.com/OPAE/linux-dfl.git -b fpga-ofs-dev-5.15-lts\n
Configure the kernel.
$ cd linux-dfl\n$ cp /boot/config-`uname -r` .config\n$ cat configs/dfl-config >> .config\n$ echo 'CONFIG_LOCALVERSION=\"-dfl\"' >> .config\n$ echo 'CONFIG_LOCALVERSION_AUTO=y' >> .config\n$ sed -i -r 's/CONFIG_SYSTEM_TRUSTED_KEYS=.*/CONFIG_SYSTEM_TRUSTED_KEYS=\"\"/' .config\n$ sed -i '/^CONFIG_DEBUG_INFO_BTF/ s/./#&/' .config\n$ echo 'CONFIG_DEBUG_ATOMIC_SLEEP=y' >> .config\n$ make olddefconfig\n
Compile and install the new kernel.
$ make -j $(nproc)\n$ sudo make modules_install -j $(nproc)\n$ sudo make install\n
Build linux DFL Kernel instructions please also refer to: https://github.com/OPAE/linux-dfl/wiki/Build-the-linux-dfl-kernel
When install finished, reboot your system. When the system login again, verify the kernel version is correct. For example:
[figo@localhost linux-dfl]$ uname -a\nLinux localhost.localdomain 5.15.lts-dfl-g73e16386cda0 #6 SMP Mon Jun 13 21:21:31 -04 2022 x86_64 x86_64 x86_64\n
And also you can check the OPAE dfl drivers have auto-loaded.
[figo@localhost linux-dfl]$ lsmod | grep fpga\nifpga_sec_mgr 20480 1 intel_m10_bmc_secure\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n[figo@localhost linux-dfl]$ lsmod | grep dfl\ndfl_eth_group 36864 0\ndfl_fme_region 20480 0\ndfl_emif 16384 0\ndfl_n3000_nios 20480 0\ndfl_fme_br 16384 0\ndfl_fme_mgr 20480 1\ndfl_fme 49152 0\ndfl_afu 36864 0\ndfl_pci 20480 0\ndfl 40960 7 dfl_pci,dfl_fme,dfl_fme_br,dfl_eth_group,dfl_n3000_nios,dfl_afu,dfl_emif\nfpga_region 20480 3 dfl_fme_region,dfl_fme,dfl\nfpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br\nfpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme\n
"},{"location":"sw/install_guide/installation_guide/#build-the-opae-sdk","title":"Build the OPAE-SDK","text":"Before you build the OPAE SDK, you must install the required packages. Run the following commands:
"},{"location":"sw/install_guide/installation_guide/#rocky-linux-85","title":"Rocky Linux 8.5","text":"# dnf install -y 'dnf-command(config-manager)'\n# dnf config-manager --set-enabled powertools\n# dnf install -y epel-release\n# dnf check-update\n# dnf upgrade -y\n# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel linuxptp numactl-devel\n# python3 -m pip install jsonschema virtualenv pyyaml\n
"},{"location":"sw/install_guide/installation_guide/#fedora","title":"Fedora","text":"# dnf check-update\n# dnf upgrade -y\n# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc g++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel libedit-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel cli11-devel spdlog-devel linuxptp numactl-devel\n# pip3 install jsonschema virtualenv pyyaml\n
"},{"location":"sw/install_guide/installation_guide/#ubuntu-2204","title":"Ubuntu 22.04","text":"# apt-get update\n# apt-get upgrade -y\n# apt-get install -y python3 python3-pip python3-dev git gcc g++ make cmake uuid-dev libjson-c-dev libhwloc-dev libtbb-dev libedit-dev libudev-dev linuxptp pandoc devscripts debhelper doxygen libnuma-dev\n# pip3 install jsonschema virtualenv pyyaml pybind11\n
"},{"location":"sw/install_guide/installation_guide/#rhel-82","title":"RHEL 8.2","text":"Register and enable Red Hat subscription to install any packages on the system.
# subscription-manager register --proxy=PROXY --username=USER --password=PASSWORD --auto-attach\n
Set the RHEL version and install packages. Set proxy name and port number.
# subscription-manager release --set=8.2 --proxy proxy-name.com:port number\n# subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms\n# dnf upgrade -y\n# dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n# dnf install -y python3 python3-pip python3-devel gdb vim git gcc gcc-c++ make cmake libuuid-devel rpm-build systemd-devel nmap\n# dnf install -y python3-jsonschema json-c-devel tbb-devel rpmdevtools libcap-devel numactl-devel\n# dnf check-update || true\n# dnf install -y spdlog-devel cli11-devel python3-pyyaml python3-pybind11 hwloc-devel libedit-devel\n# python3 -m pip install --user jsonschema virtualenv pudb pyyaml\n
Install the latest version of cmake on top of the outdated cmake package from the package manager.
# cd cmake-3.25.1/\n# ./bootstrap --prefix=/usr\n# make\n# make install\n# which cmake\n/usr/bin/cmake\n
"},{"location":"sw/install_guide/installation_guide/#create-opae-sdk-packages","title":"Create opae-sdk packages","text":"Download the OPAE-SDK source code from github. For example, download from Master branch.
$ git clone https://github.com/OPAE/opae-sdk.git\n
Compile and build the OPAE-SDK RPMs (Fedora, Rocky, RHEL 8.2).
$ cd opae-sdk/packaging/opae/rpm\n$ ./create fedora\n
Note that if you find that your distribution has changed package names such that there is a conflict when building RPMs, you can install all of the build dependencies so that the SDK compiles and then build the RPMs in unrestricted mode:
$ cd opae-sdk/packaging/opae/rpm\n$ ./create unrestricted\n
After a successful compile, there are 3 rpm packages generated (Fedora, Rocky, RHEL8.2). For example:
opae-2.1.0-1.fc34.x86_64.rpm\nopae-devel-2.1.0-1.fc34.x86_64.rpm\nopae-extra-tools-2.1.0-1.fc34.x86_64.rpm\n
Compile and build the OPAE-SDK deb packages (Ubuntu 22.04).
$ cd opae-sdk/packaging/opae/deb\n$ ./create\n
After a successful compile, there are 3 deb packages generated (Ubuntu 22.04). For example:
opae_2.1.1-1_amd64.deb \nopae-devel_2.1.1-1_amd64.deb \nopae-extra-tools_2.1.1-1_amd64.deb\n
"},{"location":"sw/install_guide/installation_guide/#opae-sdk-installation-with-rpmdeb-packages","title":"OPAE SDK installation with rpm/deb packages","text":"The rpm packages generated in the previous step can be installed using these commands:
$ sudo dnf install ./*.rpm\n
The deb packages generated in the previous step can be installed using these commands:
$ sudo dpkg -i ./*.deb\n
When you installed the rpms, you can run fpgainfo command to check the FPGA FME infomation. For example:
[figo@localhost install_guide]$ fpgainfo fme\nBoard Management Controller, MAX10 NIOS FW version: D.2.1.24\nBoard Management Controller, MAX10 Build version: D.2.0.7\n//****** FME ******//\nObject Id : 0xEF00000\nPCIe s:b:d.f : 0000:08:00.0\nDevice Id : 0x0B30\nSocket Id : 0x00\nPorts Num : 01\nBitstream Id : 0x2300011001030F\nBitstream Version : 0.2.3\nPr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb\nBoot Page : user\n
To uninstall the OPAE rpms, you can use this commands
$ dnf list installed | grep opae\n$ sudo dnf remove opae*.x86_64\n
To uninstall the OPAE deb, you can use this commands
$ dpkg -l | grep opae\n$ dpkg -r opae-extra-tools:amd64\n$ dpkg -r opae-devel:amd64\n$ dpkg -r opae\n
"},{"location":"sw/install_guide/installation_guide/#fpga-device-access-permissions","title":"FPGA Device Access Permissions","text":"Access to FPGA accelerators and devices is controlled using file access permissions on the Intel\u00ae FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
$ sudo chmod a+rw /dev/dfl-port.0\n
"},{"location":"sw/install_guide/installation_guide/#memlock-limit","title":"Memlock limit","text":"Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
$ ulimit -l\n
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited\nuser1 soft memlock unlimited\n
This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:
* hard memlock unlimited\n* soft memlock unlimited\n
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
[Service]\nLimitMEMLOCK=infinity\n
"},{"location":"sw/install_guide/installation_guide/#hugepage-settings","title":"Hugepage Settings","text":"Users need to configure system hugepages to reserve 2MB-hugepages or 1GB-hugepages. For example, the 'hello_fpga' sample requires several 2MB-hugepages. And the fpgadiag tool requires several 1GB-hugepages.
The command below reserves 20 2M-hugepages:
$ sudo sh -c 'echo 20 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages'\n
The command below reserves 4 1GB-hugepages:
$ sudo sh -c 'echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'\n
For x86_64 architecture processors, user can use following command to find out avaiable hugepage sizes:
$ grep pse /proc/cpuinfo | uniq\nflags : ... pse ...\n
If this commands returns a non-empty string, 2MB pages are supported.
$ grep pse /proc/cpuinfo | uniq\nflags : ... pdpe1gb ...\n
If this commands returns a non-empty string, 1GB pages are supported.
"},{"location":"sw/pyopae/python_bindings/","title":"OPAE Python Bindings","text":"OPAE (Open Programmable Acceleration Engine) now includes Python bindings for interacting with FPGA resources. The OPAE Python API is built on top of the OPAE C++ Core API and its object model. Because of this, developing OPAE applications in Python is very similar to developing OPAE applications in C++ which significantly reduces the learning curve required to adapt to the Python API. While the object model remains the same, some static factory functions in the OPAE C++ Core API have been moved to module level methods in the OPAE Python API with the exception of the properties class. The goal of the OPAE Python API is to enable fast prototyping, test automation, infrastructure managment, and an easy to use framework for FPGA resource interactions that don\u2019t rely on software algorithms with a high runtime complexity.
Currently, the only Python package that is part of OPAE is opae.fpga
"},{"location":"sw/pyopae/python_bindings/#implementation","title":"Implementation","text":"The OPAE Python API is implemented by creating a Python extension using pybind11 <http://pybind11.readthedocs.io/en/stable>. This extension is created by using the pybind11 API which relies mostly on macros and compile time introspection to define the module initialization point as well as type converters between OPAE C++ Core types and OPAE Python types.
"},{"location":"sw/pyopae/python_bindings/#benefits","title":"Benefits","text":"The major benefits of using pybind11 for developing the OPAE Python API include, but are not limited to, the following features of pybind11:
- Uses C++ 11 standard library although it can use C++ 14 or C++17.
- Automatic conversions of shared_ptr types
- Built-in support for numpy and Eigen numerical libraries
- Interoperable with the Python C API
"},{"location":"sw/pyopae/python_bindings/#runtime-requirements","title":"Runtime Requirements","text":"Because opae.fpga is built on top of the opae-cxx-core API, it does require that the runtime libraries for both opae-cxx-core and opae-c be installed on the system (as well as any other libraries they depend on). Those libraries can be installed using the opae-libs package (from either RPM or DEB format - depending on your Linux distribution).
"},{"location":"sw/pyopae/python_bindings/#installation","title":"Installation","text":""},{"location":"sw/pyopae/python_bindings/#python-wheels","title":"Python Wheels","text":"The preferred method of installation is to use a binary wheel package for your version of Python.
The following table lists example names for different Python versions and platforms.
| Python Version | Python ABI | Linux Platform | Package Name | |\u2014\u2014\u2014\u2014\u2014-|\u2014\u2014\u2014\u2014\u2014\u2013|\u2014\u2014\u2014\u2014\u2014-|\u2014\u2014\u2014\u2014\u2013| | 2.7 | CPython w/ UCS4 | x86_64 | opae.fpga.-cp27-cp27mu-linux_x86_64.whl | | 3.4 | CPython w/ UCS4 | x86_64 | opae.fpga.-cp34-cp34mu-linux_x86_64.whl | | 3.6 | CPython w/ UCS4 | x86_64 | opae.fpga.-cp36-cp36mu-linux_x86_64.whl |
opae.fpga is currently not available in the Python Package Index but once it does become available, one should be able to install using pip by simply typing the following:
> pip install --user opae.fpga\n
"},{"location":"sw/pyopae/python_bindings/#installing-from-source","title":"Installing From Source","text":"In addition to the runtime libraries mentioned above, installing from source does require that the OPAE header files be installed as well as those header files for pybind11. The former can be installed with the opae-devel package and the latter can be installed by installing pybind11 Python module.
"},{"location":"sw/pyopae/python_bindings/#example-installation","title":"Example Installation","text":"The following example shows how to build from source by installing the prerequisites before running the setup.py file.
>sudo yum install opae-libs-<release>.x86_64.rpm\n>sudo yum install opae-devel-<release>.x86_64.rpm\n>pip install --user pybind11\n>pip install --user opae.fpga-<release>.tar.gz\n
NOTE: The pip examples above use the --user flag to avoid requiring root permissions. Those packages will be installed in the user\u2019s site-packages directory found in the user\u2019s .local directory.
"},{"location":"sw/pyopae/python_bindings/#example-scripts","title":"Example Scripts","text":"The following example is an implementation of the sample, hello_fpga.c, which is designed to configure the NLB0 diagnostic accelerator for a simple loopback.
import time\nfrom opae import fpga\n\nNLB0 = \"d8424dc4-a4a3-c413-f89e-433683f9040b\"\nCTL = 0x138\nCFG = 0x140\nNUM_LINES = 0x130\nSRC_ADDR = 0x0120\nDST_ADDR = 0x0128\nDSM_ADDR = 0x0110\nDSM_STATUS = 0x40\n\ndef cl_align(addr):\n return addr >> 6\n\ntokens = fpga.enumerate(type=fpga.ACCELERATOR, guid=NLB0)\nassert tokens, \"Could not enumerate accelerator: {}\".format(NlB0)\n\nwith fpga.open(tokens[0], fpga.OPEN_SHARED) as handle:\n src = fpga.allocate_shared_buffer(handle, 4096)\n dst = fpga.allocate_shared_buffer(handle, 4096)\n dsm = fpga.allocate_shared_buffer(handle, 4096)\n handle.write_csr32(CTL, 0)\n handle.write_csr32(CTL, 1)\n handle.write_csr64(DSM_ADDR, dsm.io_address())\n handle.write_csr64(SRC_ADDR, cl_align(src.io_address())) # cacheline-aligned\n handle.write_csr64(DST_ADDR, cl_align(dst.io_address())) # cacheline-aligned\n handle.write_csr32(CFG, 0x42000)\n handle.write_csr32(NUM_LINES, 4096/64)\n handle.write_csr32(CTL, 3)\n while dsm[DSM_STATUS] & 0x1 == 0:\n time.sleep(0.001)\n handle.write_csr32(CTL, 7)\n
This example shows how one might reprogram (Partial Reconfiguration) an accelerator on a given bus, 0x5e, using a bitstream file, m0.gbs.
from opae import fpga\n\nBUS = 0x5e\nGBS = 'm0.gbs'\ntokens = fpga.enumerate(type=fpga.DEVICE, bus=BUS)\nassert tokens, \"Could not enumerate device on bus: {}\".format(BUS)\nwith open(GBS, 'rb') as fd, fpga.open(tokens[0]) as device:\n device.reconfigure(0, fd)\n
"},{"location":"sw/tod/tod/","title":"Time of Day (ToD)","text":""},{"location":"sw/tod/tod/#synopsis","title":"SYNOPSIS","text":"The Intel FPGA ToD driver in the kernel space exposes ToD IP as PHC (PTP Hardware Clock) device to the Linux PTP (Precision Time Protocol) stack to synchronize the system clock to its ToD information. The phc2sys utility of Linux PTP stack is used to access ToD information and synchronize the system clock.
Install the Linux PTP utilities:
# sudo yum install linuxptp\n
phc_ctl and phc2sys utilities (linuxptp package) are used to control the PHC device and synchronize the system clock to its ToD information.
phc_ctl: directly controls PHC device clock.
Usage: phc_ctl [options] <device> -- [command]\n device ethernet or ptp clock device\nOptions:\n -l [num] set the logging level to 'num'\n -q do not print messages to the syslog\n -Q do not print messages to stdout\n -v prints the software version and exits\n -h prints this message and exits\nCommands:\n specify commands with arguments. Can specify multiple commands to be executed in order. \n Seconds are read as double precision floating point values.\n set [seconds] set PHC time (defaults to time on CLOCK_REALTIME)\n get get PHC time\n adj <seconds> adjust PHC time by offset\n freq [ppb] adjust PHC frequency (default returns current offset)\n cmp compare PHC offset to CLOCK_REALTIME\n caps display device capabilities (default if no command given)\n wait <seconds> pause between commands\n This command may be useful for sanity checking whether the PHC clock is\n running as expected.\n The arguments specified in seconds are read as double precision floating point\n values, and will scale to nanoseconds. This means providing a value of 5.5 means 5\n and one half seconds. This allows specifying fairly precise values for time.\n
phc2sys: synchronize two clocks.
Usage: phc2sys [options]\nAutomatic configuration:\n -a turn on autoconfiguration\n -r synchronize system (realtime) clock\n repeat -r to consider it also as a time source\nManual configuration:\n -c [dev|name] slave clock (CLOCK_REALTIME)\n -d [dev] master PPS device\n -s [dev|name] master clock\n -O [offset] slave-master time offset (0)\n -w wait for ptp4l\nOptions:\n -f [file] configuration file\n -E [pi|linreg] clock servo (pi)\n -P [kp] proportional constant (0.7)\n -I [ki] integration constant (0.3)\n -S [step] step threshold (disabled)\n -F [step] step threshold only on start (0.00002)\n -R [rate] slave clock update rate in HZ (1.0)\n -N [num] number of master clock readings per update (5)\n -L [limit] sanity frequency limit in ppb (200000000)\n -M [num] NTP SHM segment number (0)\n -u [num] number of clock updates in summary stats (0)\n -n [num] domain number (0)\n -x apply leap seconds by servo instead of kernel\n -z [path] server address for UDS (/var/run/ptp4l)\n -l [num] set the logging level to 'num' (6)\n -t [tag] add tag to log messages\n -m print messages to stdout\n -q do not print messages to the syslog\n -v prints the software version and exits\n -h prints this message and exits\n
"},{"location":"sw/tod/tod/#description","title":"DESCRIPTION","text":"The phc2sys utility is used to synchronize the system clock to the PTP Hardware Clock (PHC) or ToD clock. The phc_ctl utility is used to directly control PHC clock device.
"},{"location":"sw/tod/tod/#configuring-the-ptp-service","title":"Configuring the PTP service","text":" - Install the linuxptp package:
# sudo yum install linuxptp\n
- Check PTP device is created successfully by the ToD driver.
ToD driver registering as PHC device (clock_name: dfl_tod) to the Linux PTP stack and exposing to the Linux kernel to synchronize the system clock to its ToD information.
# cat /sys/class/ptp/ptp0/clock_name\ndfl_tod\n
- Configure phc2sys service on a system:
The phc2sys service is configured in the /etc/sysconfig/phc2sys configuration file. Define start-up option for phc2sys daemon in /etc/sysconfig/phc2sys. The master clock is /dev/ptp0 device and the slave clock is system clock/CLOCK_REALTIME:
OPTIONS=\"-s /dev/ptp0 -c CLOCK_REALTIME -r -O 0 -R 16\"\n
-
Start phc2sys service:
# service phc2sys start\n
-
Stop phc2sys service:
# service phc2sys stop\n
"},{"location":"sw/tod/tod/#examples","title":"Examples","text":""},{"location":"sw/tod/tod/#using-phc_ctl-utility","title":"using phc_ctl utility","text":"Read the current clock time from the PHC clock device:
# sudo phc_ctl /dev/ptp0 get\n
Set the PHC clock time to CLOCK_REALTIME:
# sudo phc_ctl /dev/ptp0 set\n
Set PHC clock time to 0:
# sudo phc_ctl /dev/ptp0 set 0.0\n
Set PHC clock time to 0 and wait for 10 sec and read the clock time:
# sudo phc_ctl /dev/ptp0 set 0.0 wait 10.0 get\n
Set and compare PHC clock time to CLOCK_REALTIME:
# sudo phc_ctl /dev/ptp0 set cmp\n
Read the PHC device capabilities:
# sudo phc_ctl /dev/ptp0 caps\n
"},{"location":"sw/tod/tod/#using-phc2sys-utility","title":"using phc2sys utility","text":"To synchronize the system clock to the PHC clock:
# sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME -r -O 0 -R 16 -m\nphc2sys[7896.789]: CLOCK_REALTIME phc offset -1259509 s0 freq -31462 delay 1338\nphc2sys[7896.852]: CLOCK_REALTIME phc offset -1261498 s1 freq -63144 delay 1328\nphc2sys[7896.914]: CLOCK_REALTIME phc offset -15 s2 freq -63159 delay 1328\nphc2sys[7896.977]: CLOCK_REALTIME phc offset -19 s2 freq -63167 delay 1327\nphc2sys[7897.039]: CLOCK_REALTIME phc offset -35 s2 freq -63189 delay 1328\nphc2sys[7897.102]: CLOCK_REALTIME phc offset -37 s2 freq -63201 delay 1331\nphc2sys[7897.165]: CLOCK_REALTIME phc offset -30 s2 freq -63205 delay 1328\nphc2sys[7897.227]: CLOCK_REALTIME phc offset -50 s2 freq -63234 delay 1331\nphc2sys[7897.290]: CLOCK_REALTIME phc offset -50 s2 freq -63249 delay 1329\nphc2sys[7897.353]: CLOCK_REALTIME phc offset -62 s2 freq -63276 delay 1334\nphc2sys[7897.415]: CLOCK_REALTIME phc offset -53 s2 freq -63286 delay 1335\nphc2sys[7897.478]: CLOCK_REALTIME phc offset -46 s2 freq -63295 delay 1325\nphc2sys[7897.541]: CLOCK_REALTIME phc offset -57 s2 freq -63320 delay 1334\nphc2sys[7897.603]: CLOCK_REALTIME phc offset -39 s2 freq -63319 delay 1327\nphc2sys[7897.666]: CLOCK_REALTIME phc offset -31 s2 freq -63323 delay 1328\nphc2sys[7897.728]: CLOCK_REALTIME phc offset -48 s2 freq -63349 delay 1327\nphc2sys[7897.791]: CLOCK_REALTIME phc offset -42 s2 freq -63357 delay 1323\nphc2sys[7897.854]: CLOCK_REALTIME phc offset -41 s2 freq -63369 delay 1324\nphc2sys[7897.916]: CLOCK_REALTIME phc offset -44 s2 freq -63384 delay 1328\nphc2sys[7897.979]: CLOCK_REALTIME phc offset -13 s2 freq -63366 delay 1327\nphc2sys[7898.042]: CLOCK_REALTIME phc offset -16 s2 freq -63373 delay 1327\nphc2sys[7898.104]: CLOCK_REALTIME phc offset -19 s2 freq -63381 delay 1328\nphc2sys[7898.167]: CLOCK_REALTIME phc offset -16 s2 freq -63384 delay 1327\nphc2sys[7898.229]: CLOCK_REALTIME phc offset 3 s2 freq -63370 delay 1327\nphc2sys[7898.292]: CLOCK_REALTIME phc offset 16 s2 freq -63356 delay 1325\nphc2sys[7898.355]: CLOCK_REALTIME phc offset 10 s2 freq -63357 delay 1319\nphc2sys[7898.417]: CLOCK_REALTIME phc offset 23 s2 freq -63341 delay 1327\nphc2sys[7898.480]: CLOCK_REALTIME phc offset 13 s2 freq -63344 delay 1335\nphc2sys[7898.543]: CLOCK_REALTIME phc offset 23 s2 freq -63330 delay 1323\nphc2sys[7898.605]: CLOCK_REALTIME phc offset 29 s2 freq -63317 delay 1312\nphc2sys[7898.668]: CLOCK_REALTIME phc offset 22 s2 freq -63315 delay 1324\nphc2sys[7898.730]: CLOCK_REALTIME phc offset 42 s2 freq -63289 delay 1325\nphc2sys[7898.793]: CLOCK_REALTIME phc offset 29 s2 freq -63289 delay 1333\nphc2sys[7898.856]: CLOCK_REALTIME phc offset 34 s2 freq -63276 delay 1327\nphc2sys[7898.918]: CLOCK_REALTIME phc offset 21 s2 freq -63278 delay 1331\nphc2sys[7898.981]: CLOCK_REALTIME phc offset 22 s2 freq -63271 delay 1335\nphc2sys[7899.044]: CLOCK_REALTIME phc offset 30 s2 freq -63256 delay 1327\nphc2sys[7899.106]: CLOCK_REALTIME phc offset 30 s2 freq -63247 delay 1328\nphc2sys[7899.169]: CLOCK_REALTIME phc offset 37 s2 freq -63231 delay 1333\nphc2sys[7899.232]: CLOCK_REALTIME phc offset 29 s2 freq -63228 delay 1331\nphc2sys[7899.294]: CLOCK_REALTIME phc offset 24 s2 freq -63225 delay 1330\n
"},{"location":"opae-code/annotated/","title":"Class List","text":"Here are the classes, structs, unions and interfaces with brief descriptions:
- struct _fpga_token_header Internal token type header.
- struct _opae_hash_map Hash map object.
- struct _opae_hash_map_item List link item.
- struct cache_line
- struct config
- struct fpga_error_info
- struct fpga_metric Metric struct.
- struct fpga_metric_info Metric info struct.
- struct fpga_version Semantic version.
- struct mem_alloc
- struct mem_link Provides an API for allocating/freeing a logical address space.
- struct metric_threshold
- union metric_value Metric value union.
- namespace opae
- namespace fpga
- namespace types
- class busy busy exception
- namespace detail
- class error An error object represents an error register for a resource.
- class event Wraps fpga event routines in OPAE C.
- struct type_t C++ struct that is interchangeable with fpga_event_type enum.
- class except Generic OPAE exception.
- class exception exception exception
- struct guid_t Representation of the guid member of a properties object.
- class handle An allocated accelerator resource.
- class invalid_param invalid_param exception
- class no_access no_access exception
- class no_daemon no_daemon exception
- class no_driver no_driver exception
- class no_memory no_memory exception
- class not_found not_found exception
- class not_supported not_supported exception
- class properties Wraps an OPAE fpga_properties object.
- struct pvalue Wraps OPAE properties defined in the OPAE C API by associating an
fpga_properties reference with the getters and setters defined for a property. - class reconf_error reconf_error exception
- class shared_buffer Host/AFU shared memory blocks.
- class src_location Identify a particular line in a source file.
- class sysobject Wraps the OPAE fpga_object primitive.
- class token Wraps the OPAE fpga_token primitive.
- class version
- struct opae_uio OPAE UIO device abstraction.
- struct opae_uio_device_region MMIO region info.
- struct opae_vfio OPAE VFIO device abstraction.
- struct opae_vfio_buffer System DMA buffer.
- struct opae_vfio_device VFIO device.
- struct opae_vfio_device_irq Interrupt info.
- struct opae_vfio_device_region MMIO region info.
- struct opae_vfio_group VFIO group.
- struct opae_vfio_iova_range IO Virtual Address Range.
- struct opae_vfio_sparse_info MMIO sparse-mappable region info.
- struct ras_inject_error
- namespace std
- struct threshold Threshold struct.
"},{"location":"opae-code/files/","title":"File List","text":"Here is a list of all files with brief descriptions:
- dir docs
- dir sw
- dir include
- dir opae
- file access.h Functions to acquire, release, and reset OPAE FPGA resources.
- file buffer.h Functions for allocating and sharing system memory with an FPGA accelerator.
- dir cxx
- file core.h
- dir core
- file errors.h
- file events.h
- file except.h
- file handle.h
- file properties.h
- file pvalue.h
- file shared_buffer.h
- file sysobject.h
- file token.h
- file version.h
- file enum.h APIs for resource enumeration and managing tokens.
- file error.h Functions for reading and clearing errors in resources.
- file event.h Functions for registering events and managing the lifecycle for
fpga_event_handle s. - file fpga.h FPGA API.
- file hash_map.h A general-purpose hybrid array/list hash map implementation.
- file init.h Initialization routine.
- file log.h
- file manage.h Functions for managing FPGA configurations.
- file mem_alloc.h
- file metrics.h Functions for Discover/ Enumerates metrics and retrieves values.
- file mmio.h Functions for mapping and accessing MMIO space.
- file properties.h Functions for examining and manipulating
fpga_properties objects. - file sysobject.h Functions to read/write from system objects.
- file types.h Type definitions for FPGA API.
- file types_enum.h Definitions of enumerated types for the OPAE C API.
- file uio.h APIs to manage a PCIe device via UIO.
- file umsg.h FPGA UMsg API.
- file userclk.h Functions for setting and get afu user clock.
- file utils.h Utility functions and macros for the FPGA API.
- file version.h
- file vfio.h APIs to manage a PCIe device via vfio-pci.
- dir samples
- dir hello_events
- file hello_events.c A code sample of using OPAE event API.
- dir hello_fpga
- file hello_fpga.c A code sample illustrates the basic usage of the OPAE C API.
"},{"location":"opae-code/struct__fpga__token__header/","title":"Struct _fpga_token_header","text":"ClassList > _fpga_token_header
Internal token type header. More...
#include <types.h>
"},{"location":"opae-code/struct__fpga__token__header/#public-attributes","title":"Public Attributes","text":"Type Name uint8_t bus uint8_t device uint16_t device_id uint8_t function fpga_guid guid fpga_interface interface uint64_t magic uint64_t object_id fpga_objtype objtype uint16_t segment uint16_t subsystem_device_id uint16_t subsystem_vendor_id uint16_t vendor_id"},{"location":"opae-code/struct__fpga__token__header/#detailed-description","title":"Detailed Description","text":"Each plugin (dfl: libxfpga.so, vfio: libopae-v.so) implements its own proprietary token type. This header must appear at offset zero within that structure.
eg, see lib/plugins/xfpga/types_int.h:struct _fpga_token and lib/plugins/vfio/opae_vfio.h:struct _vfio_token.
"},{"location":"opae-code/struct__fpga__token__header/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/struct__fpga__token__header/#variable-bus","title":"variable bus","text":"uint8_t _fpga_token_header::bus;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-device","title":"variable device","text":"uint8_t _fpga_token_header::device;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-device_id","title":"variable device_id","text":"uint16_t _fpga_token_header::device_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-function","title":"variable function","text":"uint8_t _fpga_token_header::function;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-guid","title":"variable guid","text":"fpga_guid _fpga_token_header::guid;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-interface","title":"variable interface","text":"fpga_interface _fpga_token_header::interface;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-magic","title":"variable magic","text":"uint64_t _fpga_token_header::magic;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-object_id","title":"variable object_id","text":"uint64_t _fpga_token_header::object_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-objtype","title":"variable objtype","text":"fpga_objtype _fpga_token_header::objtype;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-segment","title":"variable segment","text":"uint16_t _fpga_token_header::segment;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-subsystem_device_id","title":"variable subsystem_device_id","text":"uint16_t _fpga_token_header::subsystem_device_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-subsystem_vendor_id","title":"variable subsystem_vendor_id","text":"uint16_t _fpga_token_header::subsystem_vendor_id;\n
"},{"location":"opae-code/struct__fpga__token__header/#variable-vendor_id","title":"variable vendor_id","text":"uint16_t _fpga_token_header::vendor_id;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/struct__opae__hash__map/","title":"Struct _opae_hash_map","text":"ClassList > _opae_hash_map
Hash map object. More...
#include <hash_map.h>
"},{"location":"opae-code/struct__opae__hash__map/#public-attributes","title":"Public Attributes","text":"Type Name opae_hash_map_item ** buckets void * cleanup_context Optional second parameter to key_cleanup and value_cleanup. int flags uint32_t hash_seed void(* key_cleanup (optional) int(* key_compare (required) uint32_t(* key_hash uint32_t num_buckets void(* value_cleanup (optional)"},{"location":"opae-code/struct__opae__hash__map/#detailed-description","title":"Detailed Description","text":"This structure defines the internals of the hash map. Each of the parameters supplied to opae_hash_map_init() is stored in the structure. All parameters are required, except key_cleanup and value_cleanup, which may optionally be NULL.
"},{"location":"opae-code/struct__opae__hash__map/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/struct__opae__hash__map/#variable-buckets","title":"variable buckets","text":"opae_hash_map_item** _opae_hash_map::buckets;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-cleanup_context","title":"variable cleanup_context","text":"void* _opae_hash_map::cleanup_context;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-flags","title":"variable flags","text":"int _opae_hash_map::flags;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-hash_seed","title":"variable hash_seed","text":"uint32_t _opae_hash_map::hash_seed;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-key_cleanup","title":"variable key_cleanup","text":"void(* _opae_hash_map::key_cleanup) (void *key, void *context);\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-key_compare","title":"variable key_compare","text":"int(* _opae_hash_map::key_compare) (void *keya, void *keyb);\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-key_hash","title":"variable key_hash","text":"uint32_t(* _opae_hash_map::key_hash) (uint32_t num_buckets, uint32_t hash_seed, void *key);\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-num_buckets","title":"variable num_buckets","text":"uint32_t _opae_hash_map::num_buckets;\n
"},{"location":"opae-code/struct__opae__hash__map/#variable-value_cleanup","title":"variable value_cleanup","text":"void(* _opae_hash_map::value_cleanup) (void *value, void *context);\n
The documentation for this class was generated from the following file docs/sw/include/opae/hash_map.h
"},{"location":"opae-code/struct__opae__hash__map__item/","title":"Struct _opae_hash_map_item","text":"ClassList > _opae_hash_map_item
List link item. More...
#include <hash_map.h>
"},{"location":"opae-code/struct__opae__hash__map__item/#public-attributes","title":"Public Attributes","text":"Type Name void * key struct _opae_hash_map_item * next void * value"},{"location":"opae-code/struct__opae__hash__map__item/#detailed-description","title":"Detailed Description","text":"This structure provides the association between key and value. When the supplied hash function for keys A and B returns the same bucket index, both A and B can co-exist on the same list rooted at the bucket index.
"},{"location":"opae-code/struct__opae__hash__map__item/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/struct__opae__hash__map__item/#variable-key","title":"variable key","text":"void* _opae_hash_map_item::key;\n
"},{"location":"opae-code/struct__opae__hash__map__item/#variable-next","title":"variable next","text":"struct _opae_hash_map_item* _opae_hash_map_item::next;\n
"},{"location":"opae-code/struct__opae__hash__map__item/#variable-value","title":"variable value","text":"void* _opae_hash_map_item::value;\n
The documentation for this class was generated from the following file docs/sw/include/opae/hash_map.h
"},{"location":"opae-code/structcache__line/","title":"Struct cache_line","text":"ClassList > cache_line
"},{"location":"opae-code/structcache__line/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t uint"},{"location":"opae-code/structcache__line/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structcache__line/#variable-uint","title":"variable uint","text":"uint32_t cache_line::uint[16];\n
The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/hello_fpga.c
"},{"location":"opae-code/structconfig/","title":"Struct config","text":"ClassList > config
"},{"location":"opae-code/structconfig/#public-attributes","title":"Public Attributes","text":"Type Name int open_flags int run_n3000"},{"location":"opae-code/structconfig/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structconfig/#variable-open_flags","title":"variable open_flags","text":"int config::open_flags;\n
"},{"location":"opae-code/structconfig/#variable-run_n3000","title":"variable run_n3000","text":"int config::run_n3000;\n
The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/hello_fpga.c
"},{"location":"opae-code/structfpga__error__info/","title":"Struct fpga_error_info","text":"ClassList > fpga_error_info
#include <types.h>
"},{"location":"opae-code/structfpga__error__info/#public-attributes","title":"Public Attributes","text":"Type Name bool can_clear name of the error char name"},{"location":"opae-code/structfpga__error__info/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__error__info/#variable-can_clear","title":"variable can_clear","text":"bool fpga_error_info::can_clear;\n
"},{"location":"opae-code/structfpga__error__info/#variable-name","title":"variable name","text":"char fpga_error_info::name[FPGA_ERROR_NAME_MAX];\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structfpga__metric/","title":"Struct fpga_metric","text":"ClassList > fpga_metric
Metric struct.
#include <types.h>
"},{"location":"opae-code/structfpga__metric/#public-attributes","title":"Public Attributes","text":"Type Name bool isvalid uint64_t metric_num metric_value value"},{"location":"opae-code/structfpga__metric/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__metric/#variable-isvalid","title":"variable isvalid","text":"bool fpga_metric::isvalid;\n
"},{"location":"opae-code/structfpga__metric/#variable-metric_num","title":"variable metric_num","text":"uint64_t fpga_metric::metric_num;\n
"},{"location":"opae-code/structfpga__metric/#variable-value","title":"variable value","text":"metric_value fpga_metric::value;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structfpga__metric__info/","title":"Struct fpga_metric_info","text":"ClassList > fpga_metric_info
Metric info struct.
#include <types.h>
"},{"location":"opae-code/structfpga__metric__info/#public-attributes","title":"Public Attributes","text":"Type Name char group_name enum fpga_metric_datatype metric_datatype fpga_guid metric_guid char metric_name uint64_t metric_num enum fpga_metric_type metric_type char metric_units char qualifier_name"},{"location":"opae-code/structfpga__metric__info/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__metric__info/#variable-group_name","title":"variable group_name","text":"char fpga_metric_info::group_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_datatype","title":"variable metric_datatype","text":"enum fpga_metric_datatype fpga_metric_info::metric_datatype;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_guid","title":"variable metric_guid","text":"fpga_guid fpga_metric_info::metric_guid;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_name","title":"variable metric_name","text":"char fpga_metric_info::metric_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_num","title":"variable metric_num","text":"uint64_t fpga_metric_info::metric_num;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_type","title":"variable metric_type","text":"enum fpga_metric_type fpga_metric_info::metric_type;\n
"},{"location":"opae-code/structfpga__metric__info/#variable-metric_units","title":"variable metric_units","text":"char fpga_metric_info::metric_units[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structfpga__metric__info/#variable-qualifier_name","title":"variable qualifier_name","text":"char fpga_metric_info::qualifier_name[FPGA_METRIC_STR_SIZE];\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structfpga__version/","title":"Struct fpga_version","text":"ClassList > fpga_version
Semantic version. More...
#include <types.h>
"},{"location":"opae-code/structfpga__version/#public-attributes","title":"Public Attributes","text":"Type Name uint8_t major Major version. uint8_t minor Minor version. uint16_t patch Revision or patchlevel."},{"location":"opae-code/structfpga__version/#detailed-description","title":"Detailed Description","text":"Data structure for expressing version identifiers following the semantic versioning scheme. Used in various properties for tracking component versions.
"},{"location":"opae-code/structfpga__version/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structfpga__version/#variable-major","title":"variable major","text":"uint8_t fpga_version::major;\n
"},{"location":"opae-code/structfpga__version/#variable-minor","title":"variable minor","text":"uint8_t fpga_version::minor;\n
"},{"location":"opae-code/structfpga__version/#variable-patch","title":"variable patch","text":"uint16_t fpga_version::patch;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/structmem__alloc/","title":"Struct mem_alloc","text":"ClassList > mem_alloc
#include <mem_alloc.h>
"},{"location":"opae-code/structmem__alloc/#public-attributes","title":"Public Attributes","text":"Type Name struct mem_link allocated struct mem_link free"},{"location":"opae-code/structmem__alloc/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structmem__alloc/#variable-allocated","title":"variable allocated","text":"struct mem_link mem_alloc::allocated;\n
"},{"location":"opae-code/structmem__alloc/#variable-free","title":"variable free","text":"struct mem_link mem_alloc::free;\n
The documentation for this class was generated from the following file docs/sw/include/opae/mem_alloc.h
"},{"location":"opae-code/structmem__link/","title":"Struct mem_link","text":"ClassList > mem_link
Provides an API for allocating/freeing a logical address space. More...
#include <mem_alloc.h>
"},{"location":"opae-code/structmem__link/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t address struct mem_link * next struct mem_link * prev uint64_t size"},{"location":"opae-code/structmem__link/#detailed-description","title":"Detailed Description","text":"There is no interaction with any OS memory allocation infrastructure, whether malloc, mmap, etc. The \"address ranges\" tracked by this allocator are arbitrary 64-bit integers. The allocator simply provides the bookeeping logic that ensures that a unique address with the appropriate size is returned for each allocation request, and that an allocation can be freed, ie released back to the available pool of logical address space for future allocations. The memory backing the allocator's internal data structures is managed by malloc()/free().
"},{"location":"opae-code/structmem__link/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structmem__link/#variable-address","title":"variable address","text":"uint64_t mem_link::address;\n
"},{"location":"opae-code/structmem__link/#variable-next","title":"variable next","text":"struct mem_link* mem_link::next;\n
"},{"location":"opae-code/structmem__link/#variable-prev","title":"variable prev","text":"struct mem_link* mem_link::prev;\n
"},{"location":"opae-code/structmem__link/#variable-size","title":"variable size","text":"uint64_t mem_link::size;\n
The documentation for this class was generated from the following file docs/sw/include/opae/mem_alloc.h
"},{"location":"opae-code/structmetric__threshold/","title":"Struct metric_threshold","text":"ClassList > metric_threshold
#include <types.h>
"},{"location":"opae-code/structmetric__threshold/#public-attributes","title":"Public Attributes","text":"Type Name threshold hysteresis threshold lower_c_threshold threshold lower_nc_threshold threshold lower_nr_threshold char metric_name threshold upper_c_threshold threshold upper_nc_threshold threshold upper_nr_threshold"},{"location":"opae-code/structmetric__threshold/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structmetric__threshold/#variable-hysteresis","title":"variable hysteresis","text":"threshold metric_threshold::hysteresis;\n
"},{"location":"opae-code/structmetric__threshold/#variable-lower_c_threshold","title":"variable lower_c_threshold","text":"threshold metric_threshold::lower_c_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-lower_nc_threshold","title":"variable lower_nc_threshold","text":"threshold metric_threshold::lower_nc_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-lower_nr_threshold","title":"variable lower_nr_threshold","text":"threshold metric_threshold::lower_nr_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-metric_name","title":"variable metric_name","text":"char metric_threshold::metric_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structmetric__threshold/#variable-upper_c_threshold","title":"variable upper_c_threshold","text":"threshold metric_threshold::upper_c_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-upper_nc_threshold","title":"variable upper_nc_threshold","text":"threshold metric_threshold::upper_nc_threshold;\n
"},{"location":"opae-code/structmetric__threshold/#variable-upper_nr_threshold","title":"variable upper_nr_threshold","text":"threshold metric_threshold::upper_nr_threshold;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/unionmetric__value/","title":"Union metric_value","text":"ClassList > metric_value
Metric value union.
#include <types.h>
"},{"location":"opae-code/unionmetric__value/#public-attributes","title":"Public Attributes","text":"Type Name bool bvalue double dvalue float fvalue uint64_t ivalue"},{"location":"opae-code/unionmetric__value/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/unionmetric__value/#variable-bvalue","title":"variable bvalue","text":"bool metric_value::bvalue;\n
"},{"location":"opae-code/unionmetric__value/#variable-dvalue","title":"variable dvalue","text":"double metric_value::dvalue;\n
"},{"location":"opae-code/unionmetric__value/#variable-fvalue","title":"variable fvalue","text":"float metric_value::fvalue;\n
"},{"location":"opae-code/unionmetric__value/#variable-ivalue","title":"variable ivalue","text":"uint64_t metric_value::ivalue;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/namespaceopae/","title":"Namespace opae","text":"Namespace List > opae
"},{"location":"opae-code/namespaceopae/#namespaces","title":"Namespaces","text":"Type Name namespace fpga The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/namespaceopae_1_1fpga/","title":"Namespace opae::fpga","text":"Namespace List > opae > fpga
"},{"location":"opae-code/namespaceopae_1_1fpga/#namespaces","title":"Namespaces","text":"Type Name namespace types The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types/","title":"Namespace opae::fpga::types","text":"Namespace List > opae > fpga > types
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types/#namespaces","title":"Namespaces","text":"Type Name namespace detail"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types/#classes","title":"Classes","text":"Type Name class busy busy exception class error An error object represents an error register for a resource. class event Wraps fpga event routines in OPAE C. class except Generic OPAE exception. class exception exception exception struct guid_t Representation of the guid member of a properties object. class handle An allocated accelerator resource. class invalid_param invalid_param exception class no_access no_access exception class no_daemon no_daemon exception class no_driver no_driver exception class no_memory no_memory exception class not_found not_found exception class not_supported not_supported exception class properties Wraps an OPAE fpga_properties object. struct pvalue <typename T>Wraps OPAE properties defined in the OPAE C API by associating an fpga_properties reference with the getters and setters defined for a property. class reconf_error reconf_error exception class shared_buffer Host/AFU shared memory blocks. class src_location Identify a particular line in a source file. class sysobject Wraps the OPAE fpga_object primitive. class token Wraps the OPAE fpga_token primitive. class version The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/","title":"Class opae::fpga::types::busy","text":"ClassList > opae > fpga > types > busy
busy exception More...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-functions","title":"Public Functions","text":"Type Name busy (src_location loc) noexceptbusy constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#detailed-description","title":"Detailed Description","text":"busy tracks the source line of origin for exceptions thrown when the error code FPGA_BUSY is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1busy/#function-busy","title":"function busy","text":"busy constructor
inline opae::fpga::types::busy::busy (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/","title":"Namespace opae::fpga::types::detail","text":"Namespace List > opae > fpga > types > detail
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-types","title":"Public Types","text":"Type Name typedef bool(* exception_fn typedef function pointer that returns bool if result is FPGA_OK"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-attributes","title":"Public Static Attributes","text":"Type Name exception_fn opae_exceptions = = { is_ok<opae::fpga::types::invalid_param>, is_ok<opae::fpga::types::busy>, is_ok<opae::fpga::types::exception>, is_ok<opae::fpga::types::not_found>, is_ok<opae::fpga::types::no_memory>, is_ok<opae::fpga::types::not_supported>, is_ok<opae::fpga::types::no_driver>, is_ok<opae::fpga::types::no_daemon>, is_ok<opae::fpga::types::no_access>, is_ok<opae::fpga::types::reconf_error>}"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-functions","title":"Public Functions","text":"Type Name constexpr bool is_ok (fpga_result result, const opae::fpga::types::src_location & loc) is_ok is a template function that throws an excpetion of its template argument type if the result code is not FPGA_OK."},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-functions","title":"Public Static Functions","text":"Type Name void assert_fpga_ok (fpga_result result, const opae::fpga::types::src_location & loc)"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#typedef-exception_fn","title":"typedef exception_fn","text":"typedef bool(* opae::fpga::types::detail::exception_fn) (fpga_result, const opae::fpga::types::src_location &loc);\n
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#variable-opae_exceptions","title":"variable opae_exceptions","text":"exception_fn opae::fpga::types::detail::opae_exceptions[12];\n
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#function-is_ok","title":"function is_ok","text":"is_ok is a template function that throws an excpetion of its template argument type if the result code is not FPGA_OK.
template<typename T typename T>\nconstexpr bool opae::fpga::types::detail::is_ok (\nfpga_result result,\nconst opae::fpga::types::src_location & loc\n)
Otherwise it returns true.
"},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/namespaceopae_1_1fpga_1_1types_1_1detail/#function-assert_fpga_ok","title":"function assert_fpga_ok","text":"static inline void opae::fpga::types::detail::assert_fpga_ok (\nfpga_result result,\nconst opae::fpga::types::src_location & loc\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/","title":"Class opae::fpga::types::error","text":"ClassList > opae > fpga > types > error
An error object represents an error register for a resource. More...
#include <errors.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< error > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-functions","title":"Public Functions","text":"Type Name fpga_error_info c_type () constGet the C data structure. bool can_clear () Indicates whether an error register can be cleared. error (const error & e) = delete std::string name () Get the error register name. error & operator= (const error & e) = delete uint64_t read_value () Read the raw value contained in the associated error register. ~error ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-static-functions","title":"Public Static Functions","text":"Type Name error::ptr_t get (token::ptr_t tok, uint32_t num) Factory function for creating an error object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#detailed-description","title":"Detailed Description","text":"This is used to read out the raw value in the register. No parsing is done by this class.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<error> opae::fpga::types::error::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-c_type","title":"function c_type","text":"Get the C data structure.
inline fpga_error_info opae::fpga::types::error::c_type () const\n
Returns:
The fpga_error_info that contains the name and the can_clear boolean.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-can_clear","title":"function can_clear","text":"Indicates whether an error register can be cleared.
inline bool opae::fpga::types::error::can_clear ()
Returns:
A boolean value indicating if the error register can be cleared.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-error-12","title":"function error [\u00bd]","text":"opae::fpga::types::error::error (\nconst error & e\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-name","title":"function name","text":"Get the error register name.
inline std::string opae::fpga::types::error::name ()
Returns:
A std::string object set to the error name.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-operator","title":"function operator=","text":"error & opae::fpga::types::error::operator= (\nconst error & e\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-read_value","title":"function read_value","text":"Read the raw value contained in the associated error register.
uint64_t opae::fpga::types::error::read_value ()
Returns:
A 64-bit value (unparsed) read from the error register
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-error","title":"function ~error","text":"inline opae::fpga::types::error::~error ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1error/#function-get","title":"function get","text":"Factory function for creating an error object.
static error::ptr_t opae::fpga::types::error::get (\ntoken::ptr_t tok,\nuint32_t num\n)
Parameters:
tok The token object representing a resource. num The index of the error register. This must be lower than the num_errors property of the resource.
Returns:
A shared_ptr containing the error object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/","title":"Class opae::fpga::types::event","text":"ClassList > opae > fpga > types > event
Wraps fpga event routines in OPAE C.
#include <events.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#classes","title":"Classes","text":"Type Name struct type_t C++ struct that is interchangeable with fpga_event_type enum."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< event > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-functions","title":"Public Functions","text":"Type Name fpga_event_handle get () Get the fpga_event_handle contained in this object. operator fpga_event_handle () Coversion operator for converting to fpga_event_handle objects. int os_object () constGet OS Object from the event object. virtual ~event () Destroy event and associated resources."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-static-functions","title":"Public Static Functions","text":"Type Name event::ptr_t register_event (handle::ptr_t h, event::type_t t, int flags=0) Factory function to create event objects."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<event> opae::fpga::types::event::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-get","title":"function get","text":"Get the fpga_event_handle contained in this object.
inline fpga_event_handle opae::fpga::types::event::get ()
Returns:
The fpga_event_handle contained in this object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-operator-fpga_event_handle","title":"function operator fpga_event_handle","text":"Coversion operator for converting to fpga_event_handle objects.
opae::fpga::types::event::operator fpga_event_handle ()
Returns:
The fpga_event_handle contained in this object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-os_object","title":"function os_object","text":"Get OS Object from the event object.
int opae::fpga::types::event::os_object () const\n
Get an OS specific object from the event which can be used to subscribe for events. On Linux, the object corresponds to a file descriptor that can be used with select/poll/epoll calls.
Returns:
An integer object representing the OS object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-event","title":"function ~event","text":"virtual opae::fpga::types::event::~event ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1event/#function-register_event","title":"function register_event","text":"Factory function to create event objects.
static event::ptr_t opae::fpga::types::event::register_event (\nhandle::ptr_t h,\nevent::type_t t,\nint flags=0\n)
Parameters:
h A shared ptr of a resource handle t The resource type flags Event registration flags passed on to fpgaRegisterEvent
Returns:
A shared ptr to an event object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/events.h
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/","title":"Struct opae::fpga::types::event::type_t","text":"ClassList > opae > fpga > types > event > type_t
C++ struct that is interchangeable with fpga_event_type enum.
#include <events.h>
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-static-attributes","title":"Public Static Attributes","text":"Type Name constexpr fpga_event_type error = = FPGA_EVENT_ERROR constexpr fpga_event_type interrupt = = FPGA_EVENT_INTERRUPT constexpr fpga_event_type power_thermal = = FPGA_EVENT_POWER_THERMAL"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-functions","title":"Public Functions","text":"Type Name operator fpga_event_type () type_t (fpga_event_type c_type)"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#variable-error","title":"variable error","text":"constexpr fpga_event_type opae::fpga::types::event::type_t::error;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#variable-interrupt","title":"variable interrupt","text":"constexpr fpga_event_type opae::fpga::types::event::type_t::interrupt;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#variable-power_thermal","title":"variable power_thermal","text":"constexpr fpga_event_type opae::fpga::types::event::type_t::power_thermal;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#function-operator-fpga_event_type","title":"function operator fpga_event_type","text":"inline opae::fpga::types::event::type_t::operator fpga_event_type ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1event_1_1type__t/#function-type_t","title":"function type_t","text":"inline opae::fpga::types::event::type_t::type_t (\nfpga_event_type c_type\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/events.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/","title":"Class opae::fpga::types::except","text":"ClassList > opae > fpga > types > except
Generic OPAE exception. More...
#include <except.h>
Inherits the following classes: std::exception
Inherited by the following classes: opae::fpga::types::busy, opae::fpga::types::exception, opae::fpga::types::invalid_param, opae::fpga::types::no_access, opae::fpga::types::no_daemon, opae::fpga::types::no_driver, opae::fpga::types::no_memory, opae::fpga::types::not_found, opae::fpga::types::not_supported, opae::fpga::types::reconf_error
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-static-attributes","title":"Public Static Attributes","text":"Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-functions","title":"Public Functions","text":"Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#protected-attributes","title":"Protected Attributes","text":"Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#detailed-description","title":"Detailed Description","text":"An except tracks the source line of origin and an optional fpga_result. If no fpga_result is given, then FPGA_EXCEPTION is used.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-max_except","title":"variable MAX_EXCEPT","text":"const std::size_t opae::fpga::types::except::MAX_EXCEPT;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-except-13","title":"function except [\u2153]","text":"except constructor The fpga_result value is FPGA_EXCEPTION.
opae::fpga::types::except::except (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-except-23","title":"function except [\u2154]","text":"except constructor
opae::fpga::types::except::except (\nfpga_result res,\nsrc_location loc\n) noexcept\n
Parameters:
res The fpga_result value associated with this exception. loc Location where the exception was constructed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-except-33","title":"function except [3/3]","text":"except constructor
opae::fpga::types::except::except (\nfpga_result res,\nconst char * msg,\nsrc_location loc\n) noexcept\n
Parameters:
res The fpga_result value associated with this exception. msg The error message as a string loc Location where the exception was constructed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-operator-fpga_result","title":"function operator fpga_result","text":"inline opae::fpga::types::except::operator fpga_result () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#function-what","title":"function what","text":"virtual const char * opae::fpga::types::except::what () noexcept override const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#protected-attributes-documentation","title":"Protected Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-buf_","title":"variable buf_","text":"char opae::fpga::types::except::buf_[MAX_EXCEPT];\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-loc_","title":"variable loc_","text":"src_location opae::fpga::types::except::loc_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-msg_","title":"variable msg_","text":"const char* opae::fpga::types::except::msg_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1except/#variable-res_","title":"variable res_","text":"fpga_result opae::fpga::types::except::res_;\n
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/","title":"Class opae::fpga::types::exception","text":"ClassList > opae > fpga > types > exception
exception exception More...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-functions","title":"Public Functions","text":"Type Name exception (src_location loc) noexceptexception constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#detailed-description","title":"Detailed Description","text":"exception tracks the source line of origin for exceptions thrown when the error code FPGA_EXCEPTION is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1exception/#function-exception","title":"function exception","text":"exception constructor
inline opae::fpga::types::exception::exception (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/","title":"Struct opae::fpga::types::guid_t","text":"ClassList > opae > fpga > types > guid_t
Representation of the guid member of a properties object.
#include <pvalue.h>
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#public-functions","title":"Public Functions","text":"Type Name const uint8_t * c_type () constReturn a raw pointer to the guid. guid_t (fpga_properties * p) Construct the guid_t given its containing fpga_properties. void invalidate () Invalidate the cached local copy of the guid. bool is_set () constTracks whether the cached local copy of the guid is valid. operator uint8_t * () Return a raw pointer to the guid. guid_t & operator= (fpga_guid g) Assign from fpga_guid Sets the guid field of the associated properties object using the OPAE properties API. bool operator== (const fpga_guid & g) Compare contents with an fpga_guid. void parse (const char * str) Convert a string representation of a guid to binary. void update () Update the local cached copy of the guid."},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-c_type","title":"function c_type","text":"inline const uint8_t * opae::fpga::types::guid_t::c_type () const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-guid_t","title":"function guid_t","text":"inline opae::fpga::types::guid_t::guid_t (\nfpga_properties * p\n)
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-invalidate","title":"function invalidate","text":"inline void opae::fpga::types::guid_t::invalidate ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-is_set","title":"function is_set","text":"inline bool opae::fpga::types::guid_t::is_set () const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-operator-uint8_t","title":"function operator uint8_t *","text":"Return a raw pointer to the guid.
inline opae::fpga::types::guid_t::operator uint8_t * ()
Return value:
nullptr if the guid could not be queried.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-operator","title":"function operator=","text":"Assign from fpga_guid Sets the guid field of the associated properties object using the OPAE properties API.
inline guid_t & opae::fpga::types::guid_t::operator= (\nfpga_guid g\n)
Parameters:
g The given fpga_guid.
Returns:
a reference to this guid_t.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-operator_1","title":"function operator==","text":"Compare contents with an fpga_guid.
inline bool opae::fpga::types::guid_t::operator== (\nconst fpga_guid & g\n)
Return value:
The result of memcmp of the two objects.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-parse","title":"function parse","text":"Convert a string representation of a guid to binary.
inline void opae::fpga::types::guid_t::parse (\nconst char * str\n)
Parameters:
str The guid string.
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#function-update","title":"function update","text":"inline void opae::fpga::types::guid_t::update ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#friends-documentation","title":"Friends Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1guid__t/#friend-operator","title":"friend operator<<","text":"inline std::ostream & opae::fpga::types::guid_t::operator<< (\nstd::ostream & ostr,\nconst guid_t & g\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/pvalue.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/","title":"Class opae::fpga::types::handle","text":"ClassList > opae > fpga > types > handle
An allocated accelerator resource. More...
#include <handle.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< handle > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-functions","title":"Public Functions","text":"Type Name fpga_handle c_type () constRetrieve the underlying OPAE handle. fpga_result close () Close an accelerator resource (if opened) token::ptr_t get_token () constRetrieve the token corresponding to this handle object. handle (const handle &) = delete uint8_t * mmio_ptr (uint64_t offset, uint32_t csr_space=0) constRetrieve a pointer to the MMIO region. operator fpga_handle () constRetrieve the underlying OPAE handle. handle & operator= (const handle &) = delete uint32_t read_csr32 (uint64_t offset, uint32_t csr_space=0) constRead 32 bits from a CSR belonging to a resource associated with a handle. uint64_t read_csr64 (uint64_t offset, uint32_t csr_space=0) constRead 64 bits from a CSR belonging to a resource associated with a handle. void reconfigure (uint32_t slot, const uint8_t * bitstream, size_t size, int flags) Load a bitstream into an FPGA slot. virtual void reset () Reset the accelerator identified by this handle. void write_csr32 (uint64_t offset, uint32_t value, uint32_t csr_space=0) Write 32 bit to a CSR belonging to a resource associated with a handle. void write_csr512 (uint64_t offset, const void * value, uint32_t csr_space=0) Write 512 bits to a CSR belonging to a resource associated with a handle. void write_csr64 (uint64_t offset, uint64_t value, uint32_t csr_space=0) Write 64 bits to a CSR belonging to a resource associated with a handle. virtual ~handle ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-static-functions","title":"Public Static Functions","text":"Type Name handle::ptr_t open (fpga_token token, int flags) Open an accelerator resource, given a raw fpga_token. handle::ptr_t open (token::ptr_t token, int flags) Open an accelerator resource, given a token object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#detailed-description","title":"Detailed Description","text":"Represents an accelerator resource that has been allocated by OPAE. Depending on the type of resource, its register space may be read/written using a handle object.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<handle> opae::fpga::types::handle::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-c_type","title":"function c_type","text":"inline fpga_handle opae::fpga::types::handle::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-close","title":"function close","text":"Close an accelerator resource (if opened)
fpga_result opae::fpga::types::handle::close ()
Returns:
fpga_result indication the result of closing the handle or FPGA_EXCEPTION if handle is not opened
Note:
This is available for explicitly closing a handle. The destructor for handle will call close.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-get_token","title":"function get_token","text":"token::ptr_t opae::fpga::types::handle::get_token () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-handle-12","title":"function handle [\u00bd]","text":"opae::fpga::types::handle::handle (\nconst handle &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-mmio_ptr","title":"function mmio_ptr","text":"Retrieve a pointer to the MMIO region.
uint8_t * opae::fpga::types::handle::mmio_ptr (\nuint64_t offset,\nuint32_t csr_space=0\n) const\n
Parameters:
offset The byte offset to add to MMIO base. csr_space The desired CSR space. Default is 0.
Returns:
MMIO base + offset
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-operator-fpga_handle","title":"function operator fpga_handle","text":"inline opae::fpga::types::handle::operator fpga_handle () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-operator","title":"function operator=","text":"handle & opae::fpga::types::handle::operator= (\nconst handle &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-read_csr32","title":"function read_csr32","text":"Read 32 bits from a CSR belonging to a resource associated with a handle.
uint32_t opae::fpga::types::handle::read_csr32 (\nuint64_t offset,\nuint32_t csr_space=0\n) const\n
Parameters:
offset The register offset csr_space The CSR space to read from. Default is 0.
Returns:
The 32-bit value read from the CSR
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-read_csr64","title":"function read_csr64","text":"Read 64 bits from a CSR belonging to a resource associated with a handle.
uint64_t opae::fpga::types::handle::read_csr64 (\nuint64_t offset,\nuint32_t csr_space=0\n) const\n
Parameters:
offset The register offset csr_space The CSR space to read from. Default is 0.
Returns:
The 64-bit value read from the CSR
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-reconfigure","title":"function reconfigure","text":"Load a bitstream into an FPGA slot.
void opae::fpga::types::handle::reconfigure (\nuint32_t slot,\nconst uint8_t * bitstream,\nsize_t size,\nint flags\n)
Parameters:
slot The slot number to program bitstream The bitstream binary data size The size of the bitstream flags Flags that control behavior of reconfiguration. Value of 0 indicates no flags. FPGA_RECONF_FORCE indicates that the bitstream is programmed into the slot without checking if the resource is currently in use.
Exception:
- invalid_param if the handle is not an FPGA device handle or if the other parameters are not valid.
exception if an internal error is encountered. busy if the accelerator for the given slot is in use. - reconf_error if errors are reported by the driver (CRC or protocol errors).
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-reset","title":"function reset","text":"virtual void opae::fpga::types::handle::reset ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-write_csr32","title":"function write_csr32","text":"Write 32 bit to a CSR belonging to a resource associated with a handle.
void opae::fpga::types::handle::write_csr32 (\nuint64_t offset,\nuint32_t value,\nuint32_t csr_space=0\n)
Parameters:
offset The register offset. value The 32-bit value to write to the register. csr_space The CSR space to read from. Default is 0.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-write_csr512","title":"function write_csr512","text":"Write 512 bits to a CSR belonging to a resource associated with a handle.
void opae::fpga::types::handle::write_csr512 (\nuint64_t offset,\nconst void * value,\nuint32_t csr_space=0\n)
Parameters:
offset The register offset. value Pointer to the 512-bit value to write to the register. csr_space The CSR space to read from. Default is 0.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-write_csr64","title":"function write_csr64","text":"Write 64 bits to a CSR belonging to a resource associated with a handle.
void opae::fpga::types::handle::write_csr64 (\nuint64_t offset,\nuint64_t value,\nuint32_t csr_space=0\n)
Parameters:
offset The register offset. value The 64-bit value to write to the register. csr_space The CSR space to read from. Default is 0.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-handle","title":"function ~handle","text":"virtual opae::fpga::types::handle::~handle ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-open-12","title":"function open [\u00bd]","text":"Open an accelerator resource, given a raw fpga_token.
static handle::ptr_t opae::fpga::types::handle::open (\nfpga_token token,\nint flags\n)
Parameters:
token A token describing the accelerator resource to be allocated.flags The flags parameter to fpgaOpen().
Returns:
pointer to the mmio base + offset for the given csr space
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1handle/#function-open-22","title":"function open [2/2]","text":"Open an accelerator resource, given a token object.
static handle::ptr_t opae::fpga::types::handle::open (\ntoken::ptr_t token,\nint flags\n)
Parameters:
token A token object describing the accelerator resource to be allocated.flags The flags parameter to fpgaOpen().
Returns:
shared ptr to a handle object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/handle.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/","title":"Class opae::fpga::types::invalid_param","text":"ClassList > opae > fpga > types > invalid_param
invalid_param exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-functions","title":"Public Functions","text":"Type Name invalid_param (src_location loc) noexceptinvalid_param constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#detailed-description","title":"Detailed Description","text":"invalid_param tracks the source line of origin for exceptions thrown when the error code FPGA_INVALID_PARAM is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1invalid__param/#function-invalid_param","title":"function invalid_param","text":"invalid_param constructor
inline opae::fpga::types::invalid_param::invalid_param (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/","title":"Class opae::fpga::types::no_access","text":"ClassList > opae > fpga > types > no_access
no_access exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-functions","title":"Public Functions","text":"Type Name no_access (src_location loc) noexceptno_access constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#detailed-description","title":"Detailed Description","text":"no_access tracks the source line of origin for exceptions thrown when the error code FPGA_NO_ACCESS is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__access/#function-no_access","title":"function no_access","text":"no_access constructor
inline opae::fpga::types::no_access::no_access (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/","title":"Class opae::fpga::types::no_daemon","text":"ClassList > opae > fpga > types > no_daemon
no_daemon exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-functions","title":"Public Functions","text":"Type Name no_daemon (src_location loc) noexceptno_daemon constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#detailed-description","title":"Detailed Description","text":"no_daemon tracks the source line of origin for exceptions thrown when the error code FPGA_NO_DAEMON is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__daemon/#function-no_daemon","title":"function no_daemon","text":"no_daemon constructor
inline opae::fpga::types::no_daemon::no_daemon (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/","title":"Class opae::fpga::types::no_driver","text":"ClassList > opae > fpga > types > no_driver
no_driver exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-functions","title":"Public Functions","text":"Type Name no_driver (src_location loc) noexceptno_driver constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#detailed-description","title":"Detailed Description","text":"no_driver tracks the source line of origin for exceptions thrown when the error code FPGA_NO_DRIVER is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__driver/#function-no_driver","title":"function no_driver","text":"no_driver constructor
inline opae::fpga::types::no_driver::no_driver (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/","title":"Class opae::fpga::types::no_memory","text":"ClassList > opae > fpga > types > no_memory
no_memory exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-functions","title":"Public Functions","text":"Type Name no_memory (src_location loc) noexceptno_memory constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#detailed-description","title":"Detailed Description","text":"no_memory tracks the source line of origin for exceptions thrown when the error code FPGA_NO_MEMORY is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1no__memory/#function-no_memory","title":"function no_memory","text":"no_memory constructor
inline opae::fpga::types::no_memory::no_memory (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/","title":"Class opae::fpga::types::not_found","text":"ClassList > opae > fpga > types > not_found
not_found exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-functions","title":"Public Functions","text":"Type Name not_found (src_location loc) noexceptnot_found constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#detailed-description","title":"Detailed Description","text":"not_found tracks the source line of origin for exceptions thrown when the error code FPGA_NOT_FOUND is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__found/#function-not_found","title":"function not_found","text":"not_found constructor
inline opae::fpga::types::not_found::not_found (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/","title":"Class opae::fpga::types::not_supported","text":"ClassList > opae > fpga > types > not_supported
not_supported exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-functions","title":"Public Functions","text":"Type Name not_supported (src_location loc) noexceptnot_supported constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#detailed-description","title":"Detailed Description","text":"not_supported tracks the source line of origin for exceptions thrown when the error code FPGA_NOT_SUPPORTED is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1not__supported/#function-not_supported","title":"function not_supported","text":"not_supported constructor
inline opae::fpga::types::not_supported::not_supported (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/","title":"Class opae::fpga::types::properties","text":"ClassList > opae > fpga > types > properties
Wraps an OPAE fpga_properties object. More...
#include <properties.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< properties > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-attributes","title":"Public Attributes","text":"Type Name pvalue< fpga_accelerator_state > accelerator_state pvalue< uint64_t > bbs_id pvalue< fpga_version > bbs_version pvalue< uint8_t > bus pvalue< uint64_t > capabilities pvalue< uint8_t > device pvalue< uint16_t > device_id pvalue< uint8_t > function guid_t guid pvalue< fpga_interface > interface pvalue< uint64_t > local_memory_size pvalue< char * > model pvalue< uint32_t > num_errors pvalue< uint32_t > num_interrupts pvalue< uint32_t > num_mmio pvalue< uint32_t > num_slots pvalue< uint64_t > object_id pvalue< fpga_token > parent pvalue< uint16_t > segment pvalue< uint8_t > socket_id pvalue< uint16_t > subsystem_device_id pvalue< uint16_t > subsystem_vendor_id pvalue< fpga_objtype > type pvalue< uint16_t > vendor_id"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-attributes","title":"Public Static Attributes","text":"Type Name const std::vector< properties::ptr_t > none An empty vector of properties."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-functions","title":"Public Functions","text":"Type Name fpga_properties c_type () constGet the underlying fpga_properties object. operator fpga_properties () constGet the underlying fpga_properties object. properties & operator= (const properties & p) = delete properties (const properties & p) = delete ~properties ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-functions","title":"Public Static Functions","text":"Type Name properties::ptr_t get () Create a new properties object. properties::ptr_t get (fpga_guid guid_in) Create a new properties object from a guid. properties::ptr_t get (fpga_objtype objtype) Create a new properties object from an fpga_objtype. properties::ptr_t get (std::shared_ptr< token > t) Retrieve the properties for a given token object. properties::ptr_t get (fpga_token t) Retrieve the properties for a given fpga_token. properties::ptr_t get (std::shared_ptr< handle > h) Retrieve the properties for a given handle object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#detailed-description","title":"Detailed Description","text":"properties are information describing an accelerator resource that is identified by its token. The properties are used during enumeration to narrow the search for an accelerator resource, and after enumeration to provide the configuration of that resource.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<properties> opae::fpga::types::properties::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-accelerator_state","title":"variable accelerator_state","text":"pvalue<fpga_accelerator_state> opae::fpga::types::properties::accelerator_state;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-bbs_id","title":"variable bbs_id","text":"pvalue<uint64_t> opae::fpga::types::properties::bbs_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-bbs_version","title":"variable bbs_version","text":"pvalue<fpga_version> opae::fpga::types::properties::bbs_version;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-bus","title":"variable bus","text":"pvalue<uint8_t> opae::fpga::types::properties::bus;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-capabilities","title":"variable capabilities","text":"pvalue<uint64_t> opae::fpga::types::properties::capabilities;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-device","title":"variable device","text":"pvalue<uint8_t> opae::fpga::types::properties::device;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-device_id","title":"variable device_id","text":"pvalue<uint16_t> opae::fpga::types::properties::device_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-function","title":"variable function","text":"pvalue<uint8_t> opae::fpga::types::properties::function;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-guid","title":"variable guid","text":"guid_t opae::fpga::types::properties::guid;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-interface","title":"variable interface","text":"pvalue<fpga_interface> opae::fpga::types::properties::interface;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-local_memory_size","title":"variable local_memory_size","text":"pvalue<uint64_t> opae::fpga::types::properties::local_memory_size;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-model","title":"variable model","text":"pvalue<char *> opae::fpga::types::properties::model;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_errors","title":"variable num_errors","text":"pvalue<uint32_t> opae::fpga::types::properties::num_errors;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_interrupts","title":"variable num_interrupts","text":"pvalue<uint32_t> opae::fpga::types::properties::num_interrupts;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_mmio","title":"variable num_mmio","text":"pvalue<uint32_t> opae::fpga::types::properties::num_mmio;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-num_slots","title":"variable num_slots","text":"pvalue<uint32_t> opae::fpga::types::properties::num_slots;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-object_id","title":"variable object_id","text":"pvalue<uint64_t> opae::fpga::types::properties::object_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-parent","title":"variable parent","text":"pvalue<fpga_token> opae::fpga::types::properties::parent;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-segment","title":"variable segment","text":"pvalue<uint16_t> opae::fpga::types::properties::segment;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-socket_id","title":"variable socket_id","text":"pvalue<uint8_t> opae::fpga::types::properties::socket_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-subsystem_device_id","title":"variable subsystem_device_id","text":"pvalue<uint16_t> opae::fpga::types::properties::subsystem_device_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-subsystem_vendor_id","title":"variable subsystem_vendor_id","text":"pvalue<uint16_t> opae::fpga::types::properties::subsystem_vendor_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-type","title":"variable type","text":"pvalue<fpga_objtype> opae::fpga::types::properties::type;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-vendor_id","title":"variable vendor_id","text":"pvalue<uint16_t> opae::fpga::types::properties::vendor_id;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-attributes-documentation","title":"Public Static Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#variable-none","title":"variable none","text":"An empty vector of properties.
const std::vector<properties::ptr_t> opae::fpga::types::properties::none;\n
Useful for enumerating based on a \"match all\" criteria.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-c_type","title":"function c_type","text":"inline fpga_properties opae::fpga::types::properties::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-operator-fpga_properties","title":"function operator fpga_properties","text":"inline opae::fpga::types::properties::operator fpga_properties () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-operator","title":"function operator=","text":"properties & opae::fpga::types::properties::operator= (\nconst properties & p\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-properties-12","title":"function properties [\u00bd]","text":"opae::fpga::types::properties::properties (\nconst properties & p\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-properties","title":"function ~properties","text":"opae::fpga::types::properties::~properties ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-16","title":"function get [\u2159]","text":"Create a new properties object.
static properties::ptr_t opae::fpga::types::properties::get ()
Returns:
A properties smart pointer.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-26","title":"function get [2/6]","text":"Create a new properties object from a guid.
static properties::ptr_t opae::fpga::types::properties::get (\nfpga_guid guid_in\n)
Parameters:
guid_in A guid to set in the properties
Returns:
A properties smart pointer with its guid initialized to guid_in
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-36","title":"function get [3/6]","text":"Create a new properties object from an fpga_objtype.
static properties::ptr_t opae::fpga::types::properties::get (\nfpga_objtype objtype\n)
Parameters:
objtype An object type to set in the properties
Returns:
A properties smart pointer with its object type set to objtype.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-46","title":"function get [4/6]","text":"Retrieve the properties for a given token object.
static properties::ptr_t opae::fpga::types::properties::get (\nstd::shared_ptr< token > t\n)
Parameters:
t A token identifying the accelerator resource.
Returns:
A properties smart pointer for the given token.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-56","title":"function get [\u215a]","text":"Retrieve the properties for a given fpga_token.
static properties::ptr_t opae::fpga::types::properties::get (\nfpga_token t\n)
Parameters:
t An fpga_token identifying the accelerator resource.
Returns:
A properties smart pointer for the given fpga_token.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1properties/#function-get-66","title":"function get [6/6]","text":"Retrieve the properties for a given handle object.
static properties::ptr_t opae::fpga::types::properties::get (\nstd::shared_ptr< handle > h\n)
Parameters:
h A handle identifying the accelerator resource.
Returns:
A properties smart pointer for the given handle.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/properties.h
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/","title":"Struct opae::fpga::types::pvalue","text":"template <typename T typename T>
ClassList > opae > fpga > types > pvalue
Wraps OPAE properties defined in the OPAE C API by associating an fpga_properties reference with the getters and setters defined for a property.More...
#include <pvalue.h>
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-types","title":"Public Types","text":"Type Name typedef std::conditional< std::is_same< T, char * >::value, typename std::string, T >::type copy_t Define the type of our copy variable For char* types use std::string as the copy. typedef std::conditional< std::is_same< T, char * >::value, fpga_result(*)(fpga_properties, T), fpga_result(*)(fpga_properties, T *)>::type getter_t Define getter function as getter_t For char* types, do not use T* as the second argument but instead use T. typedef fpga_result(* setter_t Define the setter function as setter_t."},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-functions","title":"Public Functions","text":"Type Name fpga_result get_value (T & value) const void invalidate () Invalidate the cached local copy of the pvalue. bool is_set () constTracks whether the cached local copy of the pvalue is valid. operator copy_t () Implicit converter operator - calls the wrapped getter. pvalue< T > & operator= (const T & v) Overload of = operator that calls the wrapped setter. bool operator== (const T & other) Compare a property for equality with a value. pvalue () pvalue (fpga_properties * p, getter_t g, setter_t s) pvalue contructor that takes in a reference to fpga_properties and corresponding accessor methods for a property void update () void update () Template specialization of char* type property updater."},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#detailed-description","title":"Detailed Description","text":"Template parameters:
T The type of the property value being wrapped
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#typedef-copy_t","title":"typedef copy_t","text":"typedef std::conditional<std::is_same<T, char *>::value, typename std::string, T>::type opae::fpga::types::pvalue< T >::copy_t;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#typedef-getter_t","title":"typedef getter_t","text":"typedef std::conditional< std::is_same<T, char *>::value, fpga_result (*)(fpga_properties, T), fpga_result (*)(fpga_properties, T *)>::type opae::fpga::types::pvalue< T >::getter_t;\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#typedef-setter_t","title":"typedef setter_t","text":"typedef fpga_result(* opae::fpga::types::pvalue< T >::setter_t) (fpga_properties, T);\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-get_value","title":"function get_value","text":"inline fpga_result opae::fpga::types::pvalue::get_value (\nT & value\n) const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-invalidate","title":"function invalidate","text":"inline void opae::fpga::types::pvalue::invalidate ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-is_set","title":"function is_set","text":"inline bool opae::fpga::types::pvalue::is_set () const\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-operator-copy_t","title":"function operator copy_t","text":"Implicit converter operator - calls the wrapped getter.
inline opae::fpga::types::pvalue::operator copy_t ()
Returns:
The property value after calling the getter or a default value of the value type
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-operator","title":"function operator=","text":"Overload of = operator that calls the wrapped setter.
inline pvalue < T > & opae::fpga::types::pvalue::operator= (\nconst T & v\n)
Parameters:
v The value to set
Returns:
A reference to itself
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-operator_1","title":"function operator==","text":"Compare a property for equality with a value.
inline bool opae::fpga::types::pvalue::operator== (\nconst T & other\n)
Parameters:
other The value being compared to
Returns:
Whether or not the property is equal to the value
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-pvalue-12","title":"function pvalue [\u00bd]","text":"inline opae::fpga::types::pvalue::pvalue ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-pvalue-22","title":"function pvalue [2/2]","text":"pvalue contructor that takes in a reference to fpga_properties and corresponding accessor methods for a property
inline opae::fpga::types::pvalue::pvalue (\nfpga_properties * p,\ngetter_t g,\nsetter_t s\n)
Parameters:
p A reference to an fpga_properties g The getter function s The setter function
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-update-12","title":"function update [\u00bd]","text":"inline void opae::fpga::types::pvalue::update ()
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#function-update-22","title":"function update [2/2]","text":"Template specialization of char* type property updater.
inline void opae::fpga::types::pvalue::update ()
Returns:
The result of the property getter function.
## Friends Documentation\n
"},{"location":"opae-code/structopae_1_1fpga_1_1types_1_1pvalue/#friend-operator","title":"friend operator<<","text":"Stream overalod operator.
inline std::ostream & opae::fpga::types::pvalue::operator<< (\nstd::ostream & ostr,\nconst pvalue < T > & p\n)
Parameters:
ostr The output stream p A reference to a pvalue<T> object
Returns:
The stream operator after streaming the property value
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/pvalue.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/","title":"Class opae::fpga::types::reconf_error","text":"ClassList > opae > fpga > types > reconf_error
reconf_error exceptionMore...
#include <except.h>
Inherits the following classes: opae::fpga::types::except
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-static-attributes-inherited-from-opaefpgatypesexcept","title":"Public Static Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name const std::size_t MAX_EXCEPT = = 256"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-functions","title":"Public Functions","text":"Type Name reconf_error (src_location loc) noexceptreconf_error constructor"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-functions-inherited-from-opaefpgatypesexcept","title":"Public Functions inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name except (src_location loc) noexceptexcept constructor The fpga_result value is FPGA_EXCEPTION. except (fpga_result res, src_location loc) noexceptexcept constructor except (fpga_result res, const char * msg, src_location loc) noexceptexcept constructor operator fpga_result () noexcept constConvert this except to its fpga_result. virtual const char * what () noexcept override constConvert this except to an informative string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#protected-attributes-inherited-from-opaefpgatypesexcept","title":"Protected Attributes inherited from opae::fpga::types::except","text":"See opae::fpga::types::except
Type Name char buf_ src_location loc_ const char * msg_ fpga_result res_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#detailed-description","title":"Detailed Description","text":"reconf_error tracks the source line of origin for exceptions thrown when the error code FPGA_RECONF_ERROR is returned from a call to an OPAE C API function
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1reconf__error/#function-reconf_error","title":"function reconf_error","text":"reconf_error constructor
inline opae::fpga::types::reconf_error::reconf_error (\nsrc_location loc\n) noexcept\n
Parameters:
loc Location where the exception was constructed.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/","title":"Class opae::fpga::types::shared_buffer","text":"ClassList > opae > fpga > types > shared_buffer
Host/AFU shared memory blocks. More...
#include <shared_buffer.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< shared_buffer > ptr_t typedef std::size_t size_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-functions","title":"Public Functions","text":"Type Name uint8_t * c_type () constRetrieve the virtual address of the buffer base. int compare (ptr_t other, size_t len) constCompare this shared_buffer (the first len bytes) to that held in other, using memcmp(). void fill (int c) Write c to each byte location in the buffer. uint64_t io_address () constRetrieve the address of the buffer suitable for programming into the accelerator device. shared_buffer & operator= (const shared_buffer &) = delete handle::ptr_t owner () constRetrieve the handle smart pointer associated with this buffer. T read (size_t offset) constRead a T-sized block of memory at the given location. void release () Disassociate the shared_buffer object from the resource used to create it. shared_buffer (const shared_buffer &) = delete size_t size () constRetrieve the length of the buffer in bytes. void write (const T & value, size_t offset) Write a T-sized block of memory to the given location. uint64_t wsid () constRetrieve the underlying buffer's workspace id. virtual ~shared_buffer () shared_buffer destructor."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-static-functions","title":"Public Static Functions","text":"Type Name shared_buffer::ptr_t allocate (handle::ptr_t handle, size_t len, bool read_only=false) shared_buffer factory method - allocate ashared_buffer . shared_buffer::ptr_t attach (handle::ptr_t handle, uint8_t * base, size_t len, bool read_only=false) Attach a pre-allocated buffer to a shared_buffer object."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-attributes","title":"Protected Attributes","text":"Type Name handle::ptr_t handle_ uint64_t io_address_ size_t len_ uint8_t * virt_ uint64_t wsid_"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-functions","title":"Protected Functions","text":"Type Name shared_buffer (handle::ptr_t handle, size_t len, uint8_t * virt, uint64_t wsid, uint64_t io_address)"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#detailed-description","title":"Detailed Description","text":"shared_buffer abstracts a memory block that may be shared between the host cpu and an accelerator. The block may be allocated by the shared_buffer class itself (see allocate), or it may be allocated elsewhere and then attached to a shared_buffer object via attach.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<shared_buffer> opae::fpga::types::shared_buffer::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#typedef-size_t","title":"typedef size_t","text":"typedef std::size_t opae::fpga::types::shared_buffer::size_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-c_type","title":"function c_type","text":"Retrieve the virtual address of the buffer base.
inline uint8_t * opae::fpga::types::shared_buffer::c_type () const\n
Note:
Instances of a shared buffer can only be created using either 'allocate' or 'attach' static factory function. Because these functions return a shared pointer (std::shared_ptr) to the instance, references to an instance are counted automatically by design of the shared_ptr class. Calling 'c_type()' function is provided to get access to the raw data but isn't used in tracking its reference count. Assigning this to a variable should be done in limited scopes as this variable can be defined in an outer scope and may outlive the shared_buffer object. Once the reference count in the shared_ptr reaches zero, the shared_buffer object will be released and deallocated, turning any variables assigned from a call to 'c_type()' into dangling pointers.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-compare","title":"function compare","text":"int opae::fpga::types::shared_buffer::compare (\nptr_t other,\nsize_t len\n) const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-fill","title":"function fill","text":"void opae::fpga::types::shared_buffer::fill (\nint c\n)
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-io_address","title":"function io_address","text":"inline uint64_t opae::fpga::types::shared_buffer::io_address () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-operator","title":"function operator=","text":"shared_buffer & opae::fpga::types::shared_buffer::operator= (\nconst shared_buffer &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-owner","title":"function owner","text":"inline handle::ptr_t opae::fpga::types::shared_buffer::owner () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-read","title":"function read","text":"Read a T-sized block of memory at the given location.
template<typename T typename T>\ninline T opae::fpga::types::shared_buffer::read (\nsize_t offset\n) const\n
Parameters:
offset The byte offset from the start of the buffer.
Returns:
A T from buffer base + offset.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-release","title":"function release","text":"Disassociate the shared_buffer object from the resource used to create it.
void opae::fpga::types::shared_buffer::release ()
If the buffer was allocated using the allocate function then the buffer is freed.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-shared_buffer-12","title":"function shared_buffer [\u00bd]","text":"opae::fpga::types::shared_buffer::shared_buffer (\nconst shared_buffer &\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-size","title":"function size","text":"inline size_t opae::fpga::types::shared_buffer::size () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-write","title":"function write","text":"Write a T-sized block of memory to the given location.
template<typename T typename T>\ninline void opae::fpga::types::shared_buffer::write (\nconst T & value,\nsize_t offset\n)
Parameters:
value The value to write. offset The byte offset from the start of the buffer.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-wsid","title":"function wsid","text":"inline uint64_t opae::fpga::types::shared_buffer::wsid () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-shared_buffer","title":"function ~shared_buffer","text":"virtual opae::fpga::types::shared_buffer::~shared_buffer ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-allocate","title":"function allocate","text":"shared_buffer factory method - allocate ashared_buffer .
static shared_buffer::ptr_t opae::fpga::types::shared_buffer::allocate (\nhandle::ptr_t handle,\nsize_t len,\nbool read_only=false\n)
Parameters:
handle The handle used to allocate the buffer. len The length in bytes of the requested buffer.
Returns:
A valid shared_buffer smart pointer on success, or an empty smart pointer on failure.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-attach","title":"function attach","text":"Attach a pre-allocated buffer to a shared_buffer object.
static shared_buffer::ptr_t opae::fpga::types::shared_buffer::attach (\nhandle::ptr_t handle,\nuint8_t * base,\nsize_t len,\nbool read_only=false\n)
Parameters:
handle The handle used to attach the buffer. base The base of the pre-allocated memory. len The size of the pre-allocated memory, which must be a multiple of the page size.
Returns:
A valid shared_buffer smart pointer on success, or an empty smart pointer on failure.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-attributes-documentation","title":"Protected Attributes Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-handle_","title":"variable handle_","text":"handle::ptr_t opae::fpga::types::shared_buffer::handle_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-io_address_","title":"variable io_address_","text":"uint64_t opae::fpga::types::shared_buffer::io_address_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-len_","title":"variable len_","text":"size_t opae::fpga::types::shared_buffer::len_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-virt_","title":"variable virt_","text":"uint8_t* opae::fpga::types::shared_buffer::virt_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#variable-wsid_","title":"variable wsid_","text":"uint64_t opae::fpga::types::shared_buffer::wsid_;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#protected-functions-documentation","title":"Protected Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1shared__buffer/#function-shared_buffer-22","title":"function shared_buffer [2/2]","text":"opae::fpga::types::shared_buffer::shared_buffer (\nhandle::ptr_t handle,\nsize_t len,\nuint8_t * virt,\nuint64_t wsid,\nuint64_t io_address\n)
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/shared_buffer.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/","title":"Class opae::fpga::types::src_location","text":"ClassList > opae > fpga > types > src_location
Identify a particular line in a source file.
#include <except.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#public-functions","title":"Public Functions","text":"Type Name const char * file () noexcept constRetrieve the file name component of the location. const char * fn () noexcept constRetrieve the function name component of the location. int line () noexcept constRetrieve the line number component of the location. src_location & operator= (const src_location & other) noexcept src_location (const char * file, const char * fn, int line) noexceptsrc_location constructor src_location (const src_location & other) noexcept"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-file","title":"function file","text":"const char * opae::fpga::types::src_location::file () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-fn","title":"function fn","text":"inline const char * opae::fpga::types::src_location::fn () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-line","title":"function line","text":"inline int opae::fpga::types::src_location::line () noexcept const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-operator","title":"function operator=","text":"src_location & opae::fpga::types::src_location::operator= (\nconst src_location & other\n) noexcept\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-src_location-12","title":"function src_location [\u00bd]","text":"src_location constructor
opae::fpga::types::src_location::src_location (\nconst char * file,\nconst char * fn,\nint line\n) noexcept\n
Parameters:
file The source file name, typically FILE. fn The current function, typically func. line The current line number, typically LINE.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1src__location/#function-src_location-22","title":"function src_location [2/2]","text":"opae::fpga::types::src_location::src_location (\nconst src_location & other\n) noexcept\n
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/","title":"Class opae::fpga::types::sysobject","text":"ClassList > opae > fpga > types > sysobject
Wraps the OPAE fpga_object primitive. More...
#include <sysobject.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< sysobject > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-functions","title":"Public Functions","text":"Type Name std::vector< uint8_t > bytes (int flags=0) constGet all raw bytes from the object. std::vector< uint8_t > bytes (uint32_t offset, uint32_t size, int flags=0) constGet a subset of raw bytes from the object. fpga_object c_type () constRetrieve the underlying fpga_object primitive. sysobject::ptr_t get (const std::string & name, int flags=0) Get a sysobject from an object. sysobject::ptr_t get (int index) Get a sysobject from a container object. operator fpga_object () constRetrieve the underlying fpga_object primitive. sysobject & operator= (const sysobject & o) = delete uint64_t read64 (int flags=0) constRead a 64-bit value from an FPGA object. uint32_t size () constGet the size (in bytes) of the object. sysobject () = delete sysobject (const sysobject & o) = delete enum fpga_sysobject_type type () constGet the object type (attribute or container) void write64 (uint64_t value, int flags=0) constWrite 64-bit value to an FPGA object. virtual ~sysobject ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-static-functions","title":"Public Static Functions","text":"Type Name sysobject::ptr_t get (token::ptr_t t, const std::string & name, int flags=0) Get a sysobject from a token. sysobject::ptr_t get (handle::ptr_t h, const std::string & name, int flags=0) Get a sysobject from a handle."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#detailed-description","title":"Detailed Description","text":"sysobject's are created from a call to fpgaTokenGetObject, fpgaHandleGetObject, or fpgaObjectGetObject
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<sysobject> opae::fpga::types::sysobject::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-bytes-12","title":"function bytes [\u00bd]","text":"Get all raw bytes from the object.
std::vector< uint8_t > opae::fpga::types::sysobject::bytes (\nint flags=0\n) const\n
Parameters:
flags Flags that control how object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data.
Returns:
A vector of all bytes in the object.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-bytes-22","title":"function bytes [2/2]","text":"Get a subset of raw bytes from the object.
std::vector< uint8_t > opae::fpga::types::sysobject::bytes (\nuint32_t offset,\nuint32_t size,\nint flags=0\n) const\n
Parameters:
offset The bytes offset for the start of the returned vector. size The number of bytes for the returned vector. flags Flags that control how object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data.
Returns:
A vector of size bytes in the object starting at offset.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-c_type","title":"function c_type","text":"inline fpga_object opae::fpga::types::sysobject::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-14","title":"function get [\u00bc]","text":"Get a sysobject from an object.
sysobject::ptr_t opae::fpga::types::sysobject::get (\nconst std::string & name,\nint flags=0\n)
This will be read-write if its parent was created from a handle..
Parameters:
name An identifier representing an object belonging to this object. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects. Flags are defaulted to 0 meaning no flags.
Returns:
A shared_ptr to a sysobject instance.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-24","title":"function get [2/4]","text":"Get a sysobject from a container object.
sysobject::ptr_t opae::fpga::types::sysobject::get (\nint index\n)
This will be read-write if its parent was created from a handle..
Parameters:
index An index number to get.
Returns:
A shared_ptr to a sysobject instance.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-operator-fpga_object","title":"function operator fpga_object","text":"inline opae::fpga::types::sysobject::operator fpga_object () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-operator","title":"function operator=","text":"sysobject & opae::fpga::types::sysobject::operator= (\nconst sysobject & o\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-read64","title":"function read64","text":"Read a 64-bit value from an FPGA object.
uint64_t opae::fpga::types::sysobject::read64 (\nint flags=0\n) const\n
The value is assumed to be in string format and will be parsed. See flags below for changing that behavior.
Parameters:
flags Flags that control how the object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data. If FPGA_OBJECT_RAW is used, then the data will be read as raw bytes into the uint64_t pointer variable. Flags are defaulted to 0 meaning no flags.
Returns:
A 64-bit value from the object.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-size","title":"function size","text":"Get the size (in bytes) of the object.
uint32_t opae::fpga::types::sysobject::size () const\n
Returns:
The number of bytes that the object occupies in memory.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-sysobject-13","title":"function sysobject [\u2153]","text":"opae::fpga::types::sysobject::sysobject () = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-sysobject-23","title":"function sysobject [\u2154]","text":"opae::fpga::types::sysobject::sysobject (\nconst sysobject & o\n) = delete\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-type","title":"function type","text":"enum fpga_sysobject_type opae::fpga::types::sysobject::type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-write64","title":"function write64","text":"Write 64-bit value to an FPGA object.
void opae::fpga::types::sysobject::write64 (\nuint64_t value,\nint flags=0\n) const\n
The value will be converted to string before writing. See flags below for changing that behavior.
Parameters:
value The value to write to the object. flags Flags that control how the object is written If FPGA_OBJECT_RAW is used, then the value will be written as raw bytes. Flags are defaulted to 0 meaning no flags.
Note:
This operation will force a sync operation to update its cached buffer
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-sysobject","title":"function ~sysobject","text":"virtual opae::fpga::types::sysobject::~sysobject ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-34","title":"function get [\u00be]","text":"Get a sysobject from a token.
static sysobject::ptr_t opae::fpga::types::sysobject::get (\ntoken::ptr_t t,\nconst std::string & name,\nint flags=0\n)
This will be read-only.
Parameters:
t Token object representing a resource. name An identifier representing an object belonging to a resource represented by the token. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
A shared_ptr to a sysobject instance.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1sysobject/#function-get-44","title":"function get [4/4]","text":"Get a sysobject from a handle.
static sysobject::ptr_t opae::fpga::types::sysobject::get (\nhandle::ptr_t h,\nconst std::string & name,\nint flags=0\n)
This will be read-write.
Parameters:
h Handle object representing an open resource. name An identifier representing an object belonging to a resource represented by the handle. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
A shared_ptr to a sysobject instance.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/sysobject.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/","title":"Class opae::fpga::types::token","text":"ClassList > opae > fpga > types > token
Wraps the OPAE fpga_token primitive. More...
#include <token.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-types","title":"Public Types","text":"Type Name typedef std::shared_ptr< token > ptr_t"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-functions","title":"Public Functions","text":"Type Name fpga_token c_type () constRetrieve the underlying fpga_token primitive. ptr_t get_parent () constRetrieve the parent token, or an empty pointer if there is none. operator fpga_token () constRetrieve the underlying fpga_token primitive. ~token ()"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-static-functions","title":"Public Static Functions","text":"Type Name std::vector< token::ptr_t > enumerate (const std::vector< properties::ptr_t > & props) Obtain a vector of token smart pointers for given search criteria."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#detailed-description","title":"Detailed Description","text":"token's are created from an enumeration operation that uses properties describing an accelerator resource as search criteria.
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#typedef-ptr_t","title":"typedef ptr_t","text":"typedef std::shared_ptr<token> opae::fpga::types::token::ptr_t;\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-c_type","title":"function c_type","text":"inline fpga_token opae::fpga::types::token::c_type () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-get_parent","title":"function get_parent","text":"ptr_t opae::fpga::types::token::get_parent () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-operator-fpga_token","title":"function operator fpga_token","text":"inline opae::fpga::types::token::operator fpga_token () const\n
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-token","title":"function ~token","text":"opae::fpga::types::token::~token ()
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1token/#function-enumerate","title":"function enumerate","text":"Obtain a vector of token smart pointers for given search criteria.
static std::vector< token::ptr_t > opae::fpga::types::token::enumerate (\nconst std::vector< properties::ptr_t > & props\n)
Parameters:
props The search criteria.
Returns:
A set of known tokens that match the search.
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/token.h
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/","title":"Class opae::fpga::types::version","text":"ClassList > opae > fpga > types > version
#include <version.h>
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#public-static-functions","title":"Public Static Functions","text":"Type Name std::string as_string () Get the package version information as a string. fpga_version as_struct () Get the package version information as a struct. std::string build () Get the package build information as a string."},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#public-static-functions-documentation","title":"Public Static Functions Documentation","text":""},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#function-as_string","title":"function as_string","text":"Get the package version information as a string.
static std::string opae::fpga::types::version::as_string ()
Returns:
The package version as an std::string object
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#function-as_struct","title":"function as_struct","text":"Get the package version information as a struct.
static fpga_version opae::fpga::types::version::as_struct ()
Returns:
The package version as an fpga_version struct
"},{"location":"opae-code/classopae_1_1fpga_1_1types_1_1version/#function-build","title":"function build","text":"Get the package build information as a string.
static std::string opae::fpga::types::version::build ()
Returns:
The package build as an std::string object
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/version.h
"},{"location":"opae-code/structopae__uio/","title":"Struct opae_uio","text":"ClassList > opae_uio
OPAE UIO device abstraction. More...
#include <uio.h>
"},{"location":"opae-code/structopae__uio/#public-attributes","title":"Public Attributes","text":"Type Name int device_fd char device_path struct opae_uio_device_region * regions"},{"location":"opae-code/structopae__uio/#detailed-description","title":"Detailed Description","text":"This structure is used to interact with the OPAE UIO API. Each UIO device has a file descriptor that is used to mmap its regions into user address space. Each device also has one or more MMIO regions.
"},{"location":"opae-code/structopae__uio/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__uio/#variable-device_fd","title":"variable device_fd","text":"int opae_uio::device_fd;\n
"},{"location":"opae-code/structopae__uio/#variable-device_path","title":"variable device_path","text":"char opae_uio::device_path[OPAE_UIO_PATH_MAX];\n
"},{"location":"opae-code/structopae__uio/#variable-regions","title":"variable regions","text":"struct opae_uio_device_region* opae_uio::regions;\n
The documentation for this class was generated from the following file docs/sw/include/opae/uio.h
"},{"location":"opae-code/structopae__uio__device__region/","title":"Struct opae_uio_device_region","text":"ClassList > opae_uio_device_region
MMIO region info. More...
#include <uio.h>
"},{"location":"opae-code/structopae__uio__device__region/#public-attributes","title":"Public Attributes","text":"Type Name struct opae_uio_device_region * next uint32_t region_index size_t region_page_offset uint8_t * region_ptr size_t region_size"},{"location":"opae-code/structopae__uio__device__region/#detailed-description","title":"Detailed Description","text":"Describes a range of mappable MMIO.
"},{"location":"opae-code/structopae__uio__device__region/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__uio__device__region/#variable-next","title":"variable next","text":"struct opae_uio_device_region* opae_uio_device_region::next;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_index","title":"variable region_index","text":"uint32_t opae_uio_device_region::region_index;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_page_offset","title":"variable region_page_offset","text":"size_t opae_uio_device_region::region_page_offset;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_ptr","title":"variable region_ptr","text":"uint8_t* opae_uio_device_region::region_ptr;\n
"},{"location":"opae-code/structopae__uio__device__region/#variable-region_size","title":"variable region_size","text":"size_t opae_uio_device_region::region_size;\n
The documentation for this class was generated from the following file docs/sw/include/opae/uio.h
"},{"location":"opae-code/structopae__vfio/","title":"Struct opae_vfio","text":"ClassList > opae_vfio
OPAE VFIO device abstraction. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio/#public-attributes","title":"Public Attributes","text":"Type Name opae_hash_map cont_buffers Map of allocated DMA buffers. char * cont_device \"/dev/vfio/vfio\" int cont_fd Container file descriptor. char * cont_pciaddr PCIe address, eg 0000:00:00.0. struct opae_vfio_iova_range * cont_ranges List of IOVA ranges. struct opae_vfio_device device The VFIO device. struct opae_vfio_group group The VFIO device group. struct mem_alloc iova_alloc Allocator for IOVA space. pthread_mutex_t lock For thread safety."},{"location":"opae-code/structopae__vfio/#detailed-description","title":"Detailed Description","text":"This structure is used to interact with the OPAE VFIO API. It tracks data related to the VFIO container, group, and device. A mutex is provided for thread safety.
"},{"location":"opae-code/structopae__vfio/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio/#variable-cont_buffers","title":"variable cont_buffers","text":"opae_hash_map opae_vfio::cont_buffers;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_device","title":"variable cont_device","text":"char* opae_vfio::cont_device;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_fd","title":"variable cont_fd","text":"int opae_vfio::cont_fd;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_pciaddr","title":"variable cont_pciaddr","text":"char* opae_vfio::cont_pciaddr;\n
"},{"location":"opae-code/structopae__vfio/#variable-cont_ranges","title":"variable cont_ranges","text":"struct opae_vfio_iova_range* opae_vfio::cont_ranges;\n
"},{"location":"opae-code/structopae__vfio/#variable-device","title":"variable device","text":"struct opae_vfio_device opae_vfio::device;\n
"},{"location":"opae-code/structopae__vfio/#variable-group","title":"variable group","text":"struct opae_vfio_group opae_vfio::group;\n
"},{"location":"opae-code/structopae__vfio/#variable-iova_alloc","title":"variable iova_alloc","text":"struct mem_alloc opae_vfio::iova_alloc;\n
"},{"location":"opae-code/structopae__vfio/#variable-lock","title":"variable lock","text":"pthread_mutex_t opae_vfio::lock;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__buffer/","title":"Struct opae_vfio_buffer","text":"ClassList > opae_vfio_buffer
System DMA buffer. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__buffer/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t buffer_iova Buffer IOVA address. uint8_t * buffer_ptr Buffer virtual address. size_t buffer_size Buffer size. int flags See opae_vfio_buffer_flags."},{"location":"opae-code/structopae__vfio__buffer/#detailed-description","title":"Detailed Description","text":"Describes a system memory address space that is capable of DMA.
"},{"location":"opae-code/structopae__vfio__buffer/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__buffer/#variable-buffer_iova","title":"variable buffer_iova","text":"uint64_t opae_vfio_buffer::buffer_iova;\n
"},{"location":"opae-code/structopae__vfio__buffer/#variable-buffer_ptr","title":"variable buffer_ptr","text":"uint8_t* opae_vfio_buffer::buffer_ptr;\n
"},{"location":"opae-code/structopae__vfio__buffer/#variable-buffer_size","title":"variable buffer_size","text":"size_t opae_vfio_buffer::buffer_size;\n
"},{"location":"opae-code/structopae__vfio__buffer/#variable-flags","title":"variable flags","text":"int opae_vfio_buffer::flags;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__device/","title":"Struct opae_vfio_device","text":"ClassList > opae_vfio_device
VFIO device. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__device/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t device_config_offset Offset of PCIe config space. int device_fd Device file descriptor. uint32_t device_num_irqs IRQ index count. uint32_t device_num_regions Total MMIO region count. struct opae_vfio_device_irq * irqs IRQ list pointer. struct opae_vfio_device_region * regions Region list pointer."},{"location":"opae-code/structopae__vfio__device/#detailed-description","title":"Detailed Description","text":"Each VFIO device has a file descriptor that is used to query information about the device MMIO regions and config space.
"},{"location":"opae-code/structopae__vfio__device/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__device/#variable-device_config_offset","title":"variable device_config_offset","text":"uint64_t opae_vfio_device::device_config_offset;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-device_fd","title":"variable device_fd","text":"int opae_vfio_device::device_fd;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-device_num_irqs","title":"variable device_num_irqs","text":"uint32_t opae_vfio_device::device_num_irqs;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-device_num_regions","title":"variable device_num_regions","text":"uint32_t opae_vfio_device::device_num_regions;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-irqs","title":"variable irqs","text":"struct opae_vfio_device_irq* opae_vfio_device::irqs;\n
"},{"location":"opae-code/structopae__vfio__device/#variable-regions","title":"variable regions","text":"struct opae_vfio_device_region* opae_vfio_device::regions;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__device__irq/","title":"Struct opae_vfio_device_irq","text":"ClassList > opae_vfio_device_irq
Interrupt info. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__device__irq/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t count Number of IRQs at this index. int32_t * event_fds Event file descriptors. uint32_t flags Flags. uint32_t index The IRQ index. int32_t * masks IRQ masks. struct opae_vfio_device_irq * next Pointer to next in list."},{"location":"opae-code/structopae__vfio__device__irq/#detailed-description","title":"Detailed Description","text":"Describes an interrupt capability.
"},{"location":"opae-code/structopae__vfio__device__irq/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__device__irq/#variable-count","title":"variable count","text":"uint32_t opae_vfio_device_irq::count;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-event_fds","title":"variable event_fds","text":"int32_t* opae_vfio_device_irq::event_fds;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-flags","title":"variable flags","text":"Flags.
uint32_t opae_vfio_device_irq::flags;\n
See struct vfio_irq_info.
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-index","title":"variable index","text":"uint32_t opae_vfio_device_irq::index;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-masks","title":"variable masks","text":"int32_t* opae_vfio_device_irq::masks;\n
"},{"location":"opae-code/structopae__vfio__device__irq/#variable-next","title":"variable next","text":"struct opae_vfio_device_irq* opae_vfio_device_irq::next;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__device__region/","title":"Struct opae_vfio_device_region","text":"ClassList > opae_vfio_device_region
MMIO region info. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__device__region/#public-attributes","title":"Public Attributes","text":"Type Name struct opae_vfio_device_region * next Pointer to next in list. uint32_t region_index Region index, from 0. uint8_t * region_ptr Virtual address of region. size_t region_size Size of region. struct opae_vfio_sparse_info * region_sparse For sparse regions."},{"location":"opae-code/structopae__vfio__device__region/#detailed-description","title":"Detailed Description","text":"Describes a range of mappable MMIO.
"},{"location":"opae-code/structopae__vfio__device__region/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__device__region/#variable-next","title":"variable next","text":"struct opae_vfio_device_region* opae_vfio_device_region::next;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_index","title":"variable region_index","text":"uint32_t opae_vfio_device_region::region_index;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_ptr","title":"variable region_ptr","text":"uint8_t* opae_vfio_device_region::region_ptr;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_size","title":"variable region_size","text":"size_t opae_vfio_device_region::region_size;\n
"},{"location":"opae-code/structopae__vfio__device__region/#variable-region_sparse","title":"variable region_sparse","text":"struct opae_vfio_sparse_info* opae_vfio_device_region::region_sparse;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__group/","title":"Struct opae_vfio_group","text":"ClassList > opae_vfio_group
VFIO group. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__group/#public-attributes","title":"Public Attributes","text":"Type Name char * group_device Full path to the group device. int group_fd File descriptor for the group device."},{"location":"opae-code/structopae__vfio__group/#detailed-description","title":"Detailed Description","text":"Each device managed by vfio-pci belongs to a VFIO group rooted at /dev/vfio/N, where N is the group number.
"},{"location":"opae-code/structopae__vfio__group/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__group/#variable-group_device","title":"variable group_device","text":"char* opae_vfio_group::group_device;\n
"},{"location":"opae-code/structopae__vfio__group/#variable-group_fd","title":"variable group_fd","text":"int opae_vfio_group::group_fd;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__iova__range/","title":"Struct opae_vfio_iova_range","text":"ClassList > opae_vfio_iova_range
IO Virtual Address Range. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__iova__range/#public-attributes","title":"Public Attributes","text":"Type Name uint64_t end End of this range of offsets. struct opae_vfio_iova_range * next Pointer to next in list. uint64_t start Start of this range of offsets."},{"location":"opae-code/structopae__vfio__iova__range/#detailed-description","title":"Detailed Description","text":"A range of allocatable IOVA offsets. Used for mapping DMA buffers.
"},{"location":"opae-code/structopae__vfio__iova__range/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__iova__range/#variable-end","title":"variable end","text":"uint64_t opae_vfio_iova_range::end;\n
"},{"location":"opae-code/structopae__vfio__iova__range/#variable-next","title":"variable next","text":"struct opae_vfio_iova_range* opae_vfio_iova_range::next;\n
"},{"location":"opae-code/structopae__vfio__iova__range/#variable-start","title":"variable start","text":"uint64_t opae_vfio_iova_range::start;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structopae__vfio__sparse__info/","title":"Struct opae_vfio_sparse_info","text":"ClassList > opae_vfio_sparse_info
MMIO sparse-mappable region info. More...
#include <vfio.h>
"},{"location":"opae-code/structopae__vfio__sparse__info/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t index Region index, from 0. struct opae_vfio_sparse_info * next Pointer to next in list. uint32_t offset Offset of sparse region. uint8_t * ptr Virtual address of sparse region. uint32_t size Size of sparse region."},{"location":"opae-code/structopae__vfio__sparse__info/#detailed-description","title":"Detailed Description","text":"Describes a range of sparse-mappable MMIO, for MMIO ranges that have non-contiguous addresses.
"},{"location":"opae-code/structopae__vfio__sparse__info/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structopae__vfio__sparse__info/#variable-index","title":"variable index","text":"uint32_t opae_vfio_sparse_info::index;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-next","title":"variable next","text":"struct opae_vfio_sparse_info* opae_vfio_sparse_info::next;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-offset","title":"variable offset","text":"uint32_t opae_vfio_sparse_info::offset;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-ptr","title":"variable ptr","text":"uint8_t* opae_vfio_sparse_info::ptr;\n
"},{"location":"opae-code/structopae__vfio__sparse__info/#variable-size","title":"variable size","text":"uint32_t opae_vfio_sparse_info::size;\n
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/structras__inject__error/","title":"Struct ras_inject_error","text":"ClassList > ras_inject_error
"},{"location":"opae-code/structras__inject__error/#public-attributes","title":"Public Attributes","text":"Type Name union ras_inject_error::@0 @1 uint64_t catastrophicr_error uint64_t csr uint64_t fatal_error uint64_t nonfatal_error uint64_t rsvd"},{"location":"opae-code/structras__inject__error/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structras__inject__error/#variable-1","title":"variable @1","text":"union ras_inject_error::@0 ras_inject_error::@1;\n
"},{"location":"opae-code/structras__inject__error/#variable-catastrophicr_error","title":"variable catastrophicr_error","text":"uint64_t ras_inject_error::catastrophicr_error;\n
"},{"location":"opae-code/structras__inject__error/#variable-csr","title":"variable csr","text":"uint64_t ras_inject_error::csr;\n
"},{"location":"opae-code/structras__inject__error/#variable-fatal_error","title":"variable fatal_error","text":"uint64_t ras_inject_error::fatal_error;\n
"},{"location":"opae-code/structras__inject__error/#variable-nonfatal_error","title":"variable nonfatal_error","text":"uint64_t ras_inject_error::nonfatal_error;\n
"},{"location":"opae-code/structras__inject__error/#variable-rsvd","title":"variable rsvd","text":"uint64_t ras_inject_error::rsvd;\n
The documentation for this class was generated from the following file docs/sw/samples/hello_events/hello_events.c
"},{"location":"opae-code/namespacestd/","title":"Namespace std","text":"Namespace List > std
The documentation for this class was generated from the following file [generated]
"},{"location":"opae-code/structthreshold/","title":"Struct threshold","text":"ClassList > threshold
Threshold struct.
#include <types.h>
"},{"location":"opae-code/structthreshold/#public-attributes","title":"Public Attributes","text":"Type Name uint32_t is_valid char threshold_name double value"},{"location":"opae-code/structthreshold/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/structthreshold/#variable-is_valid","title":"variable is_valid","text":"uint32_t threshold::is_valid;\n
"},{"location":"opae-code/structthreshold/#variable-threshold_name","title":"variable threshold_name","text":"char threshold::threshold_name[FPGA_METRIC_STR_SIZE];\n
"},{"location":"opae-code/structthreshold/#variable-value","title":"variable value","text":"double threshold::value;\n
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/dir_49e56c817e5e54854c35e136979f97ca/","title":"Dir docs","text":"FileList > docs
"},{"location":"opae-code/dir_49e56c817e5e54854c35e136979f97ca/#directories","title":"Directories","text":"Type Name dir sw The documentation for this class was generated from the following file docs/
"},{"location":"opae-code/dir_55721a669a8e0900d975c02921addb49/","title":"Dir docs/sw","text":"FileList > docs > sw
"},{"location":"opae-code/dir_55721a669a8e0900d975c02921addb49/#directories","title":"Directories","text":"Type Name dir include dir samples The documentation for this class was generated from the following file docs/sw/
"},{"location":"opae-code/dir_97b4588afba69bf89bbe554642ac6431/","title":"Dir docs/sw/include","text":"FileList > docs > sw > include
"},{"location":"opae-code/dir_97b4588afba69bf89bbe554642ac6431/#directories","title":"Directories","text":"Type Name dir opae The documentation for this class was generated from the following file docs/sw/include/
"},{"location":"opae-code/dir_ade97cd9199f278c0723672dd8647ba4/","title":"Dir docs/sw/include/opae","text":"FileList > docs > sw > include > opae
"},{"location":"opae-code/dir_ade97cd9199f278c0723672dd8647ba4/#files","title":"Files","text":"Type Name file access.h Functions to acquire, release, and reset OPAE FPGA resources. file buffer.h Functions for allocating and sharing system memory with an FPGA accelerator. file enum.h APIs for resource enumeration and managing tokens. file error.h Functions for reading and clearing errors in resources. file event.h Functions for registering events and managing the lifecycle for fpga_event_handle s. file fpga.h FPGA API. file hash_map.h A general-purpose hybrid array/list hash map implementation. file init.h Initialization routine. file log.h file manage.h Functions for managing FPGA configurations. file mem_alloc.h file metrics.h Functions for Discover/ Enumerates metrics and retrieves values. file mmio.h Functions for mapping and accessing MMIO space. file properties.h Functions for examining and manipulating fpga_properties objects. file sysobject.h Functions to read/write from system objects. file types.h Type definitions for FPGA API. file types_enum.h Definitions of enumerated types for the OPAE C API. file uio.h APIs to manage a PCIe device via UIO. file umsg.h FPGA UMsg API. file userclk.h Functions for setting and get afu user clock. file utils.h Utility functions and macros for the FPGA API. file version.h file vfio.h APIs to manage a PCIe device via vfio-pci."},{"location":"opae-code/dir_ade97cd9199f278c0723672dd8647ba4/#directories","title":"Directories","text":"Type Name dir cxx The documentation for this class was generated from the following file docs/sw/include/opae/
"},{"location":"opae-code/access_8h/","title":"File access.h","text":"FileList > docs > sw > include > opae > access.h
Go to the source code of this file.
Functions to acquire, release, and reset OPAE FPGA resources.
#include <opae/types.h>
"},{"location":"opae-code/access_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaClose (fpga_handle handle) Close a previously opened FPGA object. fpga_result fpgaOpen (fpga_token token, fpga_handle * handle, int flags) Open an FPGA object. fpga_result fpgaReset (fpga_handle handle) Reset an FPGA object."},{"location":"opae-code/access_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/access_8h/#function-fpgaclose","title":"function fpgaClose","text":"Close a previously opened FPGA object.
fpga_result fpgaClose (\nfpga_handle handle\n)
Relinquishes ownership of a previously fpgaOpen()ed resource. This enables others to acquire ownership if the resource was opened exclusively. Also deallocates / unmaps MMIO and UMsg memory areas.
Parameters:
handle Handle to previously opened FPGA object
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to an acquired resource, or if handle is NULL. FPGA_EXCEPTION if an internal error occurred while accessing the handle.
"},{"location":"opae-code/access_8h/#function-fpgaopen","title":"function fpgaOpen","text":"Open an FPGA object.
fpga_result fpgaOpen (\nfpga_token token,\nfpga_handle * handle,\nint flags\n)
Acquires ownership of the FPGA resource referred to by 'token'.
Most often this will be used to open an accelerator object to directly interact with an accelerator function, or to open an FPGA object to perform management functions.
Parameters:
token Pointer to token identifying resource to acquire ownership of handle Pointer to preallocated memory to place a handle in. This handle will be used in subsequent API calls. flags One of the following flags: - FPGA_OPEN_SHARED allows the resource to be opened multiple times (not supported in ASE) Shared resources (including buffers) are released when all associated handles have been closed (either explicitly with fpgaClose() or by process termination).
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the resource for 'token' could not be found. FPGA_INVALID_PARAM if 'token' does not refer to a resource that can be opened, or if either argument is NULL or invalid. FPGA_EXCEPTION if an internal exception occurred while creating the handle. FPGA_NO_DRIVER if the driver is not loaded. FPGA_BUSY if trying to open a resource that has already been opened in exclusive mode. FPGA_NO_ACCESS if the current process' privileges are not sufficient to open the resource.
"},{"location":"opae-code/access_8h/#function-fpgareset","title":"function fpgaReset","text":"Reset an FPGA object.
fpga_result fpgaReset (\nfpga_handle handle\n)
Performs an accelerator reset.
Parameters:
handle Handle to previously opened FPGA object
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to an acquired resource or to a resource that cannot be reset. FPGA_EXCEPTION if an internal error occurred while trying to access the handle or resetting the resource.
The documentation for this class was generated from the following file docs/sw/include/opae/access.h
"},{"location":"opae-code/access_8h_source/","title":"File access.h","text":"File List > docs > sw > include > opae > access.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_ACCESS_H__\n#define __FPGA_ACCESS_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaOpen(fpga_token token, fpga_handle *handle,\nint flags);\nfpga_result fpgaClose(fpga_handle handle);\nfpga_result fpgaReset(fpga_handle handle);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_ACCESS_H__\n
"},{"location":"opae-code/buffer_8h/","title":"File buffer.h","text":"FileList > docs > sw > include > opae > buffer.h
Go to the source code of this file.
Functions for allocating and sharing system memory with an FPGA accelerator. More...
#include <opae/types.h>
"},{"location":"opae-code/buffer_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetIOAddress (fpga_handle handle, uint64_t wsid, uint64_t * ioaddr) Retrieve base IO address for buffer. fpga_result fpgaPrepareBuffer (fpga_handle handle, uint64_t len, void ** buf_addr, uint64_t * wsid, int flags) Prepare a shared memory buffer. fpga_result fpgaReleaseBuffer (fpga_handle handle, uint64_t wsid) Release a shared memory buffer."},{"location":"opae-code/buffer_8h/#detailed-description","title":"Detailed Description","text":"To share memory between a software application and an FPGA accelerator, these functions set up system components (e.g. an IOMMU) to allow accelerator access to a provided memory region.
There are a number of restrictions on what memory can be shared, depending on platform capabilities. Usually, FPGA accelerators to not have access to virtual address mappings of the CPU, so they can only access physical addresses. To support this, the OPAE C library on Linux uses hugepages to allocate large, contiguous pages of physical memory that can be shared with an accelerator. It also supports sharing memory that has already been allocated by an application, as long as that memory satisfies the requirements of being physically contigous and page-aligned.
"},{"location":"opae-code/buffer_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/buffer_8h/#function-fpgagetioaddress","title":"function fpgaGetIOAddress","text":"Retrieve base IO address for buffer.
fpga_result fpgaGetIOAddress (\nfpga_handle handle,\nuint64_t wsid,\nuint64_t * ioaddr\n)
This function is used to acquire the physical base address (on some platforms called IO Virtual Address or IOVA) for a shared buffer identified by wsid.
Note:
This function will disappear once the APIs for secure sharing of buffer addresses is implemented.
Parameters:
handle Handle to previously opened accelerator resource wsid Buffer handle / workspace ID referring to the buffer for which the IO address is requested ioaddr Pointer to memory where the IO address will be returned
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle. FPGA_NOT_FOUND if wsid does not refer to a previously shared buffer.
"},{"location":"opae-code/buffer_8h/#function-fpgapreparebuffer","title":"function fpgaPrepareBuffer","text":"Prepare a shared memory buffer.
fpga_result fpgaPrepareBuffer (\nfpga_handle handle,\nuint64_t len,\nvoid ** buf_addr,\nuint64_t * wsid,\nint flags\n)
Prepares a memory buffer for shared access between an accelerator and the calling process. This may either include allocation of physical memory, or preparation of already allocated memory for sharing. The latter case is indicated by supplying the FPGA_BUF_PREALLOCATED flag.
This function will ask the driver to pin the indicated memory (make it non-swappable), and program the IOMMU to allow access from the accelerator. If the buffer was not pre-allocated (flag FPGA_BUF_PREALLOCATED), the function will also allocate physical memory of the requested size and map the memory into the caller's process' virtual address space. It returns in 'wsid' an fpga_buffer object that can be used to program address registers in the accelerator for shared access to the memory.
When using FPGA_BUF_PREALLOCATED, the input len must be a non-zero multiple of the page size, else the function returns FPGA_INVALID_PARAM. When not using FPGA_BUF_PREALLOCATED, the input len is rounded up to the nearest multiple of page size.
Parameters:
handle Handle to previously opened accelerator resource len Length of the buffer to allocate/prepare in bytes buf_addr Virtual address of buffer. Contents may be NULL (OS will choose mapping) or non-NULL (OS will take contents as a hint for the virtual address). wsid Handle to the allocated/prepared buffer to be used with other functions flags Flags. FPGA_BUF_PREALLOCATED indicates that memory pointed at in '*buf_addr' is already allocated an mapped into virtual memory. FPGA_BUF_READ_ONLY pins pages with only read access from the FPGA.
Returns:
FPGA_OK on success. FPGA_NO_MEMORY if the requested memory could not be allocated. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
Note:
As a special case, when FPGA_BUF_PREALLOCATED is present in flags, if len == 0 and buf_addr == NULL, then the function returns FPGA_OK if pre-allocated buffers are supported. In this case, a return value other than FPGA_OK indicates that pre-allocated buffers are not supported.
"},{"location":"opae-code/buffer_8h/#function-fpgareleasebuffer","title":"function fpgaReleaseBuffer","text":"Release a shared memory buffer.
fpga_result fpgaReleaseBuffer (\nfpga_handle handle,\nuint64_t wsid\n)
Releases a previously prepared shared buffer. If the buffer was allocated using fpgaPrepareBuffer (FPGA_BUF_PREALLOCATED was not specified), this call will deallocate/free that memory. Otherwise, it will only be returned to it's previous state (pinned/unpinned, cached/non-cached).
Parameters:
handle Handle to previously opened accelerator resource wsid Handle to the allocated/prepared buffer
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
The documentation for this class was generated from the following file docs/sw/include/opae/buffer.h
"},{"location":"opae-code/buffer_8h_source/","title":"File buffer.h","text":"File List > docs > sw > include > opae > buffer.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_BUFFER_H__\n#define __FPGA_BUFFER_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaPrepareBuffer(fpga_handle handle,\nuint64_t len,\nvoid **buf_addr, uint64_t *wsid, int flags);\nfpga_result fpgaReleaseBuffer(fpga_handle handle, uint64_t wsid);\nfpga_result fpgaGetIOAddress(fpga_handle handle, uint64_t wsid,\nuint64_t *ioaddr);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_BUFFER_H__\n
"},{"location":"opae-code/dir_3731a7e5669218b938d292e51b4e531c/","title":"Dir docs/sw/include/opae/cxx","text":"FileList > cxx
"},{"location":"opae-code/dir_3731a7e5669218b938d292e51b4e531c/#files","title":"Files","text":"Type Name file core.h"},{"location":"opae-code/dir_3731a7e5669218b938d292e51b4e531c/#directories","title":"Directories","text":"Type Name dir core The documentation for this class was generated from the following file docs/sw/include/opae/cxx/
"},{"location":"opae-code/core_8h/","title":"File core.h","text":"FileList > cxx > core.h
Go to the source code of this file.
#include <opae/cxx/core/errors.h>#include <opae/cxx/core/events.h>#include <opae/cxx/core/except.h>#include <opae/cxx/core/handle.h>#include <opae/cxx/core/properties.h>#include <opae/cxx/core/pvalue.h>#include <opae/cxx/core/shared_buffer.h>#include <opae/cxx/core/token.h>#include <opae/cxx/core/version.h>
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core.h
"},{"location":"opae-code/core_8h_source/","title":"File core.h","text":"File List > cxx > core.h
Go to the documentation of this file.
// Copyright(c) 2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/errors.h>\n#include <opae/cxx/core/events.h>\n#include <opae/cxx/core/except.h>\n#include <opae/cxx/core/handle.h>\n#include <opae/cxx/core/properties.h>\n#include <opae/cxx/core/pvalue.h>\n#include <opae/cxx/core/shared_buffer.h>\n#include <opae/cxx/core/token.h>\n#include <opae/cxx/core/version.h>\n
"},{"location":"opae-code/dir_23b1b9d7ef54caa3fa7bb54d9bc2d47a/","title":"Dir docs/sw/include/opae/cxx/core","text":"FileList > core
"},{"location":"opae-code/dir_23b1b9d7ef54caa3fa7bb54d9bc2d47a/#files","title":"Files","text":"Type Name file errors.h file events.h file except.h file handle.h file properties.h file pvalue.h file shared_buffer.h file sysobject.h file token.h file version.h The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/
"},{"location":"opae-code/errors_8h/","title":"File errors.h","text":"FileList > core > errors.h
Go to the source code of this file.
#include <opae/cxx/core/token.h>#include <opae/types_enum.h>#include <memory>
"},{"location":"opae-code/errors_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/errors_8h/#classes","title":"Classes","text":"Type Name class error An error object represents an error register for a resource. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/errors.h
"},{"location":"opae-code/errors_8h_source/","title":"File errors.h","text":"File List > core > errors.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/token.h>\n#include <opae/types_enum.h>\n#include <memory>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass error {\npublic:\ntypedef std::shared_ptr<error> ptr_t;\nerror(const error &e) = delete;\nerror &operator=(const error &e) = delete;\nstatic error::ptr_t get(token::ptr_t tok, uint32_t num);\nstd::string name() { return error_info_.name; }\nbool can_clear() { return error_info_.can_clear; }\nuint64_t read_value();\n~error() {}\nfpga_error_info c_type() const { return error_info_; }\nprivate:\nerror(token::ptr_t token, uint32_t num);\ntoken::ptr_t token_;\nfpga_error_info error_info_;\nuint32_t error_num_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/events_8h/","title":"File events.h","text":"FileList > core > events.h
Go to the source code of this file.
#include <opae/cxx/core/handle.h>#include <opae/types_enum.h>#include <memory>
"},{"location":"opae-code/events_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/events_8h/#classes","title":"Classes","text":"Type Name class event Wraps fpga event routines in OPAE C. struct type_t C++ struct that is interchangeable with fpga_event_type enum. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/events.h
"},{"location":"opae-code/events_8h_source/","title":"File events.h","text":"File List > core > events.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/handle.h>\n#include <opae/types_enum.h>\n#include <memory>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass event {\npublic:\ntypedef std::shared_ptr<event> ptr_t;\nvirtual ~event();\nstruct type_t {\ntype_t(fpga_event_type c_type) : type_(c_type) {}\noperator fpga_event_type() { return type_; }\nstatic constexpr fpga_event_type interrupt = FPGA_EVENT_INTERRUPT;\nstatic constexpr fpga_event_type error = FPGA_EVENT_ERROR;\nstatic constexpr fpga_event_type power_thermal = FPGA_EVENT_POWER_THERMAL;\nprivate:\nfpga_event_type type_;\n};\nfpga_event_handle get() { return event_handle_; }\noperator fpga_event_handle();\nstatic event::ptr_t register_event(handle::ptr_t h, event::type_t t,\nint flags = 0);\nint os_object() const;\nprivate:\nevent(handle::ptr_t h, event::type_t t, fpga_event_handle event_h);\nhandle::ptr_t handle_;\nevent::type_t type_;\nfpga_event_handle event_handle_;\nint os_object_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/except_8h/","title":"File except.h","text":"FileList > core > except.h
Go to the source code of this file.
#include <opae/types_enum.h>#include <cstdint>#include <exception>
"},{"location":"opae-code/except_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types namespace detail"},{"location":"opae-code/except_8h/#classes","title":"Classes","text":"Type Name class busy busy exception class except Generic OPAE exception. class exception exception exception class invalid_param invalid_param exception class no_access no_access exception class no_daemon no_daemon exception class no_driver no_driver exception class no_memory no_memory exception class not_found not_found exception class not_supported not_supported exception class reconf_error reconf_error exception class src_location Identify a particular line in a source file."},{"location":"opae-code/except_8h/#macros","title":"Macros","text":"Type Name define ASSERT_FPGA_OK \u00ae Macro to check of result is FPGA_OK If not, throw exception that corresponds to the result code. define OPAECXX_HERE opae::fpga::types::src_location(__FILE__, __func__, __LINE__)Construct a src_location object for the current source line."},{"location":"opae-code/except_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/except_8h/#define-assert_fpga_ok","title":"define ASSERT_FPGA_OK","text":"#define ASSERT_FPGA_OK (\nr\n) opae::fpga::types::detail::assert_fpga_ok( \\\n r, opae::fpga::types::src_location (__FILE__, __func__, __LINE__));\n
"},{"location":"opae-code/except_8h/#define-opaecxx_here","title":"define OPAECXX_HERE","text":"#define OPAECXX_HERE opae::fpga::types::src_location (__FILE__, __func__, __LINE__)\n
The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/except.h
"},{"location":"opae-code/except_8h_source/","title":"File except.h","text":"File List > core > except.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/types_enum.h>\n#include <cstdint>\n#include <exception>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass src_location {\npublic:\nsrc_location(const char *file, const char *fn, int line) noexcept;\nsrc_location(const src_location &other) noexcept;\nsrc_location &operator=(const src_location &other) noexcept;\nconst char *file() const noexcept;\nconst char *fn() const noexcept { return fn_; }\nint line() const noexcept { return line_; }\nprivate:\nconst char *file_;\nconst char *fn_;\nint line_;\n};\n#define OPAECXX_HERE \\\n opae::fpga::types::src_location(__FILE__, __func__, __LINE__)\nclass except : public std::exception {\npublic:\nstatic const std::size_t MAX_EXCEPT = 256;\nexcept(src_location loc) noexcept;\nexcept(fpga_result res, src_location loc) noexcept;\nexcept(fpga_result res, const char *msg, src_location loc) noexcept;\nvirtual const char *what() const noexcept override;\noperator fpga_result() const noexcept { return res_; }\nprotected:\nfpga_result res_;\nconst char *msg_;\nsrc_location loc_;\nmutable char buf_[MAX_EXCEPT];\n};\nclass invalid_param : public except {\npublic:\ninvalid_param(src_location loc) noexcept\n: except(FPGA_INVALID_PARAM, \"failed with return code FPGA_INVALID_PARAM\",\nloc) {}\n};\nclass busy : public except {\npublic:\nbusy(src_location loc) noexcept\n: except(FPGA_BUSY, \"failed with return code FPGA_BUSY\", loc) {}\n};\nclass exception : public except {\npublic:\nexception(src_location loc) noexcept\n: except(FPGA_EXCEPTION, \"failed with return code FPGA_EXCEPTION\", loc) {}\n};\nclass not_found : public except {\npublic:\nnot_found(src_location loc) noexcept\n: except(FPGA_NOT_FOUND, \"failed with return code FPGA_NOT_FOUND\", loc) {}\n};\nclass no_memory : public except {\npublic:\nno_memory(src_location loc) noexcept\n: except(FPGA_NO_MEMORY, \"failed with return code FPGA_NO_MEMORY\", loc) {}\n};\nclass not_supported : public except {\npublic:\nnot_supported(src_location loc) noexcept\n: except(FPGA_NOT_SUPPORTED, \"failed with return code FPGA_NOT_SUPPORTED\",\nloc) {}\n};\nclass no_driver : public except {\npublic:\nno_driver(src_location loc) noexcept\n: except(FPGA_NO_DRIVER, \"failed with return code FPGA_NO_DRIVER\", loc) {}\n};\nclass no_daemon : public except {\npublic:\nno_daemon(src_location loc) noexcept\n: except(FPGA_NO_DAEMON, \"failed with return code FPGA_NO_DAEMON\", loc) {}\n};\nclass no_access : public except {\npublic:\nno_access(src_location loc) noexcept\n: except(FPGA_NO_ACCESS, \"failed with return code FPGA_NO_ACCESS\", loc) {}\n};\nclass reconf_error : public except {\npublic:\nreconf_error(src_location loc) noexcept\n: except(FPGA_RECONF_ERROR, \"failed with return code FPGA_RECONF_ERROR\",\nloc) {}\n};\nnamespace detail {\ntypedef bool (*exception_fn)(fpga_result,\nconst opae::fpga::types::src_location &loc);\ntemplate <typename T>\nconstexpr bool is_ok(fpga_result result,\nconst opae::fpga::types::src_location &loc) {\nreturn result == FPGA_OK ? true : throw T(loc);\n}\nstatic exception_fn opae_exceptions[12] = {\nis_ok<opae::fpga::types::invalid_param>,\nis_ok<opae::fpga::types::busy>,\nis_ok<opae::fpga::types::exception>,\nis_ok<opae::fpga::types::not_found>,\nis_ok<opae::fpga::types::no_memory>,\nis_ok<opae::fpga::types::not_supported>,\nis_ok<opae::fpga::types::no_driver>,\nis_ok<opae::fpga::types::no_daemon>,\nis_ok<opae::fpga::types::no_access>,\nis_ok<opae::fpga::types::reconf_error>};\nstatic inline void assert_fpga_ok(fpga_result result,\nconst opae::fpga::types::src_location &loc) {\nif (result > FPGA_OK && result <= FPGA_RECONF_ERROR)\n// our exception table above starts at invalid_param with index 0\n// but FPGA_INVALID_PARAM is actually enum 1 - let's account for that\nopae_exceptions[result - 1](result, loc);\n}\n} // end of namespace detail\n#define ASSERT_FPGA_OK(r) \\\n opae::fpga::types::detail::assert_fpga_ok( \\\n r, opae::fpga::types::src_location(__FILE__, __func__, __LINE__));\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/handle_8h/","title":"File handle.h","text":"FileList > core > handle.h
Go to the source code of this file.
#include <opae/cxx/core/token.h>#include <opae/enum.h>#include <opae/types.h>#include <memory>#include <vector>
"},{"location":"opae-code/handle_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/handle_8h/#classes","title":"Classes","text":"Type Name class handle An allocated accelerator resource. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/handle.h
"},{"location":"opae-code/handle_8h_source/","title":"File handle.h","text":"File List > core > handle.h
Go to the documentation of this file.
// Copyright(c) 2018-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/token.h>\n#include <opae/enum.h>\n#include <opae/types.h>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass handle {\npublic:\ntypedef std::shared_ptr<handle> ptr_t;\nhandle(const handle &) = delete;\nhandle &operator=(const handle &) = delete;\nvirtual ~handle();\nfpga_handle c_type() const { return handle_; }\noperator fpga_handle() const { return handle_; }\nvoid reconfigure(uint32_t slot, const uint8_t *bitstream, size_t size,\nint flags);\nuint32_t read_csr32(uint64_t offset, uint32_t csr_space = 0) const;\nvoid write_csr32(uint64_t offset, uint32_t value, uint32_t csr_space = 0);\nuint64_t read_csr64(uint64_t offset, uint32_t csr_space = 0) const;\nvoid write_csr64(uint64_t offset, uint64_t value, uint32_t csr_space = 0);\nvoid write_csr512(uint64_t offset, const void *value, uint32_t csr_space = 0);\nuint8_t *mmio_ptr(uint64_t offset, uint32_t csr_space = 0) const;\nstatic handle::ptr_t open(fpga_token token, int flags);\nstatic handle::ptr_t open(token::ptr_t token, int flags);\nvirtual void reset();\nfpga_result close();\ntoken::ptr_t get_token() const;\nprivate:\nhandle(fpga_handle h);\nfpga_handle handle_;\nfpga_token token_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/cxx_2core_2properties_8h/","title":"File properties.h","text":"FileList > core > properties.h
Go to the source code of this file.
#include <opae/cxx/core/pvalue.h>#include <opae/properties.h>#include <iostream>#include <map>#include <memory>#include <vector>
"},{"location":"opae-code/cxx_2core_2properties_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/cxx_2core_2properties_8h/#classes","title":"Classes","text":"Type Name class properties Wraps an OPAE fpga_properties object. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/properties.h
"},{"location":"opae-code/cxx_2core_2properties_8h_source/","title":"File properties.h","text":"File List > core > properties.h
Go to the documentation of this file.
// Copyright(c) 2018-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/pvalue.h>\n#include <opae/properties.h>\n#include <iostream>\n#include <map>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass token;\nclass handle;\nclass properties {\npublic:\ntypedef std::shared_ptr<properties> ptr_t;\nconst static std::vector<properties::ptr_t> none;\nproperties(const properties &p) = delete;\nproperties &operator=(const properties &p) = delete;\n~properties();\nfpga_properties c_type() const { return props_; }\noperator fpga_properties() const { return props_; }\nstatic properties::ptr_t get();\nstatic properties::ptr_t get(fpga_guid guid_in);\nstatic properties::ptr_t get(fpga_objtype objtype);\nstatic properties::ptr_t get(std::shared_ptr<token> t);\nstatic properties::ptr_t get(fpga_token t);\nstatic properties::ptr_t get(std::shared_ptr<handle> h);\nprivate:\nproperties(bool alloc_props = true);\nfpga_properties props_;\npublic:\npvalue<fpga_objtype> type;\npvalue<uint32_t> num_errors;\npvalue<uint16_t> segment;\npvalue<uint8_t> bus;\npvalue<uint8_t> device;\npvalue<uint8_t> function;\npvalue<uint8_t> socket_id;\npvalue<uint32_t> num_slots;\npvalue<uint64_t> bbs_id;\npvalue<fpga_version> bbs_version;\npvalue<uint16_t> vendor_id;\npvalue<uint16_t> device_id;\npvalue<uint16_t> subsystem_vendor_id;\npvalue<uint16_t> subsystem_device_id;\npvalue<char *> model;\npvalue<uint64_t> local_memory_size;\npvalue<uint64_t> capabilities;\npvalue<uint32_t> num_mmio;\npvalue<uint32_t> num_interrupts;\npvalue<fpga_accelerator_state> accelerator_state;\npvalue<uint64_t> object_id;\npvalue<fpga_token> parent;\npvalue<fpga_interface> interface;\nguid_t guid;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/pvalue_8h/","title":"File pvalue.h","text":"FileList > core > pvalue.h
Go to the source code of this file.
#include <opae/cxx/core/except.h>#include <opae/properties.h>#include <opae/utils.h>#include <uuid/uuid.h>#include <algorithm>#include <array>#include <cstring>#include <iostream>#include <type_traits>
"},{"location":"opae-code/pvalue_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/pvalue_8h/#classes","title":"Classes","text":"Type Name struct guid_t Representation of the guid member of a properties object. struct pvalue <typename T>Wraps OPAE properties defined in the OPAE C API by associating an fpga_properties reference with the getters and setters defined for a property. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/pvalue.h
"},{"location":"opae-code/pvalue_8h_source/","title":"File pvalue.h","text":"File List > core > pvalue.h
Go to the documentation of this file.
// Copyright(c) 2018-2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/except.h>\n#include <opae/properties.h>\n#include <opae/utils.h>\n#include <uuid/uuid.h>\n#include <algorithm>\n#include <array>\n#include <cstring>\n#include <iostream>\n#include <type_traits>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nstruct guid_t {\nguid_t(fpga_properties *p) : props_(p), is_set_(false) {}\nvoid update() {\nfpga_result res = fpgaPropertiesGetGUID(\n*props_, reinterpret_cast<fpga_guid *>(data_.data()));\nASSERT_FPGA_OK(res);\nis_set_ = true;\n}\noperator uint8_t *() {\nupdate();\nreturn data_.data();\n}\nconst uint8_t *c_type() const { return data_.data(); }\nguid_t &operator=(fpga_guid g) {\nis_set_ = false;\nASSERT_FPGA_OK(fpgaPropertiesSetGUID(*props_, g));\nis_set_ = true;\nuint8_t *begin = &g[0];\nuint8_t *end = begin + sizeof(fpga_guid);\nstd::copy(begin, end, data_.begin());\nreturn *this;\n}\nbool operator==(const fpga_guid &g) {\nreturn is_set() && (0 == std::memcmp(data_.data(), g, sizeof(fpga_guid)));\n}\nvoid parse(const char *str) {\nis_set_ = false;\nif (0 != uuid_parse(str, data_.data())) {\nthrow except(OPAECXX_HERE);\n}\nASSERT_FPGA_OK(fpgaPropertiesSetGUID(*props_, data_.data()));\nis_set_ = true;\n}\nfriend std::ostream &operator<<(std::ostream &ostr, const guid_t &g) {\nfpga_properties props = *g.props_;\nfpga_guid guid_value;\nfpga_result res;\nif ((res = fpgaPropertiesGetGUID(props, &guid_value)) == FPGA_OK) {\nchar guid_str[84];\nuuid_unparse(guid_value, guid_str);\nostr << guid_str;\n} else if (FPGA_NOT_FOUND == res) {\nstd::cerr << \"[guid_t::<<] GUID property not set\\n\";\n} else {\nASSERT_FPGA_OK(res);\n}\nreturn ostr;\n}\nbool is_set() const { return is_set_; }\nvoid invalidate() { is_set_ = false; }\nprivate:\nfpga_properties *props_;\nbool is_set_;\nstd::array<uint8_t, 16> data_;\n};\ntemplate <typename T>\nstruct pvalue {\ntypedef typename std::conditional<\nstd::is_same<T, char *>::value, fpga_result (*)(fpga_properties, T),\nfpga_result (*)(fpga_properties, T *)>::type getter_t;\ntypedef fpga_result (*setter_t)(fpga_properties, T);\ntypedef typename std::conditional<std::is_same<T, char *>::value,\ntypename std::string, T>::type copy_t;\npvalue() : props_(0), is_set_(false), get_(nullptr), set_(nullptr), copy_() {}\npvalue(fpga_properties *p, getter_t g, setter_t s)\n: props_(p), is_set_(false), get_(g), set_(s), copy_() {}\npvalue<T> &operator=(const T &v) {\nis_set_ = false;\nASSERT_FPGA_OK(set_(*props_, v));\nis_set_ = true;\ncopy_ = v;\nreturn *this;\n}\nbool operator==(const T &other) { return is_set() && (copy_ == other); }\nvoid update() {\nASSERT_FPGA_OK(get_(*props_, ©_));\nis_set_ = true;\n}\noperator copy_t() {\nupdate();\nreturn copy_;\n}\n// TODO: Remove this once all properties are tested\nfpga_result get_value(T &value) const { return get_(*props_, &value); }\nfriend std::ostream &operator<<(std::ostream &ostr, const pvalue<T> &p) {\nT value;\nfpga_properties props = *p.props_;\nfpga_result res;\nif ((res = p.get_(props, &value)) == FPGA_OK) {\nostr << +(value);\n} else if (FPGA_NOT_FOUND == res) {\nstd::cerr << \"property getter returned (\" << res << \") \"\n<< fpgaErrStr(res);\n} else {\nASSERT_FPGA_OK(res);\n}\nreturn ostr;\n}\nbool is_set() const { return is_set_; }\nvoid invalidate() { is_set_ = false; }\nprivate:\nfpga_properties *props_;\nbool is_set_;\ngetter_t get_;\nsetter_t set_;\ncopy_t copy_;\n};\ntemplate <>\ninline void pvalue<char *>::update() {\nchar buf[256];\nASSERT_FPGA_OK(get_(*props_, buf));\ncopy_.assign(buf);\nis_set_ = true;\n}\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/shared__buffer_8h/","title":"File shared_buffer.h","text":"FileList > core > shared_buffer.h
Go to the source code of this file.
#include <opae/buffer.h>#include <opae/cxx/core/except.h>#include <opae/cxx/core/handle.h>#include <chrono>#include <cstdint>#include <initializer_list>#include <memory>#include <thread>#include <vector>
"},{"location":"opae-code/shared__buffer_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/shared__buffer_8h/#classes","title":"Classes","text":"Type Name class shared_buffer Host/AFU shared memory blocks. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/shared_buffer.h
"},{"location":"opae-code/shared__buffer_8h_source/","title":"File shared_buffer.h","text":"File List > core > shared_buffer.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/buffer.h>\n#include <opae/cxx/core/except.h>\n#include <opae/cxx/core/handle.h>\n#include <chrono>\n#include <cstdint>\n#include <initializer_list>\n#include <memory>\n#include <thread>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass shared_buffer {\npublic:\ntypedef std::size_t size_t;\ntypedef std::shared_ptr<shared_buffer> ptr_t;\nshared_buffer(const shared_buffer &) = delete;\nshared_buffer &operator=(const shared_buffer &) = delete;\nvirtual ~shared_buffer();\nstatic shared_buffer::ptr_t allocate(handle::ptr_t handle, size_t len,\nbool read_only = false);\nstatic shared_buffer::ptr_t attach(handle::ptr_t handle, uint8_t *base,\nsize_t len, bool read_only = false);\nvoid release();\nvolatile uint8_t *c_type() const { return virt_; }\nhandle::ptr_t owner() const { return handle_; }\nsize_t size() const { return len_; }\nuint64_t wsid() const { return wsid_; }\nuint64_t io_address() const { return io_address_; }\nvoid fill(int c);\nint compare(ptr_t other, size_t len) const;\ntemplate <typename T>\nT read(size_t offset) const {\nif ((offset < len_) && (virt_ != nullptr)) {\nreturn *reinterpret_cast<T *>(virt_ + offset);\n} else if (offset >= len_) {\nthrow except(OPAECXX_HERE);\n} else {\nthrow except(OPAECXX_HERE);\n}\nreturn T();\n}\ntemplate <typename T>\nvoid write(const T &value, size_t offset) {\nif ((offset < len_) && (virt_ != nullptr)) {\n*reinterpret_cast<T *>(virt_ + offset) = value;\n} else if (offset >= len_) {\nthrow except(OPAECXX_HERE);\n} else {\nthrow except(OPAECXX_HERE);\n}\n}\nprotected:\nshared_buffer(handle::ptr_t handle, size_t len, uint8_t *virt, uint64_t wsid,\nuint64_t io_address);\nhandle::ptr_t handle_;\nsize_t len_;\nuint8_t *virt_;\nuint64_t wsid_;\nuint64_t io_address_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/cxx_2core_2sysobject_8h/","title":"File sysobject.h","text":"FileList > core > sysobject.h
Go to the source code of this file.
#include <opae/cxx/core/handle.h>#include <opae/cxx/core/token.h>#include <opae/types.h>#include <memory>#include <vector>
"},{"location":"opae-code/cxx_2core_2sysobject_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/cxx_2core_2sysobject_8h/#classes","title":"Classes","text":"Type Name class sysobject Wraps the OPAE fpga_object primitive. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/sysobject.h
"},{"location":"opae-code/cxx_2core_2sysobject_8h_source/","title":"File sysobject.h","text":"File List > core > sysobject.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/cxx/core/handle.h>\n#include <opae/cxx/core/token.h>\n#include <opae/types.h>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass sysobject {\npublic:\ntypedef std::shared_ptr<sysobject> ptr_t;\nsysobject() = delete;\nsysobject(const sysobject &o) = delete;\nsysobject &operator=(const sysobject &o) = delete;\nstatic sysobject::ptr_t get(token::ptr_t t, const std::string &name,\nint flags = 0);\nstatic sysobject::ptr_t get(handle::ptr_t h, const std::string &name,\nint flags = 0);\nsysobject::ptr_t get(const std::string &name, int flags = 0);\nsysobject::ptr_t get(int index);\nvirtual ~sysobject();\nuint32_t size() const;\nuint64_t read64(int flags = 0) const;\nvoid write64(uint64_t value, int flags = 0) const;\nstd::vector<uint8_t> bytes(int flags = 0) const;\nstd::vector<uint8_t> bytes(uint32_t offset, uint32_t size,\nint flags = 0) const;\nenum fpga_sysobject_type type() const;\nfpga_object c_type() const { return sysobject_; }\noperator fpga_object() const { return sysobject_; }\nprivate:\nsysobject(fpga_object sysobj, token::ptr_t token, handle::ptr_t hnd);\nfpga_object sysobject_;\ntoken::ptr_t token_;\nhandle::ptr_t handle_;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/token_8h/","title":"File token.h","text":"FileList > core > token.h
Go to the source code of this file.
#include <opae/access.h>#include <opae/cxx/core/properties.h>#include <opae/enum.h>#include <opae/types.h>#include <memory>#include <vector>
"},{"location":"opae-code/token_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/token_8h/#classes","title":"Classes","text":"Type Name class token Wraps the OPAE fpga_token primitive. The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/token.h
"},{"location":"opae-code/token_8h_source/","title":"File token.h","text":"File List > core > token.h
Go to the documentation of this file.
// Copyright(c) 2018-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/access.h>\n#include <opae/cxx/core/properties.h>\n#include <opae/enum.h>\n#include <opae/types.h>\n#include <memory>\n#include <vector>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass token {\npublic:\ntypedef std::shared_ptr<token> ptr_t;\nstatic std::vector<token::ptr_t> enumerate(\nconst std::vector<properties::ptr_t>& props);\n~token();\nfpga_token c_type() const { return token_; }\noperator fpga_token() const { return token_; }\nptr_t get_parent() const;\nprivate:\ntoken(fpga_token tok);\nfpga_token token_;\nfriend class handle;\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/cxx_2core_2version_8h/","title":"File version.h","text":"FileList > core > version.h
Go to the source code of this file.
#include <opae/types.h>#include <string>
"},{"location":"opae-code/cxx_2core_2version_8h/#namespaces","title":"Namespaces","text":"Type Name namespace opae namespace fpga namespace types"},{"location":"opae-code/cxx_2core_2version_8h/#classes","title":"Classes","text":"Type Name class version The documentation for this class was generated from the following file docs/sw/include/opae/cxx/core/version.h
"},{"location":"opae-code/cxx_2core_2version_8h_source/","title":"File version.h","text":"File List > core > version.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#pragma once\n#include <opae/types.h>\n#include <string>\nnamespace opae {\nnamespace fpga {\nnamespace types {\nclass version {\npublic:\nstatic fpga_version as_struct();\nstatic std::string as_string();\nstatic std::string build();\n};\n} // end of namespace types\n} // end of namespace fpga\n} // end of namespace opae\n
"},{"location":"opae-code/enum_8h/","title":"File enum.h","text":"FileList > docs > sw > include > opae > enum.h
Go to the source code of this file.
APIs for resource enumeration and managing tokens. More...
#include <opae/types.h>
"},{"location":"opae-code/enum_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaCloneToken (fpga_token src, fpga_token * dst) Clone a fpga_token object. fpga_result fpgaDestroyToken (fpga_token * token) Destroy a Token. fpga_result fpgaEnumerate (const fpga_properties * filters, uint32_t num_filters, fpga_token * tokens, uint32_t max_tokens, uint32_t * num_matches) Enumerate FPGA resources present in the system."},{"location":"opae-code/enum_8h/#detailed-description","title":"Detailed Description","text":"These APIs are the first step for any application using OPAE to discover resources that are present on the system. They allow selective enumeration (i.e. getting a list of resources that match a given list of criteria) and methods to manage the lifecycle of tokens generated by fpgaEnumerate().
"},{"location":"opae-code/enum_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/enum_8h/#function-fpgaclonetoken","title":"function fpgaCloneToken","text":"Clone a fpga_token object.
fpga_result fpgaCloneToken (\nfpga_token src,\nfpga_token * dst\n)
Creates a copy of an fpga_token object.
Note:
This call creates a new token object and allocates memory for it. It is the responsibility of the using application to free this memory after use by calling fpgaDestroyToken() for the cloned token.
Parameters:
src fpga_token object to copy dst New fpga_token object cloned from 'src'
Returns:
FPGA_OK on success
"},{"location":"opae-code/enum_8h/#function-fpgadestroytoken","title":"function fpgaDestroyToken","text":"Destroy a Token.
fpga_result fpgaDestroyToken (\nfpga_token * token\n)
This function destroys a token created by fpgaEnumerate() and frees the associated memory.
Note:
fpgaDestroyToken() requires the address of an fpga_token as previously created by fpgaEnumerate() or fpgaCloneToken(). Passing any other value results in undefined behavior.
Parameters:
token fpga_token to destroy
Returns:
FPGA_OK on success
"},{"location":"opae-code/enum_8h/#function-fpgaenumerate","title":"function fpgaEnumerate","text":"Enumerate FPGA resources present in the system.
fpga_result fpgaEnumerate (\nconst fpga_properties * filters,\nuint32_t num_filters,\nfpga_token * tokens,\nuint32_t max_tokens,\nuint32_t * num_matches\n)
This call allows the user to query the system for FPGA resources that match a certain set of criteria, e.g. all accelerators that are assigned to a host interface and available, all FPGAs of a specific type, etc.
fpgaEnumerate() will create a number of fpga_tokens to represent the matching resources and populate the array tokens with these tokens. The max_tokens argument can be used to limit the number of tokens allocated/returned by fpgaEnumerate(); i.e., the number of tokens in the returned tokens array will be either max_tokens or num_matches (the number of resources matching the filter), whichever is smaller. Use fpgaDestroyToken() to destroy tokens that are no longer needed.
To query the number of matches for a particular set of filters (e.g. to allocate a tokens array of the appropriate size), call fpgaEnumerate() with the parameter tokens set to NULL; this will only return the number of matches in num_matches.
Note:
fpgaEnumerate() will allocate memory for the created tokens returned in tokens. It is the responsibility of the using application to free this memory after use by calling fpgaDestroyToken() for each of the returned tokens.
Parameters:
filters Array of fpga_properties objects describing the properties of the objects that should be returned. A resource is considered matching if its properties match any one of the supplied filters. To match all FPGA resources, pass an empty filters object (one without any filter criteria set) or pass a NULL filters parameter with num_filters set to 0. num_filters Number of entries in the filters array, or 0 to match all FPGA resources when filters is NULL. tokens Pointer to an array of fpga_token variables to be populated. If NULL is supplied, fpgaEnumerate() will not create any tokens, but it will return the number of possible matches in num_match. max_tokens Maximum number of tokens that fpgaEnumerate() shall return (length of tokens array). There may be more or fewer matches than this number; num_matches is set to the number of actual matches. num_matches Number of resources matching the filter criteria. This number can be higher than the number of tokens returned in the tokens array (depending on the value of max_tokens).
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid pointers or objects are passed into the function. FPGA_NO_DRIVER if OPAE can't find the respective enumeration data structures usually provided by the driver. FPGA_NO_MEMORY if there was not enough memory to create tokens.
The documentation for this class was generated from the following file docs/sw/include/opae/enum.h
"},{"location":"opae-code/enum_8h_source/","title":"File enum.h","text":"File List > docs > sw > include > opae > enum.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_ENUM_H__\n#define __FPGA_ENUM_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaEnumerate(const fpga_properties *filters,\nuint32_t num_filters, fpga_token *tokens,\nuint32_t max_tokens, uint32_t *num_matches);\nfpga_result fpgaCloneToken(fpga_token src, fpga_token *dst);\nfpga_result fpgaDestroyToken(fpga_token *token);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_ENUM_H__\n
"},{"location":"opae-code/error_8h/","title":"File error.h","text":"FileList > docs > sw > include > opae > error.h
Go to the source code of this file.
Functions for reading and clearing errors in resources. More...
#include <opae/types.h>
"},{"location":"opae-code/error_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaClearAllErrors (fpga_token token) Clear all error registers of a particular resource. fpga_result fpgaClearError (fpga_token token, uint32_t error_num) Clear error register. fpga_result fpgaGetErrorInfo (fpga_token token, uint32_t error_num, struct fpga_error_info * error_info) Get information about a particular error register. fpga_result fpgaReadError (fpga_token token, uint32_t error_num, uint64_t * value) Read error value."},{"location":"opae-code/error_8h/#detailed-description","title":"Detailed Description","text":"Many FPGA resources have the ability to track the occurrence of errors. This file provides functions to retrieve information about errors within resources.
"},{"location":"opae-code/error_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/error_8h/#function-fpgaclearallerrors","title":"function fpgaClearAllErrors","text":"Clear all error registers of a particular resource.
fpga_result fpgaClearAllErrors (\nfpga_token token\n)
This function will clear all error registers of the resource referenced by token, observing the necessary order of clearing errors, if any.
Parameters:
token Token to accelerator resource to query
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token, and FPGA_BUSY if error could not be cleared.
"},{"location":"opae-code/error_8h/#function-fpgaclearerror","title":"function fpgaClearError","text":"Clear error register.
fpga_result fpgaClearError (\nfpga_token token,\nuint32_t error_num\n)
This function will clear the error register error_num of the resource referenced by token.
Parameters:
token Token to accelerator resource to query error_num Number of error register to clear
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token, and FPGA_BUSY if error could not be cleared.
"},{"location":"opae-code/error_8h/#function-fpgageterrorinfo","title":"function fpgaGetErrorInfo","text":"Get information about a particular error register.
fpga_result fpgaGetErrorInfo (\nfpga_token token,\nuint32_t error_num,\nstruct fpga_error_info * error_info\n)
This function will populate a fpga_error_info struct with information about error number error_num of the resource referenced by token.
Parameters:
token Token to accelerator resource to query error_num Error register to retrieve information about error_info Pointer to memory to store information into
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token.
"},{"location":"opae-code/error_8h/#function-fpgareaderror","title":"function fpgaReadError","text":"Read error value.
fpga_result fpgaReadError (\nfpga_token token,\nuint32_t error_num,\nuint64_t * value\n)
This function will read the value of error register error_num of the resource referenced by token into the memory location pointed to by value.
Parameters:
token Token to accelerator resource to query error_num Number of error register to read value Pointer to memory to store error value into (64 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the token.
The documentation for this class was generated from the following file docs/sw/include/opae/error.h
"},{"location":"opae-code/error_8h_source/","title":"File error.h","text":"File List > docs > sw > include > opae > error.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_ERROR_H__\n#define __FPGA_ERROR_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaReadError(fpga_token token, uint32_t error_num, uint64_t *value);\nfpga_result fpgaClearError(fpga_token token, uint32_t error_num);\nfpga_result fpgaClearAllErrors(fpga_token token);\nfpga_result fpgaGetErrorInfo(fpga_token token,\nuint32_t error_num,\nstruct fpga_error_info *error_info);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_ERROR_H__\n
"},{"location":"opae-code/event_8h/","title":"File event.h","text":"FileList > docs > sw > include > opae > event.h
Go to the source code of this file.
Functions for registering events and managing the lifecycle for fpga_event_handle s.More...
#include <opae/types.h>
"},{"location":"opae-code/event_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaCreateEventHandle (fpga_event_handle * event_handle) Initialize an event_handle. fpga_result fpgaDestroyEventHandle (fpga_event_handle * event_handle) Destroy an event_handle. fpga_result fpgaGetOSObjectFromEventHandle (const fpga_event_handle eh, int * fd) Get OS object from event handle. fpga_result fpgaRegisterEvent (fpga_handle handle, fpga_event_type event_type, fpga_event_handle event_handle, uint32_t flags) Register an FPGA event. fpga_result fpgaUnregisterEvent (fpga_handle handle, fpga_event_type event_type, fpga_event_handle event_handle) Unregister an FPGA event."},{"location":"opae-code/event_8h/#detailed-description","title":"Detailed Description","text":"OPAE provides an interface to asynchronous events that can be generated by different FPGA resources. The event API provides functions to register for these events; associated with every event a process has registered for is an fpga_event_handle, which encapsulates the OS-specific data structure for event objects. On Linux, an fpga_event_handle can be used as a file descriptor and passed to select(), poll(), epoll() and similar functions to wait for asynchronous events.
"},{"location":"opae-code/event_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/event_8h/#function-fpgacreateeventhandle","title":"function fpgaCreateEventHandle","text":"Initialize an event_handle.
fpga_result fpgaCreateEventHandle (\nfpga_event_handle * event_handle\n)
Platform independent way to initialize an event_handle used for notifications from the driver to application. For Linux, this function creates an eventfd and returns the eventfd file descriptor in *event_handle.
Parameters:
event_handle Pointer to event handle variable.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if event_handle is NULL. FPGA_NOT_SUPPORTED if platform does not support events.
"},{"location":"opae-code/event_8h/#function-fpgadestroyeventhandle","title":"function fpgaDestroyEventHandle","text":"Destroy an event_handle.
fpga_result fpgaDestroyEventHandle (\nfpga_event_handle * event_handle\n)
Destroy handle and free resources. On Linux this corresponds to closing the file descriptor pointed to by handle
Note:
fpgaDestroyEventHandle() requires the address of an event_handle as created by fpgaCreateEventHandle(). Passing any other value results in undefined behavior.
Parameters:
event_handle Pointer to handle to be destroyed
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if event_handle is NULL.
"},{"location":"opae-code/event_8h/#function-fpgagetosobjectfromeventhandle","title":"function fpgaGetOSObjectFromEventHandle","text":"Get OS object from event handle.
fpga_result fpgaGetOSObjectFromEventHandle (\nconst fpga_event_handle eh,\nint * fd\n)
Check validity of event handle, and get the OS object used to subscribe and unsubscribe to events. On Linux, the object corresponds to a file descriptor.
Parameters:
eh Event handle to get the descriptor value from fd integer to store the descriptor value
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if event_handle is invalid.
"},{"location":"opae-code/event_8h/#function-fpgaregisterevent","title":"function fpgaRegisterEvent","text":"Register an FPGA event.
fpga_result fpgaRegisterEvent (\nfpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle,\nuint32_t flags\n)
This function tells the driver that the caller is interested in notification for the event specified by the type and flags pair.
The event_handle points to an OS specific mechanism for event notification. An event_handle is associated with only a single event.
In case of user interrupts, the flags parameter will be used to specify the vector ID. The value of the flags parameter indicates the vector ID, no bit encoding is used.
Todo
define if calling fpgaRegisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored.
Parameters:
handle Handle to previously opened FPGA resource. event_type Type of event event_handle Handle to previously opened resource for event notification. flags Optional argument for specifying additional information about event. For example irq number for interrupt events.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to a resource supporting the requested event, or if event_handle is not valid. FPGA_EXCEPTION if an internal exception occurred while accessing the handle or the event_handle. On Linux: FPGA_NO_DAEMON if the driver does not support the requested event and there is no FPGA Daemon (fpgad) running to proxy it.
"},{"location":"opae-code/event_8h/#function-fpgaunregisterevent","title":"function fpgaUnregisterEvent","text":"Unregister an FPGA event.
fpga_result fpgaUnregisterEvent (\nfpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle\n)
This function tells the driver that the caller is no longer interested in notification for the event associated with the event_handle
The event_handle points to an OS specific mechanism for event notification. An event_handle is associated with only a single event.
Todo
define if calling fpgaUnregisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored.
Parameters:
handle Handle to previously opened FPGA resource. event_type Type of event to unregister. event_handle Handle to previously registered resource for event notification.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if handle does not refer to a resource supporting the requested event, or if event_handle is not valid. FPGA_EXCEPTION if an internal error occurred accessing the handle or the event_handle.
The documentation for this class was generated from the following file docs/sw/include/opae/event.h
"},{"location":"opae-code/event_8h_source/","title":"File event.h","text":"File List > docs > sw > include > opae > event.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_EVENT_H__\n#define __FPGA_EVENT_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaCreateEventHandle(fpga_event_handle *event_handle);\nfpga_result fpgaDestroyEventHandle(fpga_event_handle *event_handle);\nfpga_result fpgaGetOSObjectFromEventHandle(const fpga_event_handle eh, int *fd);\nfpga_result fpgaRegisterEvent(fpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle,\nuint32_t flags);\nfpga_result fpgaUnregisterEvent(fpga_handle handle,\nfpga_event_type event_type,\nfpga_event_handle event_handle);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_EVENT_H__\n
"},{"location":"opae-code/fpga_8h/","title":"File fpga.h","text":"FileList > docs > sw > include > opae > fpga.h
Go to the source code of this file.
FPGA API. More...
#include <opae/log.h>#include <opae/init.h>#include <opae/types.h>#include <opae/access.h>#include <opae/buffer.h>#include <opae/enum.h>#include <opae/event.h>#include <opae/manage.h>#include <opae/mmio.h>#include <opae/properties.h>#include <opae/umsg.h>#include <opae/utils.h>#include <opae/error.h>#include <opae/version.h>#include <opae/sysobject.h>#include <opae/userclk.h>#include <opae/metrics.h>
"},{"location":"opae-code/fpga_8h/#detailed-description","title":"Detailed Description","text":"This conveniently includes all APIs that a part of the OPAE release (base and extensions).
The documentation for this class was generated from the following file docs/sw/include/opae/fpga.h
"},{"location":"opae-code/fpga_8h_source/","title":"File fpga.h","text":"File List > docs > sw > include > opae > fpga.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_FPGA_H__\n#define __FPGA_FPGA_H__\n#include <opae/log.h>\n#include <opae/init.h>\n#include <opae/types.h>\n#include <opae/access.h>\n#include <opae/buffer.h>\n#include <opae/enum.h>\n#include <opae/event.h>\n#include <opae/manage.h>\n#include <opae/mmio.h>\n#include <opae/properties.h>\n#include <opae/umsg.h>\n#include <opae/utils.h>\n#include <opae/error.h>\n#include <opae/version.h>\n#include <opae/sysobject.h>\n#include <opae/userclk.h>\n#include <opae/metrics.h>\n#endif // __FPGA_FPGA_H__\n
"},{"location":"opae-code/hash__map_8h/","title":"File hash_map.h","text":"FileList > docs > sw > include > opae > hash_map.h
Go to the source code of this file.
A general-purpose hybrid array/list hash map implementation. More...
#include <stdint.h>#include <stdbool.h>#include <opae/types_enum.h>
"},{"location":"opae-code/hash__map_8h/#classes","title":"Classes","text":"Type Name struct _opae_hash_map Hash map object. struct _opae_hash_map_item List link item."},{"location":"opae-code/hash__map_8h/#public-types","title":"Public Types","text":"Type Name enum _opae_hash_map_flags Flags used to initialize a hash map. typedef struct _opae_hash_map opae_hash_map Hash map object. typedef enum _opae_hash_map_flags opae_hash_map_flags Flags used to initialize a hash map. typedef struct _opae_hash_map_item opae_hash_map_item List link item."},{"location":"opae-code/hash__map_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result opae_hash_map_add (opae_hash_map * hm, void * key, void * value) Map a key to a value. fpga_result opae_hash_map_destroy (opae_hash_map * hm) Tear down a hash map. fpga_result opae_hash_map_find (opae_hash_map * hm, void * key, void ** value) Retrieve the value for a given key. fpga_result opae_hash_map_init (opae_hash_map * hm, uint32_t num_buckets, uint32_t hash_seed, int flags, uint32_t(*)(uint32_t num_buckets, uint32_t hash_seed, void *key) key_hash, int(*)(void *keya, void *keyb) key_compare, void(*)(void *key, void *context) key_cleanup, void(*)(void *value, void *context) value_cleanup) Initialize a hash map. bool opae_hash_map_is_empty (opae_hash_map * hm) Determine whether a hash map is empty. fpga_result opae_hash_map_remove (opae_hash_map * hm, void * key) Remove a key/value association. int opae_u64_key_compare (void * keya, void * keyb) Convenience key comparison function for 64-bit values. uint32_t opae_u64_key_hash (uint32_t num_buckets, uint32_t hash_seed, void * key) Convenience hash function for arbitrary pointers/64-bit values."},{"location":"opae-code/hash__map_8h/#detailed-description","title":"Detailed Description","text":"Presents a generic interface for mapping key objects to value objects. Both keys and values may be arbitrary data structures. The user supplies the means by which the hash of values is generated and by which the keys are compared to each other.
"},{"location":"opae-code/hash__map_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/hash__map_8h/#enum-_opae_hash_map_flags","title":"enum _opae_hash_map_flags","text":"Flags used to initialize a hash map.
enum _opae_hash_map_flags {\nOPAE_HASH_MAP_UNIQUE_KEYSPACE = (1u << 0)\n};\n
OPAE_HASH_MAP_UNIQUE_KEYSPACE says that the user provides a guarantee that the key space is truly unique. In other words, when the provided hash function for keys A and B returns the same bucket index, the key comparison function when comparing A and B will never return a result saying that the keys are equal in value. This is helpful in situations where the key space is guaranteed to produce unique values, for example a memory allocator. When the key space is guaranteed to be unique, opae_hash_map_add() can implement a small performance improvement.
"},{"location":"opae-code/hash__map_8h/#typedef-opae_hash_map","title":"typedef opae_hash_map","text":"Hash map object.
typedef struct _opae_hash_map opae_hash_map;\n
This structure defines the internals of the hash map. Each of the parameters supplied to opae_hash_map_init() is stored in the structure. All parameters are required, except key_cleanup and value_cleanup, which may optionally be NULL.
"},{"location":"opae-code/hash__map_8h/#typedef-opae_hash_map_flags","title":"typedef opae_hash_map_flags","text":"Flags used to initialize a hash map.
typedef enum _opae_hash_map_flags opae_hash_map_flags;\n
OPAE_HASH_MAP_UNIQUE_KEYSPACE says that the user provides a guarantee that the key space is truly unique. In other words, when the provided hash function for keys A and B returns the same bucket index, the key comparison function when comparing A and B will never return a result saying that the keys are equal in value. This is helpful in situations where the key space is guaranteed to produce unique values, for example a memory allocator. When the key space is guaranteed to be unique, opae_hash_map_add() can implement a small performance improvement.
"},{"location":"opae-code/hash__map_8h/#typedef-opae_hash_map_item","title":"typedef opae_hash_map_item","text":"List link item.
typedef struct _opae_hash_map_item opae_hash_map_item;\n
This structure provides the association between key and value. When the supplied hash function for keys A and B returns the same bucket index, both A and B can co-exist on the same list rooted at the bucket index.
"},{"location":"opae-code/hash__map_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_add","title":"function opae_hash_map_add","text":"Map a key to a value.
fpga_result opae_hash_map_add (\nopae_hash_map * hm,\nvoid * key,\nvoid * value\n)
Inserts a mapping from key to value in the given hash map object. Subsequent calls to opae_hash_map_find() that are given the key will retrieve the value.
Parameters:
hm A pointer to the storage for the hash map object. key The hash map key. value The hash map value.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if hm is NULL, FPGA_NO_MEMORY if malloc() fails when allocating the list item, or FPGA_INVALID_PARAM if the key hash produced by key_hash is out of bounds.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_destroy","title":"function opae_hash_map_destroy","text":"Tear down a hash map.
fpga_result opae_hash_map_destroy (\nopae_hash_map * hm\n)
Given a hash map that was previously initialized by opae_hash_map_init(), destroy the hash map, releasing all keys, values, and the bucket array.
Parameters:
hm A pointer to the storage for the hash map object.
Returns:
FPGA_OK on success or FPGA_INVALID_PARAM is hm is NULL.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_find","title":"function opae_hash_map_find","text":"Retrieve the value for a given key.
fpga_result opae_hash_map_find (\nopae_hash_map * hm,\nvoid * key,\nvoid ** value\n)
Given a key that was previously passed to opae_hash_map_add(), retrieve its associated value.
Parameters:
hm A pointer to the storage for the hash map object. key The hash map key. value A pointer to receive the hash map value.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if hm is NULL or if the key hash produced by key_hash is out of bounds, or FPGA_NOT_FOUND if the given key was not found in the hash map.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_init","title":"function opae_hash_map_init","text":"Initialize a hash map.
fpga_result opae_hash_map_init (\nopae_hash_map * hm,\nuint32_t num_buckets,\nuint32_t hash_seed,\nint flags,\nuint32_t(*)(uint32_t num_buckets, uint32_t hash_seed, void *key) key_hash,\nint(*)(void *keya, void *keyb) key_compare,\nvoid(*)(void *key, void *context) key_cleanup,\nvoid(*)(void *value, void *context) value_cleanup\n)
Populates the hash map data structure and allocates the buckets array.
Parameters:
hm A pointer to the storage for the hash map object. num_buckets The desired size of the buckets array. Each array entry may be empty (NULL), or may contain a list of opae_hash_map_item structures for which the given key_hash function returned the same key hash value. hash_seed A seed value used during key hash computation. This value will be the hash_seed parameter to the key hash function. flags Initialization flags. See opae_hash_map_flags. key_hash A pointer to a function that produces the hash value, given the number of buckets, the hash seed, and the key. Valid values are between 0 and num_buckets - 1, inclusively. key_compare A pointer to a function that compares two keys. The return value is similar to that of strcmp(), where a negative value means that keya < keyb, 0 means that keya == keyb, and a positive values means that keya > keyb. key_cleanup A pointer to a function that is called when a key is being removed from the map. This function is optional and may be NULL. When supplied, the function is responsible for freeing any resources allocated when the key was created. value_cleanup A pointer to a function that is called when a value is being removed from the map. This function is optional and may be NULL. When supplied, the function is responsible for freeing any resources allocated when the value was created.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the required parameters are NULL, or FPGA_NO_MEMORY if the bucket array could not be allocated.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_is_empty","title":"function opae_hash_map_is_empty","text":"Determine whether a hash map is empty.
bool opae_hash_map_is_empty (\nopae_hash_map * hm\n)
Parameters:
hm A pointer to the storage for the hash map object.
Returns:
true if there are no key/value mappings present, false otherwise.
"},{"location":"opae-code/hash__map_8h/#function-opae_hash_map_remove","title":"function opae_hash_map_remove","text":"Remove a key/value association.
fpga_result opae_hash_map_remove (\nopae_hash_map * hm,\nvoid * key\n)
Given a key that was previously passed to opae_hash_map_add(), remove the key and its associated value, calling the cleanup functions as needed.
Parameters:
hm A pointer to the storage for the hash map object. key The hash map key.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM when hm is NULL or when the key hash produced by key_hash is out of bounds, or FPGA_NOT_FOUND if the key is not found in the hash map.
"},{"location":"opae-code/hash__map_8h/#function-opae_u64_key_compare","title":"function opae_u64_key_compare","text":"Convenience key comparison function for 64-bit values.
int opae_u64_key_compare (\nvoid * keya,\nvoid * keyb\n)
Simply converts the key pointers to uint64_t's and performs unsigned integer comparison.
"},{"location":"opae-code/hash__map_8h/#function-opae_u64_key_hash","title":"function opae_u64_key_hash","text":"Convenience hash function for arbitrary pointers/64-bit values.
uint32_t opae_u64_key_hash (\nuint32_t num_buckets,\nuint32_t hash_seed,\nvoid * key\n)
Simply converts the key to a uint64_t and then performs the modulus operation with the configured num_buckets. hash_seed is unused.
The documentation for this class was generated from the following file docs/sw/include/opae/hash_map.h
"},{"location":"opae-code/hash__map_8h_source/","title":"File hash_map.h","text":"File List > docs > sw > include > opae > hash_map.h
Go to the documentation of this file.
// Copyright(c) 2022-2023, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_HASH_MAP_H__\n#define __OPAE_HASH_MAP_H__\n#include <stdint.h>\n#include <stdbool.h>\n#include <opae/types_enum.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif // __cplusplus\ntypedef enum _opae_hash_map_flags {\nOPAE_HASH_MAP_UNIQUE_KEYSPACE = (1u << 0)\n} opae_hash_map_flags;\ntypedef struct _opae_hash_map_item {\nvoid *key;\nvoid *value;\nstruct _opae_hash_map_item *next;\n} opae_hash_map_item;\ntypedef struct _opae_hash_map {\nuint32_t num_buckets;\nuint32_t hash_seed;\nopae_hash_map_item **buckets;\nint flags;\nvoid *cleanup_context; uint32_t (*key_hash)(uint32_t num_buckets, uint32_t hash_seed,\nvoid *key);\nint (*key_compare)(void *keya, void *keyb); void (*key_cleanup)(void *key, void *context); void (*value_cleanup)(void *value, void *context); } opae_hash_map;\nfpga_result opae_hash_map_init(opae_hash_map *hm,\nuint32_t num_buckets,\nuint32_t hash_seed,\nint flags,\nuint32_t (*key_hash)(uint32_t num_buckets,\nuint32_t hash_seed,\nvoid *key),\nint (*key_compare)(void *keya, void *keyb),\nvoid (*key_cleanup)(void *key, void *context),\nvoid (*value_cleanup)(void *value, void *context));\nfpga_result opae_hash_map_add(opae_hash_map *hm,\nvoid *key,\nvoid *value);\nfpga_result opae_hash_map_find(opae_hash_map *hm,\nvoid *key,\nvoid **value);\nfpga_result opae_hash_map_remove(opae_hash_map *hm,\nvoid *key);\nfpga_result opae_hash_map_destroy(opae_hash_map *hm);\nbool opae_hash_map_is_empty(opae_hash_map *hm);\nuint32_t opae_u64_key_hash(uint32_t num_buckets,\nuint32_t hash_seed,\nvoid *key);\nint opae_u64_key_compare(void *keya, void *keyb);\n#ifdef __cplusplus\n}\n#endif // __cplusplus\n#endif // __OPAE_HASH_MAP_H__\n
"},{"location":"opae-code/init_8h/","title":"File init.h","text":"FileList > docs > sw > include > opae > init.h
Go to the source code of this file.
Initialization routine.
#include <opae/types_enum.h>
"},{"location":"opae-code/init_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaFinalize (void) Finalize the OPAE library. fpga_result fpgaInitialize (const char * config_file) Initialize the OPAE library."},{"location":"opae-code/init_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/init_8h/#function-fpgafinalize","title":"function fpgaFinalize","text":"Finalize the OPAE library.
fpga_result fpgaFinalize (\nvoid\n)
Returns:
Whether OPAE finalized successfully.
"},{"location":"opae-code/init_8h/#function-fpgainitialize","title":"function fpgaInitialize","text":"Initialize the OPAE library.
fpga_result fpgaInitialize (\nconst char * config_file\n)
Initialize OPAE using the given configuration file path, or perform default initialization if config_file is NULL.
Parameters:
config_file Path to OPAE configuration file.
Returns:
Whether OPAE initialized successfully.
The documentation for this class was generated from the following file docs/sw/include/opae/init.h
"},{"location":"opae-code/init_8h_source/","title":"File init.h","text":"File List > docs > sw > include > opae > init.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_INIT_H__\n#define __FPGA_INIT_H__\n#include <opae/types_enum.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaInitialize(const char *config_file);\nfpga_result fpgaFinalize(void);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_INIT_H__\n
"},{"location":"opae-code/log_8h/","title":"File log.h","text":"FileList > docs > sw > include > opae > log.h
Go to the source code of this file.
#include <stdint.h>#include <stdlib.h>#include <string.h>#include <errno.h>#include <opae/types.h>
"},{"location":"opae-code/log_8h/#public-types","title":"Public Types","text":"Type Name enum opae_loglevel"},{"location":"opae-code/log_8h/#public-functions","title":"Public Functions","text":"Type Name void opae_print (int loglevel, const char * fmt, ...)"},{"location":"opae-code/log_8h/#macros","title":"Macros","text":"Type Name define OPAE_DBG (format, ...) { } define OPAE_DEFAULT_LOGLEVEL OPAE_LOG_ERROR define OPAE_ERR (format, ...) define OPAE_MSG (format, ...) define __SHORT_FILE__"},{"location":"opae-code/log_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/log_8h/#enum-opae_loglevel","title":"enum opae_loglevel","text":"enum opae_loglevel {\nOPAE_LOG_ERROR = 0,\nOPAE_LOG_MESSAGE,\nOPAE_LOG_DEBUG\n};\n
"},{"location":"opae-code/log_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/log_8h/#function-opae_print","title":"function opae_print","text":"void opae_print (\nint loglevel,\nconst char * fmt,\n...\n)
"},{"location":"opae-code/log_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/log_8h/#define-opae_dbg","title":"define OPAE_DBG","text":"#define OPAE_DBG (\nformat,\n...\n) { }\n
"},{"location":"opae-code/log_8h/#define-opae_default_loglevel","title":"define OPAE_DEFAULT_LOGLEVEL","text":"#define OPAE_DEFAULT_LOGLEVEL OPAE_LOG_ERROR\n
"},{"location":"opae-code/log_8h/#define-opae_err","title":"define OPAE_ERR","text":"#define OPAE_ERR (\nformat,\n...\n) opae_print ( OPAE_LOG_ERROR , \\\n \"%s:%u:%s() **ERROR** : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n
"},{"location":"opae-code/log_8h/#define-opae_msg","title":"define OPAE_MSG","text":"#define OPAE_MSG (\nformat,\n...\n) opae_print ( OPAE_LOG_MESSAGE , \"%s:%u:%s() : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n
"},{"location":"opae-code/log_8h/#define-__short_file__","title":"define __SHORT_FILE__","text":"#define __SHORT_FILE__ ({ \\\n const char *file = __FILE__; \\\n const char *p = file; \\\n while (*p) \\\n ++p; \\\n while ((p > file) && ('/' != *p) && ('\\\\' != *p)) \\\n --p; \\\n if (p > file) \\\n ++p; \\\n p; \\\n })\n
The documentation for this class was generated from the following file docs/sw/include/opae/log.h
"},{"location":"opae-code/log_8h_source/","title":"File log.h","text":"File List > docs > sw > include > opae > log.h
Go to the documentation of this file.
// Copyright(c) 2018-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_LOG_H__\n#define __OPAE_LOG_H__\n#include <stdint.h>\n#include <stdlib.h>\n#include <string.h>\n#include <errno.h>\n#include <opae/types.h>\n/*\n* Convenience macros for printing messages and errors.\n*/\n#ifdef __SHORT_FILE__\n#undef __SHORT_FILE__\n#endif // __SHORT_FILE__\n#define __SHORT_FILE__ \\\n ({ \\\n const char *file = __FILE__; \\\n const char *p = file; \\\n while (*p) \\\n ++p; \\\n while ((p > file) && ('/' != *p) && ('\\\\' != *p)) \\\n --p; \\\n if (p > file) \\\n ++p; \\\n p; \\\n })\n#ifdef OPAE_MSG\n#undef OPAE_MSG\n#endif // OPAE_MSG\n#define OPAE_MSG(format, ...) \\\n opae_print(OPAE_LOG_MESSAGE, \"%s:%u:%s() : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n#ifdef OPAE_ERR\n#undef OPAE_ERR\n#endif // OPAE_ERR\n#define OPAE_ERR(format, ...) \\\n opae_print(OPAE_LOG_ERROR, \\\n \"%s:%u:%s() **ERROR** : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n#ifdef OPAE_DBG\n#undef OPAE_DBG\n#endif // OPAE_DBG\n#ifdef LIBOPAE_DEBUG\n#define OPAE_DBG(format, ...) \\\n opae_print(OPAE_LOG_DEBUG, \\\n \"%s:%u:%s() *DEBUG* : \" format \"\\n\", \\\n __SHORT_FILE__, __LINE__, __func__, ##__VA_ARGS__)\n#else\n#define OPAE_DBG(format, ...) \\\n{ }\n#endif // LIBOPAE_DEBUG\n/*\n* Logging functions\n*/\nenum opae_loglevel {\nOPAE_LOG_ERROR = 0, /* critical errors (always print) */\nOPAE_LOG_MESSAGE, /* information (i.e. explain return code */\nOPAE_LOG_DEBUG /* debugging (also needs #define DEBUG 1) */\n};\n#define OPAE_DEFAULT_LOGLEVEL OPAE_LOG_ERROR\n#ifdef __cplusplus\nextern \"C\" {\n#endif // __cplusplus\nvoid opae_print(int loglevel, const char *fmt, ...);\n#ifdef __cplusplus\n}\n#endif // __cplusplus\n#endif // __OPAE_LOG_H__\n
"},{"location":"opae-code/manage_8h/","title":"File manage.h","text":"FileList > docs > sw > include > opae > manage.h
Go to the source code of this file.
Functions for managing FPGA configurations. More...
#include <opae/types.h>
"},{"location":"opae-code/manage_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaAssignPortToInterface (fpga_handle fpga, uint32_t interface_num, uint32_t slot_num, int flags) Assign Port to a host interface. fpga_result fpgaAssignToInterface (fpga_handle fpga, fpga_token accelerator, uint32_t host_interface, int flags) Assign an accelerator to a host interface. fpga_result fpgaReconfigureSlot (fpga_handle fpga, uint32_t slot, const uint8_t * bitstream, size_t bitstream_len, int flags) Reconfigure a slot. fpga_result fpgaReleaseFromInterface (fpga_handle fpga, fpga_token accelerator) Unassign a previously assigned accelerator."},{"location":"opae-code/manage_8h/#detailed-description","title":"Detailed Description","text":"FPGA accelerators can be reprogrammed at run time by providing new partial bitstreams (\"green bitstreams\"). This file defines API functions for programming green bitstreams as well as for assigning accelerators to host interfaces for more complex deployment setups, such as virtualized systems.
"},{"location":"opae-code/manage_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/manage_8h/#function-fpgaassignporttointerface","title":"function fpgaAssignPortToInterface","text":"Assign Port to a host interface.
fpga_result fpgaAssignPortToInterface (\nfpga_handle fpga,\nuint32_t interface_num,\nuint32_t slot_num,\nint flags\n)
This function assign Port to a host interface for subsequent use. Only Port that have been assigned to a host interface can be opened by fpgaOpen().
Parameters:
fpga Handle to an FPGA object previously opened that both the host interface and the slot belong to interface_num Host interface number slot_num Slot number flags Flags (to be defined)
Returns:
FPGA_OK on success FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if an exception occcurred accessing the fpga handle. FPGA_NOT_SUPPORTED if driver does not support assignment.
"},{"location":"opae-code/manage_8h/#function-fpgaassigntointerface","title":"function fpgaAssignToInterface","text":"Assign an accelerator to a host interface.
fpga_result fpgaAssignToInterface (\nfpga_handle fpga,\nfpga_token accelerator,\nuint32_t host_interface,\nint flags\n)
This function assigns an accelerator to a host interface for subsequent use. Only accelerators that have been assigned to a host interface can be opened by fpgaOpen().
Note:
This function is currently not supported.
Parameters:
fpga Handle to an FPGA object previously opened that both the host interface and the accelerator belong to accelerator accelerator to assign host_interface Host interface to assign accelerator to flags Flags (to be defined)
Returns:
FPGA_OK on success
"},{"location":"opae-code/manage_8h/#function-fpgareconfigureslot","title":"function fpgaReconfigureSlot","text":"Reconfigure a slot.
fpga_result fpgaReconfigureSlot (\nfpga_handle fpga,\nuint32_t slot,\nconst uint8_t * bitstream,\nsize_t bitstream_len,\nint flags\n)
Sends a green bitstream file to an FPGA to reconfigure a specific slot. This call, if successful, will overwrite the currently programmed AFU in that slot with the AFU in the provided bitstream.
As part of the reconfiguration flow, all accelerators associated with this slot will be unassigned and reset.
Parameters:
fpga Handle to an FPGA object previously opened slot Token identifying the slot to reconfigure bitstream Pointer to memory holding the bitstream bitstream_len Length of the bitstream in bytes flags Flags that control behavior of reconfiguration. Value of 0 indicates no flags. FPGA_RECONF_FORCE indicates that the bitstream is programmed into the slot without checking if the resource is currently in use.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if the provided parameters are not valid. FPGA_EXCEPTION if an internal error occurred accessing the handle or while sending the bitstream data to the driver. FPGA_BUSY if the accelerator for the given slot is in use. FPGA_RECONF_ERROR on errors reported by the driver (such as CRC or protocol errors).
Note:
By default, fpgaReconfigureSlot will not allow reconfiguring a slot with an accelerator in use. Add the flag FPGA_RECONF_FORCE to force reconfiguration without checking for accelerators in use.
"},{"location":"opae-code/manage_8h/#function-fpgareleasefrominterface","title":"function fpgaReleaseFromInterface","text":"Unassign a previously assigned accelerator.
fpga_result fpgaReleaseFromInterface (\nfpga_handle fpga,\nfpga_token accelerator\n)
This function removes the assignment of an accelerator to an host interface (e.g. to be later assigned to a different host interface). As a consequence, the accelerator referred to by token 'accelerator' will be reset during the course of this function.
Note:
This function is currently not supported.
Parameters:
fpga Handle to an FPGA object previously opened that both the host interface and the accelerator belong to accelerator accelerator to unassign/release
Returns:
FPGA_OK on success
The documentation for this class was generated from the following file docs/sw/include/opae/manage.h
"},{"location":"opae-code/manage_8h_source/","title":"File manage.h","text":"File List > docs > sw > include > opae > manage.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_MANAGE_H__\n#define __FPGA_MANAGE_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaAssignPortToInterface(fpga_handle fpga,\nuint32_t interface_num,\nuint32_t slot_num,\nint flags);\nfpga_result fpgaAssignToInterface(fpga_handle fpga,\nfpga_token accelerator,\nuint32_t host_interface,\nint flags);\nfpga_result fpgaReleaseFromInterface(fpga_handle fpga,\nfpga_token accelerator);\nfpga_result fpgaReconfigureSlot(fpga_handle fpga,\nuint32_t slot,\nconst uint8_t *bitstream,\nsize_t bitstream_len, int flags);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_MANAGE_H__\n
"},{"location":"opae-code/mem__alloc_8h/","title":"File mem_alloc.h","text":"FileList > docs > sw > include > opae > mem_alloc.h
Go to the source code of this file.
#include <stdint.h>
"},{"location":"opae-code/mem__alloc_8h/#classes","title":"Classes","text":"Type Name struct mem_alloc struct mem_link Provides an API for allocating/freeing a logical address space."},{"location":"opae-code/mem__alloc_8h/#public-functions","title":"Public Functions","text":"Type Name int mem_alloc_add_free (struct mem_alloc * m, uint64_t address, uint64_t size) Add a memory region to an allocator. void mem_alloc_destroy (struct mem_alloc * m) Destroy a memory allocator object. int mem_alloc_get (struct mem_alloc * m, uint64_t * address, uint64_t size) Allocate memory. void mem_alloc_init (struct mem_alloc * m) Initialize a memory allocator object. int mem_alloc_put (struct mem_alloc * m, uint64_t address) Free memory."},{"location":"opae-code/mem__alloc_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_add_free","title":"function mem_alloc_add_free","text":"Add a memory region to an allocator.
int mem_alloc_add_free (\nstruct mem_alloc * m,\nuint64_t address,\nuint64_t size\n)
The memory region is added to the allocatable space and is immediately ready for allocation.
Parameters:
m The memory allocator object. address The beginning address of the memory region. size The size of the memory region.
Returns:
Non-zero on error. Zero on success.
Example struct mem_alloc m;
mem_alloc_init(&m);
if (mem_alloc_add_free(&m, 0x4000, 4096)) { // handle error }
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_destroy","title":"function mem_alloc_destroy","text":"Destroy a memory allocator object.
void mem_alloc_destroy (\nstruct mem_alloc * m\n)
Frees all of the allocator's internal resources.
Parameters:
m The address of the memory allocator to destroy.
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_get","title":"function mem_alloc_get","text":"Allocate memory.
int mem_alloc_get (\nstruct mem_alloc * m,\nuint64_t * address,\nuint64_t size\n)
Retrieve an available memory address for a free block that is at least size bytes.
Parameters:
m The memory allocator object. address The retrieved address for the allocation. size The request size in bytes.
Returns:
Non-zero on error. Zero on success.
Example struct mem_alloc m; uint64_t addr = 0;
mem_alloc_init(&m);
if (mem_alloc_add_free(&m, 0x4000, 4096)) { // handle error }
...
if (mem_alloc_get(&m, &addr, 4096)) { // handle allocation error }
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_init","title":"function mem_alloc_init","text":"Initialize a memory allocator object.
void mem_alloc_init (\nstruct mem_alloc * m\n)
After the call, the allocator is initialized but \"empty\". To add allocatable memory regions, further initialize the allocator with mem_alloc_add_free().
Parameters:
m The address of the memory allocator to initialize.
"},{"location":"opae-code/mem__alloc_8h/#function-mem_alloc_put","title":"function mem_alloc_put","text":"Free memory.
int mem_alloc_put (\nstruct mem_alloc * m,\nuint64_t address\n)
Release a previously-allocated memory block.
Parameters:
m The memory allocator object. address The address to free.
Returns:
Non-zero on error. Zero on success.
Example struct mem_alloc m; uint64_t addr = 0;
mem_alloc_init(&m);
if (mem_alloc_add_free(&m, 0x4000, 4096)) { // handle error }
...
if (mem_alloc_get(&m, &addr, 4096)) { // handle allocation error }
...
if (mem_alloc_put(&m, addr)) { // handle free error }
The documentation for this class was generated from the following file docs/sw/include/opae/mem_alloc.h
"},{"location":"opae-code/mem__alloc_8h_source/","title":"File mem_alloc.h","text":"File List > docs > sw > include > opae > mem_alloc.h
Go to the documentation of this file.
// Copyright(c) 2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_MEM_ALLOC_H__\n#define __OPAE_MEM_ALLOC_H__\n#include <stdint.h>\nstruct mem_link {\nuint64_t address;\nuint64_t size;\nstruct mem_link *prev;\nstruct mem_link *next;\n};\nstruct mem_alloc {\nstruct mem_link free;\nstruct mem_link allocated;\n};\n#ifdef __cplusplus\nextern \"C\" {\n#endif // __cplusplus\nvoid mem_alloc_init(struct mem_alloc *m);\nvoid mem_alloc_destroy(struct mem_alloc *m);\nint mem_alloc_add_free(struct mem_alloc *m,\nuint64_t address,\nuint64_t size);\nint mem_alloc_get(struct mem_alloc *m,\nuint64_t *address,\nuint64_t size);\nint mem_alloc_put(struct mem_alloc *m,\nuint64_t address);\n#ifdef __cplusplus\n}\n#endif // __cplusplus\n#endif // __OPAE_MEM_ALLOC_H__\n
"},{"location":"opae-code/metrics_8h/","title":"File metrics.h","text":"FileList > docs > sw > include > opae > metrics.h
Go to the source code of this file.
Functions for Discover/ Enumerates metrics and retrieves values.
#include <opae/types.h>
"},{"location":"opae-code/metrics_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetMetricsByIndex (fpga_handle handle, uint64_t * metric_num, uint64_t num_metric_indexes, fpga_metric * metrics) Retrieve metrics values by index. fpga_result fpgaGetMetricsByName (fpga_handle handle, char ** metrics_names, uint64_t num_metric_names, fpga_metric * metrics) Retrieve metric values by names. fpga_result fpgaGetMetricsInfo (fpga_handle handle, fpga_metric_info * metric_info, uint64_t * num_metrics) Retrieve metrics information. fpga_result fpgaGetMetricsThresholdInfo (fpga_handle handle, struct metric_threshold * metric_thresholds, uint32_t * num_thresholds) Retrieve metrics / sendor threshold information and values. fpga_result fpgaGetNumMetrics (fpga_handle handle, uint64_t * num_metrics) Enumerates number of metrics."},{"location":"opae-code/metrics_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsbyindex","title":"function fpgaGetMetricsByIndex","text":"Retrieve metrics values by index.
fpga_result fpgaGetMetricsByIndex (\nfpga_handle handle,\nuint64_t * metric_num,\nuint64_t num_metric_indexes,\nfpga_metric * metrics\n)
Parameters:
handle Handle to previously opened fpga resource metric_num Pointer to array of metric index user allocates metric array num_metric_indexes Size of metric array metrics pointer to array of metric struct
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found. FPGA_NO_MEMORY if there was not enough memory to enumerates metrics.
"},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsbyname","title":"function fpgaGetMetricsByName","text":"Retrieve metric values by names.
fpga_result fpgaGetMetricsByName (\nfpga_handle handle,\nchar ** metrics_names,\nuint64_t num_metric_names,\nfpga_metric * metrics\n)
Parameters:
handle Handle to previously opened fpga resource metrics_names Pointer to array of metrics name user allocates metrics name array num_metric_names Size of metric name array metrics Pointer to array of metric struct
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found
"},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsinfo","title":"function fpgaGetMetricsInfo","text":"Retrieve metrics information.
fpga_result fpgaGetMetricsInfo (\nfpga_handle handle,\nfpga_metric_info * metric_info,\nuint64_t * num_metrics\n)
Parameters:
handle Handle to previously opened fpga resource metric_info Pointer to array of metric info struct user allocates metrics info arraynum_metrics Size of metric info array
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found. FPGA_NO_MEMORY if there was not enough memory to enumerates metrics.
"},{"location":"opae-code/metrics_8h/#function-fpgagetmetricsthresholdinfo","title":"function fpgaGetMetricsThresholdInfo","text":"Retrieve metrics / sendor threshold information and values.
fpga_result fpgaGetMetricsThresholdInfo (\nfpga_handle handle,\nstruct metric_threshold * metric_thresholds,\nuint32_t * num_thresholds\n)
Parameters:
handle Handle to previously opened fpga resource metrics_threshold pointer to array of metric thresholds user allocates threshold array memory Number of thresholds returns enumerated thresholds if user pass NULL metrics_thresholds num_thresholds number of thresholds
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not found. FPGA_NO_MEMORY if there was not enough memory to enumerates metrics.
"},{"location":"opae-code/metrics_8h/#function-fpgagetnummetrics","title":"function fpgaGetNumMetrics","text":"Enumerates number of metrics.
fpga_result fpgaGetNumMetrics (\nfpga_handle handle,\nuint64_t * num_metrics\n)
Parameters:
handle Handle to previously opened fpga resource num_metrics Number of metrics are discovered in fpga resource
Returns:
FPGA_OK on success. FPGA_NOT_FOUND if the Metrics are not discovered
The documentation for this class was generated from the following file docs/sw/include/opae/metrics.h
"},{"location":"opae-code/metrics_8h_source/","title":"File metrics.h","text":"File List > docs > sw > include > opae > metrics.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_METRICS_H__\n#define __FPGA_METRICS_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetNumMetrics(fpga_handle handle,\nuint64_t *num_metrics);\nfpga_result fpgaGetMetricsInfo(fpga_handle handle,\nfpga_metric_info *metric_info,\nuint64_t *num_metrics);\nfpga_result fpgaGetMetricsByIndex(fpga_handle handle,\nuint64_t *metric_num,\nuint64_t num_metric_indexes,\nfpga_metric *metrics);\nfpga_result fpgaGetMetricsByName(fpga_handle handle,\nchar **metrics_names,\nuint64_t num_metric_names,\nfpga_metric *metrics);\nfpga_result fpgaGetMetricsThresholdInfo(fpga_handle handle,\nstruct metric_threshold *metric_thresholds,\nuint32_t *num_thresholds);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_METRICS_H__\n
"},{"location":"opae-code/mmio_8h/","title":"File mmio.h","text":"FileList > docs > sw > include > opae > mmio.h
Go to the source code of this file.
Functions for mapping and accessing MMIO space. More...
#include <opae/types.h>
"},{"location":"opae-code/mmio_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaMapMMIO (fpga_handle handle, uint32_t mmio_num, uint64_t ** mmio_ptr) Map MMIO space. fpga_result fpgaReadMMIO32 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint32_t * value) Read 32 bit value from MMIO space. fpga_result fpgaReadMMIO64 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint64_t * value) Read 64 bit value from MMIO space. fpga_result fpgaUnmapMMIO (fpga_handle handle, uint32_t mmio_num) Unmap MMIO space. fpga_result fpgaWriteMMIO32 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint32_t value) Write 32 bit value to MMIO space. fpga_result fpgaWriteMMIO512 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, const void * value) Write 512 bit value to MMIO space. fpga_result fpgaWriteMMIO64 (fpga_handle handle, uint32_t mmio_num, uint64_t offset, uint64_t value) Write 64 bit value to MMIO space."},{"location":"opae-code/mmio_8h/#detailed-description","title":"Detailed Description","text":"Most FPGA accelerators provide access to control registers through memory-mappable address spaces, commonly referred to as \"MMIO spaces\". This file provides functions to map, unmap, read, and write MMIO spaces.
Note that an accelerator may have multiple MMIO spaces, denoted by the mmio_num argument of the APIs below. The meaning and properties of each MMIO space are up to the accelerator designer.
"},{"location":"opae-code/mmio_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/mmio_8h/#function-fpgamapmmio","title":"function fpgaMapMMIO","text":"Map MMIO space.
fpga_result fpgaMapMMIO (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t ** mmio_ptr\n)
This function will return a pointer to the specified MMIO space of the target object in process virtual memory, if supported by the target. Some MMIO spaces may be restricted to privileged processes, depending on the used handle and type.
After mapping the respective MMIO space, you can access it through direct pointer operations (observing supported access sizes and alignments of the target platform and accelerator).
Note:
Some targets (such as the ASE simulator) do not support memory-mapping of IO register spaces and will not return a pointer to an actually mapped space. Instead, they will return FPGA_NOT_SUPPORTED. Usually, these platforms still allow the application to issue MMIO operations using fpgaReadMMIO32(), fpgaWriteMMIO32(), fpgeReadMMIO64(), and fpgaWriteMMIO64().
If the caller passes in NULL for mmio_ptr, no mapping will be performed, and no virtual address will be returned, though the call will return FPGA_OK. This implies that all accesses will be performed through fpgaReadMMIO32(), fpgaWriteMMIO32(), fpgeReadMMIO64(), and fpgaWriteMMIO64(). This is the only supported case for ASE.
The number of available MMIO spaces can be retrieved through the num_mmio property (fpgaPropertyGetNumMMIO()).
Parameters:
handle Handle to previously opened resource mmio_num Number of MMIO space to access mmio_ptr Pointer to memory where a pointer to the MMIO space will be returned. May be NULL, in which case no pointer is returned. Returned address may be NULL if underlying platform does not support memory mapping for register access.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle. FPGA_NO_ACCESS if the process' permissions are not sufficient to map the requested MMIO space. FPGA_NOT_SUPPORTED if platform does not support memory mapped IO.
"},{"location":"opae-code/mmio_8h/#function-fpgareadmmio32","title":"function fpgaReadMMIO32","text":"Read 32 bit value from MMIO space.
fpga_result fpgaReadMMIO32 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint32_t * value\n)
This function will read from MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Pointer to memory where read value is returned (32 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgareadmmio64","title":"function fpgaReadMMIO64","text":"Read 64 bit value from MMIO space.
fpga_result fpgaReadMMIO64 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint64_t * value\n)
This function will read from MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Pointer to memory where read value is returned (64 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgaunmapmmio","title":"function fpgaUnmapMMIO","text":"Unmap MMIO space.
fpga_result fpgaUnmapMMIO (\nfpga_handle handle,\nuint32_t mmio_num\n)
This function will unmap a previously mapped MMIO space of the target object, rendering any pointers to it invalid.
Note:
This call is only supported by hardware targets, not by ASE simulation.
Parameters:
handle Handle to previously opened resource mmio_num Number of MMIO space to access
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgawritemmio32","title":"function fpgaWriteMMIO32","text":"Write 32 bit value to MMIO space.
fpga_result fpgaWriteMMIO32 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint32_t value\n)
This function will write to MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Value to write (32 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgawritemmio512","title":"function fpgaWriteMMIO512","text":"Write 512 bit value to MMIO space.
fpga_result fpgaWriteMMIO512 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nconst void * value\n)
512 bit MMIO writes may not be supported on all platforms.
This function will write to MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Pointer to memory holding value to write (512 bits)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/mmio_8h/#function-fpgawritemmio64","title":"function fpgaWriteMMIO64","text":"Write 64 bit value to MMIO space.
fpga_result fpgaWriteMMIO64 (\nfpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset,\nuint64_t value\n)
This function will write to MMIO space of the target object at a specified offset.
Parameters:
handle Handle to previously opened accelerator resource mmio_num Number of MMIO space to access offset Byte offset into MMIO space value Value to write (64 bit)
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
The documentation for this class was generated from the following file docs/sw/include/opae/mmio.h
"},{"location":"opae-code/mmio_8h_source/","title":"File mmio.h","text":"File List > docs > sw > include > opae > mmio.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_MMIO_H__\n#define __FPGA_MMIO_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaWriteMMIO64(fpga_handle handle,\nuint32_t mmio_num, uint64_t offset,\nuint64_t value);\nfpga_result fpgaReadMMIO64(fpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset, uint64_t *value);\nfpga_result fpgaWriteMMIO32(fpga_handle handle,\nuint32_t mmio_num, uint64_t offset,\nuint32_t value);\nfpga_result fpgaReadMMIO32(fpga_handle handle,\nuint32_t mmio_num,\nuint64_t offset, uint32_t *value);\nfpga_result fpgaWriteMMIO512(fpga_handle handle,\nuint32_t mmio_num, uint64_t offset,\nconst void *value);\nfpga_result fpgaMapMMIO(fpga_handle handle,\nuint32_t mmio_num, uint64_t **mmio_ptr);\nfpga_result fpgaUnmapMMIO(fpga_handle handle,\nuint32_t mmio_num);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_MMIO_H__\n
"},{"location":"opae-code/properties_8h/","title":"File properties.h","text":"FileList > docs > sw > include > opae > properties.h
Go to the source code of this file.
Functions for examining and manipulating fpga_properties objects.More...
#include <opae/types.h>
"},{"location":"opae-code/properties_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaClearProperties (fpga_properties prop) Clear a fpga_properties object. fpga_result fpgaCloneProperties (fpga_properties src, fpga_properties * dst) Clone a fpga_properties object. fpga_result fpgaDestroyProperties (fpga_properties * prop) Destroy a fpga_properties object. fpga_result fpgaGetProperties (fpga_token token, fpga_properties * prop) Create a fpga_properties object. fpga_result fpgaGetPropertiesFromHandle (fpga_handle handle, fpga_properties * prop) Create a fpga_properties object. fpga_result fpgaPropertiesGetAcceleratorState (const fpga_properties prop, fpga_accelerator_state * state) Get the state of a accelerator resource property. fpga_result fpgaPropertiesGetBBSID (const fpga_properties prop, uint64_t * bbs_id) Get the BBS ID of an FPGA resource property. fpga_result fpgaPropertiesGetBBSVersion (const fpga_properties prop, fpga_version * bbs_version) Get the BBS Version of an FPGA resource property. fpga_result fpgaPropertiesGetBus (const fpga_properties prop, uint8_t * bus) Get the PCI bus number of a resource. fpga_result fpgaPropertiesGetCapabilities (const fpga_properties prop, uint64_t * capabilities) Get the capabilities FPGA resource property. fpga_result fpgaPropertiesGetDevice (const fpga_properties prop, uint8_t * device) Get the PCI device number of a resource. fpga_result fpgaPropertiesGetDeviceID (const fpga_properties prop, uint16_t * device_id) Get the device id of the resource. fpga_result fpgaPropertiesGetFunction (const fpga_properties prop, uint8_t * function) Get the PCI function number of a resource. fpga_result fpgaPropertiesGetGUID (const fpga_properties prop, fpga_guid * guid) Get the GUID of a resource. fpga_result fpgaPropertiesGetInterface (const fpga_properties prop, fpga_interface * interface) Get the OPAE plugin interface implemented by a resource. fpga_result fpgaPropertiesGetLocalMemorySize (const fpga_properties prop, uint64_t * lms) Get the local memory size of an FPGA resource property. fpga_result fpgaPropertiesGetModel (const fpga_properties prop, char * model) Get the model of an FPGA resource property. fpga_result fpgaPropertiesGetNumErrors (const fpga_properties prop, uint32_t * num_errors) Get the number of errors that can be reported by a resource. fpga_result fpgaPropertiesGetNumInterrupts (const fpga_properties prop, uint32_t * num_interrupts) Get the number of interrupts. fpga_result fpgaPropertiesGetNumMMIO (const fpga_properties prop, uint32_t * mmio_spaces) Get the number of mmio spaces. fpga_result fpgaPropertiesGetNumSlots (const fpga_properties prop, uint32_t * num_slots) Get the number of slots of an FPGA resource property. fpga_result fpgaPropertiesGetObjectID (const fpga_properties prop, uint64_t * object_id) Get the object ID of a resource. fpga_result fpgaPropertiesGetObjectType (const fpga_properties prop, fpga_objtype * objtype) Get the object type of a resource. fpga_result fpgaPropertiesGetParent (const fpga_properties prop, fpga_token * parent) Get the token of the parent object. fpga_result fpgaPropertiesGetSegment (const fpga_properties prop, uint16_t * segment) Get the PCI segment number of a resource. fpga_result fpgaPropertiesGetSocketID (const fpga_properties prop, uint8_t * socket_id) Get the socket id of a resource. fpga_result fpgaPropertiesGetSubsystemDeviceID (const fpga_properties prop, uint16_t * subsystem_device_id) Get the subsystem device id of an FPGA resource property. fpga_result fpgaPropertiesGetSubsystemVendorID (const fpga_properties prop, uint16_t * subsystem_vendor_id) Get the subsystem vendor id of an FPGA resource property. fpga_result fpgaPropertiesGetVendorID (const fpga_properties prop, uint16_t * vendor_id) Get the vendor id of an FPGA resource property. fpga_result fpgaPropertiesSetAcceleratorState (fpga_properties prop, fpga_accelerator_state state) Set the state of an accelerator resource property. fpga_result fpgaPropertiesSetBBSID (fpga_properties prop, uint64_t bbs_id) Set the BBS ID of an FPGA resource property. fpga_result fpgaPropertiesSetBBSVersion (fpga_properties prop, fpga_version version) Set the BBS Version of an FPGA resource property. fpga_result fpgaPropertiesSetBus (fpga_properties prop, uint8_t bus) Set the PCI bus number of a resource. fpga_result fpgaPropertiesSetCapabilities (fpga_properties prop, uint64_t capabilities) Set the capabilities of an FPGA resource property. fpga_result fpgaPropertiesSetDevice (fpga_properties prop, uint8_t device) Set the PCI device number of a resource. fpga_result fpgaPropertiesSetDeviceID (fpga_properties prop, uint16_t device_id) Set the device id of the resource. fpga_result fpgaPropertiesSetFunction (fpga_properties prop, uint8_t function) Set the PCI function number of a resource. fpga_result fpgaPropertiesSetGUID (fpga_properties prop, fpga_guid guid) Set the GUID of a resource. fpga_result fpgaPropertiesSetInterface (const fpga_properties prop, fpga_interface interface) Set the OPAE plugin interface implemented by a resource. fpga_result fpgaPropertiesSetLocalMemorySize (fpga_properties prop, uint64_t lms) Set the local memory size of an FPGA resource property. fpga_result fpgaPropertiesSetModel (fpga_properties prop, char * model) Set the model of an FPGA resource property. fpga_result fpgaPropertiesSetNumErrors (const fpga_properties prop, uint32_t num_errors) Set the number of error registers. fpga_result fpgaPropertiesSetNumInterrupts (fpga_properties prop, uint32_t num_interrupts) Set the number of interrupts. fpga_result fpgaPropertiesSetNumMMIO (fpga_properties prop, uint32_t mmio_spaces) Set the number of mmio spaces. fpga_result fpgaPropertiesSetNumSlots (fpga_properties prop, uint32_t num_slots) Set the number of slots of an FPGA resource property. fpga_result fpgaPropertiesSetObjectID (const fpga_properties prop, uint64_t object_id) Set the object ID of a resource. fpga_result fpgaPropertiesSetObjectType (fpga_properties prop, fpga_objtype objtype) Set the object type of a resource. fpga_result fpgaPropertiesSetParent (fpga_properties prop, fpga_token parent) Set the token of the parent object. fpga_result fpgaPropertiesSetSegment (fpga_properties prop, uint16_t segment) Set the PCI segment number of a resource. fpga_result fpgaPropertiesSetSocketID (fpga_properties prop, uint8_t socket_id) Set the socket id of the resource. fpga_result fpgaPropertiesSetSubsystemDeviceID (fpga_properties prop, uint16_t subsystem_device_id) Set the subsystem device id of an FPGA resource property. fpga_result fpgaPropertiesSetSubsystemVendorID (fpga_properties prop, uint16_t subsystem_vendor_id) Set the subsystem vendor id of an FPGA resource property. fpga_result fpgaPropertiesSetVendorID (fpga_properties prop, uint16_t vendor_id) Set the vendor id of an FPGA resource property. fpga_result fpgaUpdateProperties (fpga_token token, fpga_properties prop) Update a fpga_properties object."},{"location":"opae-code/properties_8h/#detailed-description","title":"Detailed Description","text":"In OPAE, fpga_properties objects are used both for obtaining information about resources and for selectively enumerating resources based on their properties. This file provides accessor functions (get/set) to allow reading and writing individual items of an fpga_properties object. Generally, not all object types supported by OPAE carry all properties. If you call a property accessor method on a fpga_properties object that does not support this particular property, it will return FPGA_INVALID_PARAM.
"},{"location":"opae-code/properties_8h/#accessor-return-values","title":"Accessor Return Values","text":"In addition to the return values specified in the documentation below, all accessor functions return FPGA_OK on success, FPGA_INVALID_PARAM if you pass NULL or invalid parameters (i.e. non-initialized properties objects), FPGA_EXCEPTION if an internal exception occurred trying to access the properties object, FPGA_NOT_FOUND if the requested property is not part of the supplied properties object.
"},{"location":"opae-code/properties_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/properties_8h/#function-fpgaclearproperties","title":"function fpgaClearProperties","text":"Clear a fpga_properties object.
fpga_result fpgaClearProperties (\nfpga_properties prop\n)
Sets all fields of the properties object pointed at by 'prop' to 'don't care', which implies that the fpga_properties object would match all FPGA resources if used for an fpgaEnumerate() query. The matching criteria can be further refined by using fpgaSet* functions on the properties object.
Instead of creating a new fpga_properties object every time, this function can be used to re-use fpga_properties objects from previous queries.
Parameters:
prop fpga_properties object to clear
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if prop is not a valid object. FPGA_EXCEPTION if an * internal exception occured when trying to access prop.
"},{"location":"opae-code/properties_8h/#function-fpgacloneproperties","title":"function fpgaCloneProperties","text":"Clone a fpga_properties object.
fpga_result fpgaCloneProperties (\nfpga_properties src,\nfpga_properties * dst\n)
Creates a copy of an fpga_properties object.
Note:
This call creates a new properties object and allocates memory for it. Both the 'src' and the newly created 'dst' objects will eventually need to be destroyed using fpgaDestroyProperties().
Parameters:
src fpga_properties object to copy dst New fpga_properties object cloned from 'src'
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if src is not a valid object, or if dst is NULL. FPGA_NO_MEMORY if there was not enough memory to allocate an fpga_properties object for dst. FPGA_EXCEPTION if an internal exception occurred either accessing src or updating dst.
"},{"location":"opae-code/properties_8h/#function-fpgadestroyproperties","title":"function fpgaDestroyProperties","text":"Destroy a fpga_properties object.
fpga_result fpgaDestroyProperties (\nfpga_properties * prop\n)
Destroys an existing fpga_properties object that the caller has previously created using fpgaGetProperties() or fpgaCloneProperties().
Note:
fpgaDestroyProperties() requires the address of an fpga_properties object, similar to fpgaGetPropertiesFromHandle(), fpgaGetProperties(), and fpgaCloneProperties(). Passing any other value results in undefined behavior.
Parameters:
prop Pointer to the fpga_properties object to destroy
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM is prop is not a valid object. FPGA_EXCEPTION if an internal exception occurrred while trying to access prop.
"},{"location":"opae-code/properties_8h/#function-fpgagetproperties","title":"function fpgaGetProperties","text":"Create a fpga_properties object.
fpga_result fpgaGetProperties (\nfpga_token token,\nfpga_properties * prop\n)
Initializes the memory pointed at by prop to represent a properties object, and populates it with the properties of the resource referred to by token. Individual properties can then be queried using fpgaPropertiesGet*() accessor functions.
If token is NULL, an \"empty\" properties object is created to be used as a filter for fpgaEnumerate(). All individual fields are set to dont care`, which implies that the fpga_properties object would match all FPGA resources if used for an fpgaEnumerate() query. The matching criteria can be further refined by using fpgaSet* functions on the properties object, or the object can be populated with the actual properties of a resource by using fpgaUpdateProperties().
Note:
fpgaGetProperties() will allocate memory for the created properties object returned in prop. It is the responsibility of the caller to free this memory after use by calling fpgaDestroyProperties().
Parameters:
token Token to get properties for. Can be NULL, which will create an empty properties object to be used as a filter for fpgaEnumerate(). prop Pointer to a variable of type fpga_properties
Returns:
FPGA_OK on success. FPGA_NO_MEMORY if no memory could be allocated to create the fpga_properties object. FPGA_EXCEPTION if an exception happend while initializing the fpga_properties object.
"},{"location":"opae-code/properties_8h/#function-fpgagetpropertiesfromhandle","title":"function fpgaGetPropertiesFromHandle","text":"Create a fpga_properties object.
fpga_result fpgaGetPropertiesFromHandle (\nfpga_handle handle,\nfpga_properties * prop\n)
Initializes the memory pointed at by prop to represent a properties object, and populates it with the properties of the resource referred to by handle. Individual properties can then be queried using fpgaPropertiesGet*() accessor functions.
Note:
fpgaGetPropertiesFromHandle() will allocate memory for the created properties object returned in prop. It is the responsibility of the caller to free this memory after use by calling fpgaDestroyProperties().
Parameters:
handle Open handle to get properties for. prop Pointer to a variable of type fpga_properties
Returns:
FPGA_OK on success. FPGA_NO_MEMORY if no memory could be allocated to create the fpga_properties object. FPGA_EXCEPTION if an exception happend while initializing the fpga_properties object.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetacceleratorstate","title":"function fpgaPropertiesGetAcceleratorState","text":"Get the state of a accelerator resource property.
fpga_result fpgaPropertiesGetAcceleratorState (\nconst fpga_properties prop,\nfpga_accelerator_state * state\n)
Returns the accelerator state of a accelerator.
Parameters:
prop Properties object to query - must be of type FPGA_ACCELERATOR state Pointer to a accelerator state variable of the accelerator
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetbbsid","title":"function fpgaPropertiesGetBBSID","text":"Get the BBS ID of an FPGA resource property.
fpga_result fpgaPropertiesGetBBSID (\nconst fpga_properties prop,\nuint64_t * bbs_id\n)
Returns the blue bitstream id of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE bbs_id Pointer to a bbs id variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetbbsversion","title":"function fpgaPropertiesGetBBSVersion","text":"Get the BBS Version of an FPGA resource property.
fpga_result fpgaPropertiesGetBBSVersion (\nconst fpga_properties prop,\nfpga_version * bbs_version\n)
Returns the blue bitstream version of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE bbs_version Pointer to a bbs version variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetbus","title":"function fpgaPropertiesGetBus","text":"Get the PCI bus number of a resource.
fpga_result fpgaPropertiesGetBus (\nconst fpga_properties prop,\nuint8_t * bus\n)
Returns the bus number the queried resource.
Parameters:
prop Properties object to query bus Pointer to a PCI bus variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetcapabilities","title":"function fpgaPropertiesGetCapabilities","text":"Get the capabilities FPGA resource property.
fpga_result fpgaPropertiesGetCapabilities (\nconst fpga_properties prop,\nuint64_t * capabilities\n)
Returns the capabilities of an FPGA. Capabilities is a bitfield value
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE capabilities Pointer to a capabilities variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetdevice","title":"function fpgaPropertiesGetDevice","text":"Get the PCI device number of a resource.
fpga_result fpgaPropertiesGetDevice (\nconst fpga_properties prop,\nuint8_t * device\n)
Returns the device number the queried resource.
Parameters:
prop Properties object to query device Pointer to a PCI device variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetdeviceid","title":"function fpgaPropertiesGetDeviceID","text":"Get the device id of the resource.
fpga_result fpgaPropertiesGetDeviceID (\nconst fpga_properties prop,\nuint16_t * device_id\n)
Parameters:
prop Properties object to query device_id Pointer to a device id variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetfunction","title":"function fpgaPropertiesGetFunction","text":"Get the PCI function number of a resource.
fpga_result fpgaPropertiesGetFunction (\nconst fpga_properties prop,\nuint8_t * function\n)
Returns the function number the queried resource.
Parameters:
prop Properties object to query function Pointer to PCI function variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetguid","title":"function fpgaPropertiesGetGUID","text":"Get the GUID of a resource.
fpga_result fpgaPropertiesGetGUID (\nconst fpga_properties prop,\nfpga_guid * guid\n)
Returns the GUID of an FPGA or accelerator object.
For an accelerator, the GUID uniquely identifies a specific accelerator context type, i.e. different accelerators will have different GUIDs. For an FPGA, the GUID is used to identify a certain instance of an FPGA, e.g. to determine whether a given bitstream would be compatible.
Parameters:
prop Properties object to query guid Pointer to a GUID of the slot variable
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetinterface","title":"function fpgaPropertiesGetInterface","text":"Get the OPAE plugin interface implemented by a resource.
fpga_result fpgaPropertiesGetInterface (\nconst fpga_properties prop,\nfpga_interface * interface\n)
Returns the plugin interface enumerator.
Parameters:
prop Properties object to query interface Pointer to an fpga_interface location to store the interface in
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetlocalmemorysize","title":"function fpgaPropertiesGetLocalMemorySize","text":"Get the local memory size of an FPGA resource property.
fpga_result fpgaPropertiesGetLocalMemorySize (\nconst fpga_properties prop,\nuint64_t * lms\n)
Returns the local memory size of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE lms Pointer to a memory size variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetmodel","title":"function fpgaPropertiesGetModel","text":"Get the model of an FPGA resource property.
fpga_result fpgaPropertiesGetModel (\nconst fpga_properties prop,\nchar * model\n)
Returns the model of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE model Model of the FPGA resource (string of minimum FPGA_MODEL_LENGTH length
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnumerrors","title":"function fpgaPropertiesGetNumErrors","text":"Get the number of errors that can be reported by a resource.
fpga_result fpgaPropertiesGetNumErrors (\nconst fpga_properties prop,\nuint32_t * num_errors\n)
Returns the number of error registers understood by a resource.
Parameters:
prop Properties object to query num_errors Pointer to a 32 bit memory location to store the number of supported errors in
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnuminterrupts","title":"function fpgaPropertiesGetNumInterrupts","text":"Get the number of interrupts.
fpga_result fpgaPropertiesGetNumInterrupts (\nconst fpga_properties prop,\nuint32_t * num_interrupts\n)
Returns the number of interrupts of an accelerator properties structure.
Parameters:
prop Properties object to query - must be of type FPGA_ACCELERATOR num_interrupts Pointer to a variable for number of interrupts
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnummmio","title":"function fpgaPropertiesGetNumMMIO","text":"Get the number of mmio spaces.
fpga_result fpgaPropertiesGetNumMMIO (\nconst fpga_properties prop,\nuint32_t * mmio_spaces\n)
Returns the number of mmio spaces of an AFU properties structure.
Parameters:
prop Properties object to query - must be of type FPGA_ACCELERATOR mmio_spaces Pointer to a variable for number of mmio spaces
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetnumslots","title":"function fpgaPropertiesGetNumSlots","text":"Get the number of slots of an FPGA resource property.
fpga_result fpgaPropertiesGetNumSlots (\nconst fpga_properties prop,\nuint32_t * num_slots\n)
Returns the number of slots present in an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE num_slots Pointer to number of slots variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetobjectid","title":"function fpgaPropertiesGetObjectID","text":"Get the object ID of a resource.
fpga_result fpgaPropertiesGetObjectID (\nconst fpga_properties prop,\nuint64_t * object_id\n)
Returns the object ID of a resource. The object ID is a 64 bit identifier that is unique within a single node or system. It represents a similar concept as the token, but can be used across processes (e.g. passed on the command line).
Parameters:
prop Properties object to query object_id Pointer to a 64bit memory location to store the object ID in
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetobjecttype","title":"function fpgaPropertiesGetObjectType","text":"Get the object type of a resource.
fpga_result fpgaPropertiesGetObjectType (\nconst fpga_properties prop,\nfpga_objtype * objtype\n)
Returns the object type of the queried resource.
Parameters:
prop Properties object to query objtype Pointer to an object type variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetparent","title":"function fpgaPropertiesGetParent","text":"Get the token of the parent object.
fpga_result fpgaPropertiesGetParent (\nconst fpga_properties prop,\nfpga_token * parent\n)
Returns the token of the parent of the queried resource in '*parent'.
Parameters:
prop Properties object to query parent Pointer to a token variable of the resource 'prop' is associated with
Returns:
FPGA_NOT_FOUND if resource does not have a parent (e.g. an FPGA_DEVICE resource does not have parents). Also see \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsegment","title":"function fpgaPropertiesGetSegment","text":"Get the PCI segment number of a resource.
fpga_result fpgaPropertiesGetSegment (\nconst fpga_properties prop,\nuint16_t * segment\n)
Returns the segment number of the queried resource.
Parameters:
prop Properties object to query segment Pointer to a PCI segment variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsocketid","title":"function fpgaPropertiesGetSocketID","text":"Get the socket id of a resource.
fpga_result fpgaPropertiesGetSocketID (\nconst fpga_properties prop,\nuint8_t * socket_id\n)
Returns the socket id of the queried resource.
Parameters:
prop Properties object to query socket_id Pointer to a socket id variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsubsystemdeviceid","title":"function fpgaPropertiesGetSubsystemDeviceID","text":"Get the subsystem device id of an FPGA resource property.
fpga_result fpgaPropertiesGetSubsystemDeviceID (\nconst fpga_properties prop,\nuint16_t * subsystem_device_id\n)
Returns the subsystem device id of an FPGA.
Parameters:
prop Properties object to query subsystem_device_id Pointer to a device id variable of the FPGA
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetsubsystemvendorid","title":"function fpgaPropertiesGetSubsystemVendorID","text":"Get the subsystem vendor id of an FPGA resource property.
fpga_result fpgaPropertiesGetSubsystemVendorID (\nconst fpga_properties prop,\nuint16_t * subsystem_vendor_id\n)
Returns the subsystem vendor id of an FPGA.
Parameters:
prop Properties object to query subsystem_vendor_id Pointer to a vendor id variable of the FPGA
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiesgetvendorid","title":"function fpgaPropertiesGetVendorID","text":"Get the vendor id of an FPGA resource property.
fpga_result fpgaPropertiesGetVendorID (\nconst fpga_properties prop,\nuint16_t * vendor_id\n)
Returns the vendor id of an FPGA.
Parameters:
prop Properties object to query - must be of type FPGA_DEVICE vendor_id Pointer to a vendor id variable of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetacceleratorstate","title":"function fpgaPropertiesSetAcceleratorState","text":"Set the state of an accelerator resource property.
fpga_result fpgaPropertiesSetAcceleratorState (\nfpga_properties prop,\nfpga_accelerator_state state\n)
Parameters:
prop Properties object to modify - must be of type FPGA_ACCELERATOR state accelerator state of the accelerator resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetbbsid","title":"function fpgaPropertiesSetBBSID","text":"Set the BBS ID of an FPGA resource property.
fpga_result fpgaPropertiesSetBBSID (\nfpga_properties prop,\nuint64_t bbs_id\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE bbs_id Blue bitstream id of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetbbsversion","title":"function fpgaPropertiesSetBBSVersion","text":"Set the BBS Version of an FPGA resource property.
fpga_result fpgaPropertiesSetBBSVersion (\nfpga_properties prop,\nfpga_version version\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE version Blue bitstream version of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetbus","title":"function fpgaPropertiesSetBus","text":"Set the PCI bus number of a resource.
fpga_result fpgaPropertiesSetBus (\nfpga_properties prop,\nuint8_t bus\n)
Parameters:
prop Properties object to modify bus PCI bus number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetcapabilities","title":"function fpgaPropertiesSetCapabilities","text":"Set the capabilities of an FPGA resource property.
fpga_result fpgaPropertiesSetCapabilities (\nfpga_properties prop,\nuint64_t capabilities\n)
Capabilities is a bitfield value
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE capabilities Capabilities of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetdevice","title":"function fpgaPropertiesSetDevice","text":"Set the PCI device number of a resource.
fpga_result fpgaPropertiesSetDevice (\nfpga_properties prop,\nuint8_t device\n)
Enforces the limitation on the number of devices as specified in the PCI spec.
Parameters:
prop Properties object to modify device PCI device number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetdeviceid","title":"function fpgaPropertiesSetDeviceID","text":"Set the device id of the resource.
fpga_result fpgaPropertiesSetDeviceID (\nfpga_properties prop,\nuint16_t device_id\n)
Parameters:
prop Properties object to modify device_id Device id of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetfunction","title":"function fpgaPropertiesSetFunction","text":"Set the PCI function number of a resource.
fpga_result fpgaPropertiesSetFunction (\nfpga_properties prop,\nuint8_t function\n)
Enforces the limitation on the number of functions as specified in the PCI spec.
Parameters:
prop Properties object to modify function PCI function number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetguid","title":"function fpgaPropertiesSetGUID","text":"Set the GUID of a resource.
fpga_result fpgaPropertiesSetGUID (\nfpga_properties prop,\nfpga_guid guid\n)
Sets the GUID of an FPGA or accelerator object.
For an accelerator, the GUID uniquely identifies a specific accelerator context type, i.e. different accelerators will have different GUIDs. For an FPGA, the GUID is used to identify a certain instance of an FPGA, e.g. to determine whether a given bitstream would be compatible.
Parameters:
prop Properties object to modify guid Pointer to a GUID of the slot variable
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetinterface","title":"function fpgaPropertiesSetInterface","text":"Set the OPAE plugin interface implemented by a resource.
fpga_result fpgaPropertiesSetInterface (\nconst fpga_properties prop,\nfpga_interface interface\n)
Set the plugin interface enumerator.
Parameters:
prop Properties object to query interface The interface enumerator to set
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetlocalmemorysize","title":"function fpgaPropertiesSetLocalMemorySize","text":"Set the local memory size of an FPGA resource property.
fpga_result fpgaPropertiesSetLocalMemorySize (\nfpga_properties prop,\nuint64_t lms\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE lms Local memory size of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetmodel","title":"function fpgaPropertiesSetModel","text":"Set the model of an FPGA resource property.
fpga_result fpgaPropertiesSetModel (\nfpga_properties prop,\nchar * model\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE model Model of the FPGA resource (string of maximum FPGA_MODEL_LENGTH length
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnumerrors","title":"function fpgaPropertiesSetNumErrors","text":"Set the number of error registers.
fpga_result fpgaPropertiesSetNumErrors (\nconst fpga_properties prop,\nuint32_t num_errors\n)
Set the number of error registers understood by a resource to enumerate.
Parameters:
prop Properties object to query num_errors Number of errors
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnuminterrupts","title":"function fpgaPropertiesSetNumInterrupts","text":"Set the number of interrupts.
fpga_result fpgaPropertiesSetNumInterrupts (\nfpga_properties prop,\nuint32_t num_interrupts\n)
Sets the number of interrupts of an accelerator properties structure.
Parameters:
prop Properties object to modify - must be of type FPGA_ACCELERATOR num_interrupts Number of interrupts of the accelerator
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnummmio","title":"function fpgaPropertiesSetNumMMIO","text":"Set the number of mmio spaces.
fpga_result fpgaPropertiesSetNumMMIO (\nfpga_properties prop,\nuint32_t mmio_spaces\n)
Sets the number of mmio spaces of an AFU properties structure.
Parameters:
prop Properties object to modify - must be of type FPGA_ACCELERATOR mmio_spaces Number of MMIO spaces of the accelerator
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_ACCELERATOR. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetnumslots","title":"function fpgaPropertiesSetNumSlots","text":"Set the number of slots of an FPGA resource property.
fpga_result fpgaPropertiesSetNumSlots (\nfpga_properties prop,\nuint32_t num_slots\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE num_slots Number of slots of the FPGA
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetobjectid","title":"function fpgaPropertiesSetObjectID","text":"Set the object ID of a resource.
fpga_result fpgaPropertiesSetObjectID (\nconst fpga_properties prop,\nuint64_t object_id\n)
Sets the object ID of a resource. The object ID is a 64 bit identifier that is unique within a single node or system. It represents a similar concept as the token, but can be used across processes (e.g. passed on the command line).
Parameters:
prop Properties object to query object_id A 64bit value to use as the object ID
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetobjecttype","title":"function fpgaPropertiesSetObjectType","text":"Set the object type of a resource.
fpga_result fpgaPropertiesSetObjectType (\nfpga_properties prop,\nfpga_objtype objtype\n)
Sets the object type of the resource. * Currently supported object types are FPGA_DEVICE and FPGA_ACCELERATOR.
Parameters:
prop Properties object to modify objtype Object type of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetparent","title":"function fpgaPropertiesSetParent","text":"Set the token of the parent object.
fpga_result fpgaPropertiesSetParent (\nfpga_properties prop,\nfpga_token parent\n)
Parameters:
prop Properties object to modify parent Pointer to a token variable of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsegment","title":"function fpgaPropertiesSetSegment","text":"Set the PCI segment number of a resource.
fpga_result fpgaPropertiesSetSegment (\nfpga_properties prop,\nuint16_t segment\n)
Parameters:
prop Properties object to modify segment PCI segment number of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsocketid","title":"function fpgaPropertiesSetSocketID","text":"Set the socket id of the resource.
fpga_result fpgaPropertiesSetSocketID (\nfpga_properties prop,\nuint8_t socket_id\n)
Parameters:
prop Properties object to modify socket_id Socket id of the resource 'prop' is associated with
Returns:
See \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsubsystemdeviceid","title":"function fpgaPropertiesSetSubsystemDeviceID","text":"Set the subsystem device id of an FPGA resource property.
fpga_result fpgaPropertiesSetSubsystemDeviceID (\nfpga_properties prop,\nuint16_t subsystem_device_id\n)
Parameters:
prop Properties object to modify subsystem_device_id Subsystem Device id of the FPGA resource
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetsubsystemvendorid","title":"function fpgaPropertiesSetSubsystemVendorID","text":"Set the subsystem vendor id of an FPGA resource property.
fpga_result fpgaPropertiesSetSubsystemVendorID (\nfpga_properties prop,\nuint16_t subsystem_vendor_id\n)
Parameters:
prop Properties object to modify subsystem_vendor_id Subsystem Vendor id of the FPGA resource
Returns:
FPGA_OK on success. See also \"Accessor Return Values\" in properties.h.
"},{"location":"opae-code/properties_8h/#function-fpgapropertiessetvendorid","title":"function fpgaPropertiesSetVendorID","text":"Set the vendor id of an FPGA resource property.
fpga_result fpgaPropertiesSetVendorID (\nfpga_properties prop,\nuint16_t vendor_id\n)
Parameters:
prop Properties object to modify - must be of type FPGA_DEVICE vendor_id Vendor id of the FPGA resource
Returns:
FPGA_INVALID_PARAM if object type is not FPGA_DEVICE. See also \"Accessor Return Values\" in properties.h.
Note:
This API is not currently supported.
"},{"location":"opae-code/properties_8h/#function-fpgaupdateproperties","title":"function fpgaUpdateProperties","text":"Update a fpga_properties object.
fpga_result fpgaUpdateProperties (\nfpga_token token,\nfpga_properties prop\n)
Populates the properties object 'prop' with properties of the resource referred to by 'token'. Unlike fpgaGetProperties(), this call will not create a new properties object or allocate memory for it, but use a previously created properties object.
Parameters:
token Token to retrieve properties for prop fpga_properties object to update
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if token or prop are not valid objects. FPGA_NOT_FOUND if the resource referred to by token was not found. FPGA_NO_DRIVER if not driver is loaded. FPGA_EXCEPTION if an internal exception occured when trying to update prop.
The documentation for this class was generated from the following file docs/sw/include/opae/properties.h
"},{"location":"opae-code/properties_8h_source/","title":"File properties.h","text":"File List > docs > sw > include > opae > properties.h
Go to the documentation of this file.
// Copyright(c) 2017-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_PROPERTIES_H__\n#define __FPGA_PROPERTIES_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetPropertiesFromHandle(fpga_handle handle, fpga_properties *prop);\nfpga_result fpgaGetProperties(fpga_token token, fpga_properties *prop);\nfpga_result fpgaUpdateProperties(fpga_token token, fpga_properties prop);\nfpga_result fpgaClearProperties(fpga_properties prop);\nfpga_result fpgaCloneProperties(fpga_properties src, fpga_properties *dst);\nfpga_result fpgaDestroyProperties(fpga_properties *prop);\nfpga_result fpgaPropertiesGetParent(const fpga_properties prop,\nfpga_token *parent);\nfpga_result fpgaPropertiesSetParent(fpga_properties prop,\nfpga_token parent);\nfpga_result fpgaPropertiesGetObjectType(const fpga_properties prop,\nfpga_objtype *objtype);\nfpga_result fpgaPropertiesSetObjectType(fpga_properties prop,\nfpga_objtype objtype);\nfpga_result fpgaPropertiesGetSegment(const fpga_properties prop, uint16_t *segment);\nfpga_result fpgaPropertiesSetSegment(fpga_properties prop, uint16_t segment);\nfpga_result fpgaPropertiesGetBus(const fpga_properties prop, uint8_t *bus);\nfpga_result fpgaPropertiesSetBus(fpga_properties prop, uint8_t bus);\nfpga_result fpgaPropertiesGetDevice(const fpga_properties prop,\nuint8_t *device);\nfpga_result fpgaPropertiesSetDevice(fpga_properties prop,\nuint8_t device);\nfpga_result fpgaPropertiesGetFunction(const fpga_properties prop,\nuint8_t *function);\nfpga_result fpgaPropertiesSetFunction(fpga_properties prop,\nuint8_t function);\nfpga_result fpgaPropertiesGetSocketID(const fpga_properties prop,\nuint8_t *socket_id);\nfpga_result fpgaPropertiesSetSocketID(fpga_properties prop,\nuint8_t socket_id);\nfpga_result fpgaPropertiesGetDeviceID(const fpga_properties prop,\nuint16_t *device_id);\nfpga_result fpgaPropertiesSetDeviceID(fpga_properties prop,\nuint16_t device_id);\nfpga_result fpgaPropertiesGetNumSlots(const fpga_properties prop,\nuint32_t *num_slots);\nfpga_result fpgaPropertiesSetNumSlots(fpga_properties prop,\nuint32_t num_slots);\nfpga_result fpgaPropertiesGetBBSID(const fpga_properties prop,\nuint64_t *bbs_id);\nfpga_result fpgaPropertiesSetBBSID(fpga_properties prop,\nuint64_t bbs_id);\nfpga_result fpgaPropertiesGetBBSVersion(const fpga_properties prop,\nfpga_version *bbs_version);\nfpga_result fpgaPropertiesSetBBSVersion(fpga_properties prop,\nfpga_version version);\nfpga_result fpgaPropertiesGetVendorID(const fpga_properties prop,\nuint16_t *vendor_id);\nfpga_result fpgaPropertiesSetVendorID(fpga_properties prop,\nuint16_t vendor_id);\nfpga_result fpgaPropertiesGetModel(const fpga_properties prop,\nchar *model);\nfpga_result fpgaPropertiesSetModel(fpga_properties prop,\nchar *model);\nfpga_result fpgaPropertiesGetLocalMemorySize(const fpga_properties prop,\nuint64_t *lms);\nfpga_result fpgaPropertiesSetLocalMemorySize(fpga_properties prop,\nuint64_t lms);\nfpga_result fpgaPropertiesGetCapabilities(const fpga_properties prop,\nuint64_t *capabilities);\nfpga_result fpgaPropertiesSetCapabilities(fpga_properties prop,\nuint64_t capabilities);\nfpga_result fpgaPropertiesGetGUID(const fpga_properties prop,\nfpga_guid *guid);\nfpga_result fpgaPropertiesSetGUID(fpga_properties prop, fpga_guid guid);\nfpga_result fpgaPropertiesGetNumMMIO(const fpga_properties prop,\nuint32_t *mmio_spaces);\nfpga_result fpgaPropertiesSetNumMMIO(fpga_properties prop,\nuint32_t mmio_spaces);\nfpga_result fpgaPropertiesGetNumInterrupts(const fpga_properties prop,\nuint32_t *num_interrupts);\nfpga_result fpgaPropertiesSetNumInterrupts(fpga_properties prop,\nuint32_t num_interrupts);\nfpga_result fpgaPropertiesGetAcceleratorState(const fpga_properties prop,\nfpga_accelerator_state *state);\nfpga_result fpgaPropertiesSetAcceleratorState(fpga_properties prop,\nfpga_accelerator_state state);\nfpga_result fpgaPropertiesGetObjectID(const fpga_properties prop,\nuint64_t *object_id);\nfpga_result fpgaPropertiesSetObjectID(const fpga_properties prop,\nuint64_t object_id);\nfpga_result fpgaPropertiesGetNumErrors(const fpga_properties prop,\nuint32_t *num_errors);\nfpga_result fpgaPropertiesSetNumErrors(const fpga_properties prop,\nuint32_t num_errors);\nfpga_result fpgaPropertiesGetInterface(const fpga_properties prop,\nfpga_interface *interface);\nfpga_result fpgaPropertiesSetInterface(const fpga_properties prop,\nfpga_interface interface);\nfpga_result fpgaPropertiesGetSubsystemVendorID(const fpga_properties prop,\nuint16_t *subsystem_vendor_id);\nfpga_result fpgaPropertiesSetSubsystemVendorID(fpga_properties prop,\nuint16_t subsystem_vendor_id);\nfpga_result fpgaPropertiesGetSubsystemDeviceID(const fpga_properties prop,\nuint16_t *subsystem_device_id);\nfpga_result fpgaPropertiesSetSubsystemDeviceID(fpga_properties prop,\nuint16_t subsystem_device_id);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_PROPERTIES_H__\n
"},{"location":"opae-code/sysobject_8h/","title":"File sysobject.h","text":"FileList > docs > sw > include > opae > sysobject.h
Go to the source code of this file.
Functions to read/write from system objects. More...
#include <opae/types.h>
"},{"location":"opae-code/sysobject_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaDestroyObject (fpga_object * obj) Free memory used for the fpga_object data structure. fpga_result fpgaHandleGetObject (fpga_handle handle, const char * name, fpga_object * object, int flags) Create an fpga_object data structure from a handle. fpga_result fpgaObjectGetObject (fpga_object parent, const char * name, fpga_object * object, int flags) Create an fpga_object data structure from a parent object. fpga_result fpgaObjectGetObjectAt (fpga_object parent, size_t idx, fpga_object * object) Create an fpga_object data structure from a parent object using a given index. fpga_result fpgaObjectGetSize (fpga_object obj, uint32_t * value, int flags) Retrieve the size of the object. fpga_result fpgaObjectGetType (fpga_object obj, enum fpga_sysobject_type * type) Get the sysobject type (container or attribute) fpga_result fpgaObjectRead (fpga_object obj, uint8_t * buffer, size_t offset, size_t len, int flags) Read bytes from an FPGA object. fpga_result fpgaObjectRead64 (fpga_object obj, uint64_t * value, int flags) Read a 64-bit value from an FPGA object. fpga_result fpgaObjectWrite64 (fpga_object obj, uint64_t value, int flags) Write 64-bit value to an FPGA object. fpga_result fpgaTokenGetObject (fpga_token token, const char * name, fpga_object * object, int flags) Create an fpga_object data structures."},{"location":"opae-code/sysobject_8h/#detailed-description","title":"Detailed Description","text":"On Linux systems with the OPAE kernel driver, this is used to access sysfs nodes created by the driver.
"},{"location":"opae-code/sysobject_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/sysobject_8h/#function-fpgadestroyobject","title":"function fpgaDestroyObject","text":"Free memory used for the fpga_object data structure.
fpga_result fpgaDestroyObject (\nfpga_object * obj\n)
Note:
fpgaDestroyObject() requires the address of an fpga_object as created by fpgaTokenGetObject(), fpgaHandleGetObject(), or fpgaObjectGetObject(). Passing any other value results in undefind behavior.
Parameters:
obj Pointer to the fpga_object instance to destroy
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if the object is NULL, FPGA_EXCEPTION if an internal error is encountered.
"},{"location":"opae-code/sysobject_8h/#function-fpgahandlegetobject","title":"function fpgaHandleGetObject","text":"Create an fpga_object data structure from a handle.
fpga_result fpgaHandleGetObject (\nfpga_handle handle,\nconst char * name,\nfpga_object * object,\nint flags\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register, or container. This object has read/write access..
Parameters:
handle Handle identifying a resource (accelerator or device) name A key identifying an object belonging to a resource. object Pointer to memory to store the object in flags Control behavior of object identification and creation FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
Note:
Names that begin with '.' or '/' or contain '..' are not allowed and result in FPGA_INVALID_PARAM being returned
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgetobject","title":"function fpgaObjectGetObject","text":"Create an fpga_object data structure from a parent object.
fpga_result fpgaObjectGetObject (\nfpga_object parent,\nconst char * name,\nfpga_object * object,\nint flags\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register, or container. If the parent object was created with a handle, then the new object will inherit the handle allowing it to have read-write access to the object data.
Parameters:
parent A parent container fpga_object. name A key identifying a sub-object of the parent container. object Pointer to memory to store the object in. flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid - this includes a parent object that is not a container object. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
Note:
Names that begin with '.' or '/' or contain '..' are not allowed and result in FPGA_INVALID_PARAM being returned
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgetobjectat","title":"function fpgaObjectGetObjectAt","text":"Create an fpga_object data structure from a parent object using a given index.
fpga_result fpgaObjectGetObjectAt (\nfpga_object parent,\nsize_t idx,\nfpga_object * object\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register, or container. If the parent object was created with a handle, then the new object will inherit the handle allowing it to have read-write access to the object data.
Parameters:
parent A parent container 'fpga_object' idx A positive index less than the size reported by the parent. object Pointer to memory to store the object in.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid - this includes a parent object that is not a container object. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgetsize","title":"function fpgaObjectGetSize","text":"Retrieve the size of the object.
fpga_result fpgaObjectGetSize (\nfpga_object obj,\nuint32_t * value,\nint flags\n)
Parameters:
obj An fpga_object instance. value Pointer to variable to store size in. flags Flags that control how the object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the size.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of supplied parameters is invalid. FPGA_EXCEPTION if error occurred.
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectgettype","title":"function fpgaObjectGetType","text":"Get the sysobject type (container or attribute)
fpga_result fpgaObjectGetType (\nfpga_object obj,\nenum fpga_sysobject_type * type\n)
Parameters:
obj An fpga_object instance type The type of object (FPGA_OBJECT_CONTAINER or FPGA_OBJECT_ATTRIBUTE)
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters are null or invalid
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectread","title":"function fpgaObjectRead","text":"Read bytes from an FPGA object.
fpga_result fpgaObjectRead (\nfpga_object obj,\nuint8_t * buffer,\nsize_t offset,\nsize_t len,\nint flags\n)
Parameters:
obj An fpga_object instance. buffer Pointer to a buffer to read bytes into. offset Byte offset relative to objects internal buffer where to begin reading bytes from. len The length, in bytes, to read from the object. flags Flags that control how object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters is invalid
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectread64","title":"function fpgaObjectRead64","text":"Read a 64-bit value from an FPGA object.
fpga_result fpgaObjectRead64 (\nfpga_object obj,\nuint64_t * value,\nint flags\n)
The value is assumed to be in string format and will be parsed. See flags below for changing that behavior.
Parameters:
obj An fpga_object instance value Pointer to a 64-bit variable to store the value in flags Flags that control how the object is read If FPGA_OBJECT_SYNC is used then object will update its buffered copy before retrieving the data. If FPGA_OBJECT_RAW is used, then the data will be read as raw bytes into the uint64_t pointer variable.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters is invalid
"},{"location":"opae-code/sysobject_8h/#function-fpgaobjectwrite64","title":"function fpgaObjectWrite64","text":"Write 64-bit value to an FPGA object.
fpga_result fpgaObjectWrite64 (\nfpga_object obj,\nuint64_t value,\nint flags\n)
The value will be converted to string before writing. See flags below for changing that behavior.
Parameters:
obj An fpga_object instance. value The value to write to the object flags Flags that control how the object is written If FPGA_OBJECT_RAW is used, then the value will be written as raw bytes.
Returns:
FPGA_OK on success, FPGA_INVALID_PARAM if any of the supplied parameters is invalid
Note:
The object must have been created using a handle to a resource.
"},{"location":"opae-code/sysobject_8h/#function-fpgatokengetobject","title":"function fpgaTokenGetObject","text":"Create an fpga_object data structures.
fpga_result fpgaTokenGetObject (\nfpga_token token,\nconst char * name,\nfpga_object * object,\nint flags\n)
An fpga_object is a handle to an FPGA resource which can be an attribute, register or a container. This object is read-only.
Parameters:
token Token identifying a resource (accelerator or device) name A key identifying an object belonging to a resource. object Pointer to memory to store the object in flags Control behavior of object identification and creation. FPGA_OBJECT_GLOB is used to indicate that the name should be treated as a globbing expression. FPGA_OBJECT_RECURSE_ONE indicates that subobjects be created for objects one level down from the object identified by name. FPGA_OBJECT_RECURSE_ALL indicates that subobjects be created for all objects below the current object identified by name.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if any of the supplied parameters is invalid. FPGA_NOT_FOUND if an object cannot be found with the given key. FPGA_NOT_SUPPORTED if this function is not supported by the current implementation of this API.
Note:
Names that begin with '.' or '/' or contain '..' are not allowed and result in FPGA_INVALID_PARAM being returned
The documentation for this class was generated from the following file docs/sw/include/opae/sysobject.h
"},{"location":"opae-code/sysobject_8h_source/","title":"File sysobject.h","text":"File List > docs > sw > include > opae > sysobject.h
Go to the documentation of this file.
// Copyright(c) 2017-2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_SYSOBJECT_H__\n#define __FPGA_SYSOBJECT_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaTokenGetObject(fpga_token token, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaHandleGetObject(fpga_handle handle, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaObjectGetObject(fpga_object parent, const char *name,\nfpga_object *object, int flags);\nfpga_result fpgaObjectGetObjectAt(fpga_object parent, size_t idx,\nfpga_object *object);\nfpga_result fpgaObjectGetType(fpga_object obj, enum fpga_sysobject_type *type);\nfpga_result fpgaDestroyObject(fpga_object *obj);\nfpga_result fpgaObjectGetSize(fpga_object obj, uint32_t *value, int flags);\nfpga_result fpgaObjectRead(fpga_object obj, uint8_t *buffer, size_t offset,\nsize_t len, int flags);\nfpga_result fpgaObjectRead64(fpga_object obj, uint64_t *value, int flags);\nfpga_result fpgaObjectWrite64(fpga_object obj, uint64_t value, int flags);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif /* !__FPGA_SYSOBJECT_H__ */\n
"},{"location":"opae-code/types_8h/","title":"File types.h","text":"FileList > docs > sw > include > opae > types.h
Go to the source code of this file.
Type definitions for FPGA API. More...
#include <stdint.h>#include <stddef.h>#include <stdbool.h>#include <opae/types_enum.h>
"},{"location":"opae-code/types_8h/#classes","title":"Classes","text":"Type Name struct _fpga_token_header Internal token type header. struct fpga_error_info struct fpga_metric Metric struct. struct fpga_metric_info Metric info struct. struct fpga_version Semantic version. struct metric_threshold struct threshold Threshold struct."},{"location":"opae-code/types_8h/#public-types","title":"Public Types","text":"Type Name typedef void * fpga_event_handle Handle to an event object. typedef uint8_t fpga_guid Globally unique identifier (GUID) typedef void * fpga_handle Handle to an FPGA resource. typedef struct fpga_metric fpga_metric Metric struct. typedef struct fpga_metric_info fpga_metric_info Metric info struct. typedef void * fpga_object Object pertaining to an FPGA resource as identified by a unique name. typedef void * fpga_properties Object for expressing FPGA resource properties. typedef void * fpga_token Token for referencing FPGA resources. typedef struct _fpga_token_header fpga_token_header Internal token type header. typedef struct metric_threshold metric_threshold union metric_value Metric value union. typedef struct threshold threshold Threshold struct."},{"location":"opae-code/types_8h/#macros","title":"Macros","text":"Type Name define FPGA_ERROR_NAME_MAX 64Information about an error register. define FPGA_METRIC_STR_SIZE 256FPGA Metric string size. define fpga_is_parent_child (__parent_hdr, __child_hdr) Determine token parent/child relationship."},{"location":"opae-code/types_8h/#detailed-description","title":"Detailed Description","text":"OPAE uses the three opaque types fpga_properties, fpga_token, and fpga_handle to create a hierarchy of objects that can be used to enumerate, reference, acquire, and query FPGA resources. This object model is designed to be extensible to account for different FPGA architectures and platforms.
"},{"location":"opae-code/types_8h/#initialization","title":"Initialization","text":"OPAEs management of the opaque types fpga_properties, fpga_token, and fpga_handle relies on the proper initialization of variables of these types. In other words, before doing anything with a variable of one of these opaque types, you need to first initialize them.
The respective functions that initialize opaque types are:
- fpgaGetProperties() and fpgaCloneProperties() for
fpga_properties - fpgaEnumerate() and fpgaCloneToken() for
fpga_token - fpgaOpen() for
fpga_handle
This should intuitively make sense - fpgaGetProperties() creates fpga_properties objects, fpgaEnumerate() creates fpga_token objects, fpgaOpen() creates fpga_handle objects, and fpgaCloneProperties() and fpgaCloneToken() clone (create) fpga_properties and fpga_token objects, respectively.
Since these opaque types are interpreted as pointers (they are typedef'd to a void *), passing an uninitialized opaque type into any function except the respective initailzation function will result in undefined behaviour, because OPAE will try to follow an invalid pointer. Undefined behaviour in this case may include an unexpected error code, or an application crash.
"},{"location":"opae-code/types_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/types_8h/#typedef-fpga_event_handle","title":"typedef fpga_event_handle","text":"Handle to an event object.
typedef void* fpga_event_handle;\n
OPAE provides an interface to asynchronous events that can be generated by different FPGA resources. The event API provides functions to register for these events; associated with every event a process has registered for is an fpga_event_handle, which encapsulates the OS-specific data structure for event objects.
After use, fpga_event_handle objects should be destroyed using fpgaDestroyEventHandle() to free backing memory used by the fpga_event_handle object.
"},{"location":"opae-code/types_8h/#typedef-fpga_guid","title":"typedef fpga_guid","text":"Globally unique identifier (GUID)
typedef uint8_t fpga_guid[16];\n
GUIDs are used widely within OPAE for helping identify FPGA resources. For example, every FPGA resource has a guid property, which can be (and in the case of FPGA_ACCELERATOR resource primarily is) used for enumerating a resource of a specific type.
fpga_guid is compatible with libuuid's uuid_t, so users can use libuuid functions like uuid_parse() to create and work with GUIDs.
"},{"location":"opae-code/types_8h/#typedef-fpga_handle","title":"typedef fpga_handle","text":"Handle to an FPGA resource.
typedef void* fpga_handle;\n
A valid fpga_handle object, as populated by fpgaOpen(), denotes ownership of an FPGA resource. Note that ownership can be exclusive or shared, depending on the flags used in fpgaOpen(). Ownership can be released by calling fpgaClose(), which will render the underlying handle invalid.
Many OPAE C API functions require a valid token (which is synonymous with ownership of the resource).
"},{"location":"opae-code/types_8h/#typedef-fpga_metric","title":"typedef fpga_metric","text":"typedef struct fpga_metric fpga_metric;\n
"},{"location":"opae-code/types_8h/#typedef-fpga_metric_info","title":"typedef fpga_metric_info","text":"typedef struct fpga_metric_info fpga_metric_info;\n
"},{"location":"opae-code/types_8h/#typedef-fpga_object","title":"typedef fpga_object","text":"Object pertaining to an FPGA resource as identified by a unique name.
typedef void* fpga_object;\n
An fpga_object represents either a device attribute or a container of attributes. Similar to filesystems, a '/' may be used to seperate objects in an object hierarchy. Once on object is acquired, it may be used to read or write data in a resource attribute or to query sub-objects if the object is a container object. The data in an object is buffered and will be kept around until the object is destroyed. Additionally, the data in an attribute can by synchronized from the owning resource using the FPGA_OBJECT_SYNC flag during read operations. The name identifying the object is unique with respect to the resource that owns it. A parent resource may be identified by an fpga_token object, by an fpga_handle object, or another fpga_object object. If a handle object is used when opening the object, then the object is opened with read-write access. Otherwise, the object is read-only.
"},{"location":"opae-code/types_8h/#typedef-fpga_properties","title":"typedef fpga_properties","text":"Object for expressing FPGA resource properties.
typedef void* fpga_properties;\n
fpga_properties objects encapsulate all enumerable information about an FPGA resources. They can be used for two purposes: selective enumeration (discovery) and querying information about existing resources.
For selective enumeration, usually an empty fpga_properties object is created (using fpgaGetProperties()) and then populated with the desired criteria for enumeration. An array of fpga_properties can then be passed to fpgaEnumerate(), which will return a list of fpga_token objects matching these criteria.
For querying properties of existing FPGA resources, fpgaGetProperties() can also take an fpga_token and will return an fpga_properties object populated with information about the resource referenced by that token.
After use, fpga_properties objects should be destroyed using fpga_destroyProperties() to free backing memory used by the fpga_properties object.
"},{"location":"opae-code/types_8h/#typedef-fpga_token","title":"typedef fpga_token","text":"Token for referencing FPGA resources.
typedef void* fpga_token;\n
An fpga_token serves as a reference to a specific FPGA resource present in the system. Holding an fpga_token does not constitute ownership of the FPGA resource - it merely allows the user to query further information about a resource, or to use fpgaOpen() to acquire ownership.
fpga_tokens are usually returned by fpgaEnumerate() or fpgaPropertiesGetParent(), and used by fpgaOpen() to acquire ownership and yield a handle to the resource. Some API calls also take fpga_tokens as arguments if they don't require ownership of the resource in question.
"},{"location":"opae-code/types_8h/#typedef-fpga_token_header","title":"typedef fpga_token_header","text":"Internal token type header.
typedef struct _fpga_token_header fpga_token_header;\n
Each plugin (dfl: libxfpga.so, vfio: libopae-v.so) implements its own proprietary token type. This header must appear at offset zero within that structure.
eg, see lib/plugins/xfpga/types_int.h:struct _fpga_token and lib/plugins/vfio/opae_vfio.h:struct _vfio_token.
"},{"location":"opae-code/types_8h/#typedef-metric_threshold","title":"typedef metric_threshold","text":"typedef struct metric_threshold metric_threshold;\n
"},{"location":"opae-code/types_8h/#union-metric_value","title":"union metric_value","text":""},{"location":"opae-code/types_8h/#typedef-threshold","title":"typedef threshold","text":"typedef struct threshold threshold;\n
"},{"location":"opae-code/types_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/types_8h/#define-fpga_error_name_max","title":"define FPGA_ERROR_NAME_MAX","text":"Information about an error register.
#define FPGA_ERROR_NAME_MAX 64\n
This data structure captures information about an error register exposed by an accelerator resource. The error API provides functions to retrieve these information structures from a particular resource.
"},{"location":"opae-code/types_8h/#define-fpga_metric_str_size","title":"define FPGA_METRIC_STR_SIZE","text":"#define FPGA_METRIC_STR_SIZE 256\n
"},{"location":"opae-code/types_8h/#define-fpga_is_parent_child","title":"define fpga_is_parent_child","text":"Determine token parent/child relationship.
#define fpga_is_parent_child (\n__parent_hdr,\n__child_hdr\n) (((__parent_hdr)->objtype == FPGA_DEVICE ) && \\\n ((__child_hdr)->objtype == FPGA_ACCELERATOR ) && \\\n ((__parent_hdr)->segment == (__child_hdr)->segment) && \\\n ((__parent_hdr)->bus == (__child_hdr)->bus) && \\\n ((__parent_hdr)->device == (__child_hdr)->device))\n
Given pointers to two fpga_token_header structs, determine whether the first is the parent of the second. A parent will have objtype == FPGA_DEVICE. A child will have objtype == FPGA_ACCELERATOR. The PCIe address of the two headers will match in all but the function fields.
The documentation for this class was generated from the following file docs/sw/include/opae/types.h
"},{"location":"opae-code/types_8h_source/","title":"File types.h","text":"File List > docs > sw > include > opae > types.h
Go to the documentation of this file.
// Copyright(c) 2018-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_TYPES_H__\n#define __FPGA_TYPES_H__\n#include <stdint.h>\n#include <stddef.h>\n#include <stdbool.h>\n#include <opae/types_enum.h>\ntypedef void *fpga_properties;\ntypedef void *fpga_token;\ntypedef void *fpga_handle;\ntypedef uint8_t fpga_guid[16];\ntypedef struct {\nuint8_t major; uint8_t minor; uint16_t patch; } fpga_version;\ntypedef void *fpga_event_handle;\n#define FPGA_ERROR_NAME_MAX 64\nstruct fpga_error_info {\nchar name[FPGA_ERROR_NAME_MAX]; bool can_clear; };\ntypedef void *fpga_object;\n#define FPGA_METRIC_STR_SIZE 256\ntypedef union {\nuint64_t ivalue; // Metric integer value\ndouble dvalue; // Metric double value\nfloat fvalue; // Metric float value\nbool bvalue; // Metric bool value\n} metric_value;\ntypedef struct fpga_metric_info {\nuint64_t metric_num; // Metric index num\nfpga_guid metric_guid; // Metric guid\nchar qualifier_name[FPGA_METRIC_STR_SIZE]; // Metric full name\nchar group_name[FPGA_METRIC_STR_SIZE]; // Metric group name\nchar metric_name[FPGA_METRIC_STR_SIZE]; // Metric name\nchar metric_units[FPGA_METRIC_STR_SIZE]; // Metric units\nenum fpga_metric_datatype metric_datatype; // Metric data type\nenum fpga_metric_type metric_type; // Metric group type\n} fpga_metric_info;\ntypedef struct fpga_metric {\nuint64_t metric_num; // Metric index num\nmetric_value value; // Metric value\nbool isvalid; // Metric value is valid\n} fpga_metric;\ntypedef struct threshold {\nchar threshold_name[FPGA_METRIC_STR_SIZE]; // Threshold name\nuint32_t is_valid; // Threshold is valid\ndouble value; // Threshold value\n} threshold;\ntypedef struct metric_threshold {\nchar metric_name[FPGA_METRIC_STR_SIZE]; // Metric Threshold name\nthreshold upper_nr_threshold; // Upper Non-Recoverable Threshold\nthreshold upper_c_threshold; // Upper Critical Threshold\nthreshold upper_nc_threshold; // Upper Non-Critical Threshold\nthreshold lower_nr_threshold; // Lower Non-Recoverable Threshold\nthreshold lower_c_threshold; // Lower Critical Threshold\nthreshold lower_nc_threshold; // Lower Non-Critical Threshold\nthreshold hysteresis; // Hysteresis\n} metric_threshold;\ntypedef struct _fpga_token_header {\nuint64_t magic;\nuint16_t vendor_id;\nuint16_t device_id;\nuint16_t segment;\nuint8_t bus;\nuint8_t device;\nuint8_t function;\nfpga_interface interface;\nfpga_objtype objtype;\nuint64_t object_id;\nfpga_guid guid;\nuint16_t subsystem_vendor_id;\nuint16_t subsystem_device_id;\n} fpga_token_header;\n#define fpga_is_parent_child(__parent_hdr, __child_hdr) \\\n(((__parent_hdr)->objtype == FPGA_DEVICE) && \\\n ((__child_hdr)->objtype == FPGA_ACCELERATOR) && \\\n ((__parent_hdr)->segment == (__child_hdr)->segment) && \\\n ((__parent_hdr)->bus == (__child_hdr)->bus) && \\\n ((__parent_hdr)->device == (__child_hdr)->device))\n#endif // __FPGA_TYPES_H__\n
"},{"location":"opae-code/types__enum_8h/","title":"File types_enum.h","text":"FileList > docs > sw > include > opae > types_enum.h
Go to the source code of this file.
Definitions of enumerated types for the OPAE C API. More...
"},{"location":"opae-code/types__enum_8h/#public-types","title":"Public Types","text":"Type Name enum fpga_accelerator_state accelerator state enum fpga_buffer_flags Buffer flags. enum fpga_event_type FPGA events. enum fpga_interface OPAE plugin interface. enum fpga_metric_datatype Metrics data type. enum fpga_metric_type fpga metrics types opae defines power,thermal, performance counter and afu metric types enum fpga_objtype OPAE FPGA resources (objects) enum fpga_open_flags Open flags. enum fpga_reconf_flags Reconfiguration flags. enum fpga_result OPAE C API function return codes. enum fpga_sysobject_flags enum fpga_sysobject_type"},{"location":"opae-code/types__enum_8h/#detailed-description","title":"Detailed Description","text":"This file defines return and error codes, event and object types, states, and flags as used or reported by OPAE C API functions.
"},{"location":"opae-code/types__enum_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/types__enum_8h/#enum-fpga_accelerator_state","title":"enum fpga_accelerator_state","text":"enum fpga_accelerator_state {\nFPGA_ACCELERATOR_ASSIGNED = 0,\nFPGA_ACCELERATOR_UNASSIGNED\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_buffer_flags","title":"enum fpga_buffer_flags","text":"Buffer flags.
enum fpga_buffer_flags {\nFPGA_BUF_PREALLOCATED = (1u << 0),\nFPGA_BUF_QUIET = (1u << 1),\nFPGA_BUF_READ_ONLY = (1u << 2)\n};\n
These flags can be passed to the fpgaPrepareBuffer() function.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_event_type","title":"enum fpga_event_type","text":"FPGA events.
enum fpga_event_type {\nFPGA_EVENT_INTERRUPT = 0,\nFPGA_EVENT_ERROR,\nFPGA_EVENT_POWER_THERMAL\n};\n
OPAE currently defines the following event types that applications can register for. Note that not all FPGA resources and target platforms may support all event types.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_interface","title":"enum fpga_interface","text":"OPAE plugin interface.
enum fpga_interface {\nFPGA_IFC_DFL = 0,\nFPGA_IFC_VFIO,\nFPGA_IFC_SIM_DFL,\nFPGA_IFC_SIM_VFIO,\nFPGA_IFC_UIO\n};\n
These are the supported plugin interfaces.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_metric_datatype","title":"enum fpga_metric_datatype","text":"enum fpga_metric_datatype {\nFPGA_METRIC_DATATYPE_INT,\nFPGA_METRIC_DATATYPE_FLOAT,\nFPGA_METRIC_DATATYPE_DOUBLE,\nFPGA_METRIC_DATATYPE_BOOL,\nFPGA_METRIC_DATATYPE_UNKNOWN\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_metric_type","title":"enum fpga_metric_type","text":"enum fpga_metric_type {\nFPGA_METRIC_TYPE_POWER,\nFPGA_METRIC_TYPE_THERMAL,\nFPGA_METRIC_TYPE_PERFORMANCE_CTR,\nFPGA_METRIC_TYPE_AFU,\nFPGA_METRIC_TYPE_UNKNOWN\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_objtype","title":"enum fpga_objtype","text":"OPAE FPGA resources (objects)
enum fpga_objtype {\nFPGA_DEVICE = 0,\nFPGA_ACCELERATOR\n};\n
These are the FPGA resources currently supported by the OPAE object model.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_open_flags","title":"enum fpga_open_flags","text":"Open flags.
enum fpga_open_flags {\nFPGA_OPEN_SHARED = (1u << 0)\n};\n
These flags can be passed to the fpgaOpen() function.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_reconf_flags","title":"enum fpga_reconf_flags","text":"Reconfiguration flags.
enum fpga_reconf_flags {\nFPGA_RECONF_FORCE = (1u << 0),\nFPGA_RECONF_SKIP_USRCLK = (1u << 1)\n};\n
These flags can be passed to the fpgaReconfigureSlot() function.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_result","title":"enum fpga_result","text":"OPAE C API function return codes.
enum fpga_result {\nFPGA_OK = 0,\nFPGA_INVALID_PARAM,\nFPGA_BUSY,\nFPGA_EXCEPTION,\nFPGA_NOT_FOUND,\nFPGA_NO_MEMORY,\nFPGA_NOT_SUPPORTED,\nFPGA_NO_DRIVER,\nFPGA_NO_DAEMON,\nFPGA_NO_ACCESS,\nFPGA_RECONF_ERROR\n};\n
Every public API function exported by the OPAE C library will return one of these codes. Usually, FPGA_OK denotes successful completion of the requested operation, while any return code other than FPGA_OK indicates an error or other deviation from the expected behavior. Users of the OPAE C API should always check the return codes of the APIs they call, and not use output parameters of functions that did not execute successfully.
The fpgaErrStr() function converts error codes into printable messages.
OPAE also has a logging mechanism that allows a developer to get more information about why a particular call failed with a specific message. If enabled, any function that returns an error code different from FPGA_OK will also print out a message with further details. This mechanism can be enabled by setting the environment variable LIBOPAE_LOG to 1 before running the respective application.
"},{"location":"opae-code/types__enum_8h/#enum-fpga_sysobject_flags","title":"enum fpga_sysobject_flags","text":"enum fpga_sysobject_flags {\nFPGA_OBJECT_SYNC = (1u << 0),\nFPGA_OBJECT_GLOB = (1u << 1),\nFPGA_OBJECT_RAW =\n(1u << 2),\nFPGA_OBJECT_RECURSE_ONE =\n(1u\n<< 3),\nFPGA_OBJECT_RECURSE_ALL =\n(1u\n<< 4)\n};\n
"},{"location":"opae-code/types__enum_8h/#enum-fpga_sysobject_type","title":"enum fpga_sysobject_type","text":"enum fpga_sysobject_type {\nFPGA_OBJECT_CONTAINER =\n(1u << 0),\nFPGA_OBJECT_ATTRIBUTE =\n(1u << 1)\n};\n
The documentation for this class was generated from the following file docs/sw/include/opae/types_enum.h
"},{"location":"opae-code/types__enum_8h_source/","title":"File types_enum.h","text":"File List > docs > sw > include > opae > types_enum.h
Go to the documentation of this file.
// Copyright(c) 2017-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_TYPES_ENUM_H__\n#define __FPGA_TYPES_ENUM_H__\ntypedef enum {\nFPGA_OK = 0, FPGA_INVALID_PARAM, FPGA_BUSY, FPGA_EXCEPTION, FPGA_NOT_FOUND, FPGA_NO_MEMORY, FPGA_NOT_SUPPORTED, FPGA_NO_DRIVER, FPGA_NO_DAEMON, FPGA_NO_ACCESS, FPGA_RECONF_ERROR } fpga_result;\ntypedef enum {\nFPGA_EVENT_INTERRUPT = 0, FPGA_EVENT_ERROR, FPGA_EVENT_POWER_THERMAL } fpga_event_type;\n/* TODO: consider adding lifecycle events in the future\n * to help with orchestration. Need a complete specification\n * before including them in the API. Proposed events:\n * FPGA_EVENT_APPEAR\n * FPGA_EVENT_DISAPPEAR\n * FPGA_EVENT_CHANGE\n */\ntypedef enum {\nFPGA_ACCELERATOR_ASSIGNED = 0,\nFPGA_ACCELERATOR_UNASSIGNED\n} fpga_accelerator_state;\ntypedef enum {\nFPGA_DEVICE = 0,\nFPGA_ACCELERATOR\n} fpga_objtype;\ntypedef enum {\nFPGA_IFC_DFL = 0,\nFPGA_IFC_VFIO,\nFPGA_IFC_SIM_DFL,\nFPGA_IFC_SIM_VFIO,\nFPGA_IFC_UIO,\n} fpga_interface;\nenum fpga_buffer_flags {\nFPGA_BUF_PREALLOCATED = (1u << 0), FPGA_BUF_QUIET = (1u << 1), FPGA_BUF_READ_ONLY = (1u << 2) };\nenum fpga_open_flags {\nFPGA_OPEN_SHARED = (1u << 0)\n};\nenum fpga_reconf_flags {\nFPGA_RECONF_FORCE = (1u << 0),\nFPGA_RECONF_SKIP_USRCLK = (1u << 1)\n};\nenum fpga_sysobject_flags {\nFPGA_OBJECT_SYNC = (1u << 0), FPGA_OBJECT_GLOB = (1u << 1), FPGA_OBJECT_RAW =\n(1u << 2), FPGA_OBJECT_RECURSE_ONE =\n(1u\n<< 3), FPGA_OBJECT_RECURSE_ALL =\n(1u\n<< 4) };\nenum fpga_sysobject_type {\nFPGA_OBJECT_CONTAINER =\n(1u << 0), FPGA_OBJECT_ATTRIBUTE =\n(1u << 1) };\nenum fpga_metric_type {\nFPGA_METRIC_TYPE_POWER, // Metric power\nFPGA_METRIC_TYPE_THERMAL, // Metric Thermal\nFPGA_METRIC_TYPE_PERFORMANCE_CTR, // Metric Performance counter\nFPGA_METRIC_TYPE_AFU, // Metric AFU\nFPGA_METRIC_TYPE_UNKNOWN // Unknown\n};\nenum fpga_metric_datatype {\nFPGA_METRIC_DATATYPE_INT, // Metric datatype integer\nFPGA_METRIC_DATATYPE_FLOAT, // Metric datatype float\nFPGA_METRIC_DATATYPE_DOUBLE, // Metric datatype double\nFPGA_METRIC_DATATYPE_BOOL, // Metric datatype bool\nFPGA_METRIC_DATATYPE_UNKNOWN // Metric datatype unknown\n};\n#endif // __FPGA_TYPES_ENUM_H__\n
"},{"location":"opae-code/uio_8h/","title":"File uio.h","text":"FileList > docs > sw > include > opae > uio.h
Go to the source code of this file.
APIs to manage a PCIe device via UIO. More...
#include <stdio.h>#include <stdint.h>
"},{"location":"opae-code/uio_8h/#classes","title":"Classes","text":"Type Name struct opae_uio OPAE UIO device abstraction. struct opae_uio_device_region MMIO region info."},{"location":"opae-code/uio_8h/#public-functions","title":"Public Functions","text":"Type Name void opae_uio_close (struct opae_uio * u) Release and close a UIO device. int opae_uio_open (struct opae_uio * u, const char * dfl_device) Open and populate a UIO device. int opae_uio_region_get (struct opae_uio * u, uint32_t index, uint8_t ** ptr, size_t * size) Query device MMIO region."},{"location":"opae-code/uio_8h/#macros","title":"Macros","text":"Type Name define OPAE_UIO_PATH_MAX 256"},{"location":"opae-code/uio_8h/#detailed-description","title":"Detailed Description","text":"Presents a simple interface for interacting with a DFL device that is bound to its UIO driver. See https://kernel.org/doc/html/v4.14/driver-api/uio-howto.html for a description of UIO.
Provides APIs for opening/closing the device and for querying info about the MMIO regions of the device.
"},{"location":"opae-code/uio_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/uio_8h/#function-opae_uio_close","title":"function opae_uio_close","text":"Release and close a UIO device.
void opae_uio_close (\nstruct opae_uio * u\n)
The given device pointer must have been previously initialized by opae_uio_open. Releases all data structures.
Parameters:
u Storage for the device info. May be stack-resident.
Example struct opae_uio u;
if (opae_uio_open(&u, \"dfl_dev.10\")) { // handle error } else { // interact with the device ... // free the device opae_uio_close(&u); }
Example $ sudo opaeuio -r dfl_dev.10
"},{"location":"opae-code/uio_8h/#function-opae_uio_open","title":"function opae_uio_open","text":"Open and populate a UIO device.
int opae_uio_open (\nstruct opae_uio * u,\nconst char * dfl_device\n)
Opens the Device Feature List device corresponding to the device name given in dfl_device, eg \"dfl_dev.10\". The device must be bound to the dfl-uio-pdev driver prior to opening it. The data structures corresponding to the MMIO regions are initialized.
Parameters:
u Storage for the device. May be stack-resident. dfl_device The name of the desired DFL device.
Returns:
Non-zero on error. Zero on success.
Example $ sudo opaeuio -i -u lab -g lab dfl_dev.10
Example struct opae_uio u;
if (opae_uio_open(&u, \"dfl_dev.10\")) { // handle error }
"},{"location":"opae-code/uio_8h/#function-opae_uio_region_get","title":"function opae_uio_region_get","text":"Query device MMIO region.
int opae_uio_region_get (\nstruct opae_uio * u,\nuint32_t index,\nuint8_t ** ptr,\nsize_t * size\n)
Retrieves info describing the MMIO region with the given region index. The device structure u must have been previously opened by a call to opae_uio_open.
Parameters:
u The open OPAE UIO device. index The zero-based index of the desired MMIO region. ptr Optional pointer to receive the virtual address for the region. Pass NULL to ignore. size Optional pointer to receive the size of the MMIO region. Pass NULL to ignore.
Returns:
Non-zero on error (including index out-of-range). Zero on success.
Example struct opae_uio u;
uint8_t *virt = NULL; size_t size = 0;
if (opae_uio_open(&u, \"dfl_dev.10\")) { // handle error } else { opae_uio_region_get(&u, 0, &virt, &size); }
"},{"location":"opae-code/uio_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/uio_8h/#define-opae_uio_path_max","title":"define OPAE_UIO_PATH_MAX","text":"#define OPAE_UIO_PATH_MAX 256\n
The documentation for this class was generated from the following file docs/sw/include/opae/uio.h
"},{"location":"opae-code/uio_8h_source/","title":"File uio.h","text":"File List > docs > sw > include > opae > uio.h
Go to the documentation of this file.
// Copyright(c) 2020, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_UIO_H__\n#define __OPAE_UIO_H__\n#include <stdio.h>\n#include <stdint.h>\n#define OPAE_UIO_PATH_MAX 256\nstruct opae_uio_device_region {\nuint32_t region_index;\nuint8_t *region_ptr;\nsize_t region_page_offset;\nsize_t region_size;\nstruct opae_uio_device_region *next;\n};\nstruct opae_uio {\nchar device_path[OPAE_UIO_PATH_MAX];\nint device_fd;\nstruct opae_uio_device_region *regions;\n};\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nint opae_uio_open(struct opae_uio *u,\nconst char *dfl_device);\nint opae_uio_region_get(struct opae_uio *u,\nuint32_t index,\nuint8_t **ptr,\nsize_t *size);\nvoid opae_uio_close(struct opae_uio *u);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __OPAE_UIO_H__\n
"},{"location":"opae-code/umsg_8h/","title":"File umsg.h","text":"FileList > docs > sw > include > opae > umsg.h
Go to the source code of this file.
FPGA UMsg API.
#include <opae/types.h>
"},{"location":"opae-code/umsg_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetNumUmsg (fpga_handle handle, uint64_t * value) Get number of Umsgs. fpga_result fpgaGetUmsgPtr (fpga_handle handle, uint64_t ** umsg_ptr) Access UMsg memory directly. fpga_result fpgaSetUmsgAttributes (fpga_handle handle, uint64_t value) Sets Umsg hint. fpga_result fpgaTriggerUmsg (fpga_handle handle, uint64_t value) Trigger Umsg."},{"location":"opae-code/umsg_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/umsg_8h/#function-fpgagetnumumsg","title":"function fpgaGetNumUmsg","text":"Get number of Umsgs.
fpga_result fpgaGetNumUmsg (\nfpga_handle handle,\nuint64_t * value\n)
Retuns number of umsg supported by AFU.
Parameters:
handle Handle to previously opened accelerator resource value Returns number of UMsgs
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid.
"},{"location":"opae-code/umsg_8h/#function-fpgagetumsgptr","title":"function fpgaGetUmsgPtr","text":"Access UMsg memory directly.
fpga_result fpgaGetUmsgPtr (\nfpga_handle handle,\nuint64_t ** umsg_ptr\n)
This function will return a pointer to the memory allocated for low latency accelerator notifications (UMsgs).
Parameters:
handle Handle to previously opened accelerator resource umsg_ptr Pointer to memory where a pointer to the virtual address space will be returned
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid. FPGA_NO_MEMORY if memory allocation fails or system doesn't configure huge pages.
"},{"location":"opae-code/umsg_8h/#function-fpgasetumsgattributes","title":"function fpgaSetUmsgAttributes","text":"Sets Umsg hint.
fpga_result fpgaSetUmsgAttributes (\nfpga_handle handle,\nuint64_t value\n)
Writes usmg hint bit.
Parameters:
handle Handle to previously opened accelerator resource value Value to use for UMsg hint, Umsg hit is N wide bitvector where N = number of Umsgs.
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid.
"},{"location":"opae-code/umsg_8h/#function-fpgatriggerumsg","title":"function fpgaTriggerUmsg","text":"Trigger Umsg.
fpga_result fpgaTriggerUmsg (\nfpga_handle handle,\nuint64_t value\n)
Writes a 64-bit value to trigger low-latency accelerator notification mechanism (UMsgs).
Parameters:
handle Handle to previously opened accelerator resource value Value to use for UMsg
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if input parameter combination is not valid. FPGA_EXCEPTION if input parameter fpga handle is not valid.
The documentation for this class was generated from the following file docs/sw/include/opae/umsg.h
"},{"location":"opae-code/umsg_8h_source/","title":"File umsg.h","text":"File List > docs > sw > include > opae > umsg.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_UMSG_H__\n#define __FPGA_UMSG_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetNumUmsg(fpga_handle handle, uint64_t *value);\nfpga_result fpgaSetUmsgAttributes(fpga_handle handle,\nuint64_t value);\nfpga_result fpgaTriggerUmsg(fpga_handle handle, uint64_t value);\nfpga_result fpgaGetUmsgPtr(fpga_handle handle, uint64_t **umsg_ptr);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_UMSG_H__\n
"},{"location":"opae-code/userclk_8h/","title":"File userclk.h","text":"FileList > docs > sw > include > opae > userclk.h
Go to the source code of this file.
Functions for setting and get afu user clock.
#include <opae/types.h>
"},{"location":"opae-code/userclk_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetUserClock (fpga_handle handle, uint64_t * high_clk, uint64_t * low_clk, int flags) Get afu user clock high and low. fpga_result fpgaSetUserClock (fpga_handle handle, uint64_t high_clk, uint64_t low_clk, int flags) set afu user clock high and low"},{"location":"opae-code/userclk_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/userclk_8h/#function-fpgagetuserclock","title":"function fpgaGetUserClock","text":"Get afu user clock high and low.
fpga_result fpgaGetUserClock (\nfpga_handle handle,\nuint64_t * high_clk,\nuint64_t * low_clk,\nint flags\n)
Parameters:
handle Handle to previously opened accelerator resource. high_clk AFU High user clock frequency in MHz. low_clk AFU Low user clock frequency in MHz. flags Flags Reserved.
.*
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
"},{"location":"opae-code/userclk_8h/#function-fpgasetuserclock","title":"function fpgaSetUserClock","text":"set afu user clock high and low
fpga_result fpgaSetUserClock (\nfpga_handle handle,\nuint64_t high_clk,\nuint64_t low_clk,\nint flags\n)
Parameters:
handle Handle to previously opened accelerator resource. high_clk AFU High user clock frequency in MHz. low_clk AFU Low user clock frequency in MHz. flags Flags Reserved.
.*
Returns:
FPGA_OK on success. FPGA_INVALID_PARAM if invalid parameters were provided, or if the parameter combination is not valid. FPGA_EXCEPTION if an internal exception occurred while trying to access the handle.
The documentation for this class was generated from the following file docs/sw/include/opae/userclk.h
"},{"location":"opae-code/userclk_8h_source/","title":"File userclk.h","text":"File List > docs > sw > include > opae > userclk.h
Go to the documentation of this file.
// Copyright(c) 2018, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_AFU_USER_CLOCK_H__\n#define __FPGA_AFU_USER_CLOCK_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaSetUserClock(fpga_handle handle,\nuint64_t high_clk, uint64_t low_clk, int flags);\nfpga_result fpgaGetUserClock(fpga_handle handle,\nuint64_t *high_clk, uint64_t *low_clk, int flags);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_AFU_USER_CLOCK_H__\n
"},{"location":"opae-code/utils_8h/","title":"File utils.h","text":"FileList > docs > sw > include > opae > utils.h
Go to the source code of this file.
Utility functions and macros for the FPGA API.
#include <opae/types.h>#include <stdio.h>
"},{"location":"opae-code/utils_8h/#public-functions","title":"Public Functions","text":"Type Name const char * fpgaErrStr (fpga_result e) Return human-readable error message."},{"location":"opae-code/utils_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/utils_8h/#function-fpgaerrstr","title":"function fpgaErrStr","text":"Return human-readable error message.
const char * fpgaErrStr (\nfpga_result e\n)
Returns a pointer to a human-readable error message corresponding to the provided fpga_error error code.
Parameters:
e Error code (as returned by another FPGA API function
Returns:
Pointer to a descriptive error message string
The documentation for this class was generated from the following file docs/sw/include/opae/utils.h
"},{"location":"opae-code/utils_8h_source/","title":"File utils.h","text":"File List > docs > sw > include > opae > utils.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_UTILS_H__\n#define __FPGA_UTILS_H__\n#include <opae/types.h>\n#include <stdio.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nconst char *fpgaErrStr(fpga_result e);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_UTILS_H__\n
"},{"location":"opae-code/version_8h/","title":"File version.h","text":"FileList > docs > sw > include > opae > version.h
Go to the source code of this file.
#include <opae/types.h>
"},{"location":"opae-code/version_8h/#public-functions","title":"Public Functions","text":"Type Name fpga_result fpgaGetOPAECBuildString (char * build_str, size_t len) Get build information about the OPAE library as a string. fpga_result fpgaGetOPAECVersion (fpga_version * version) Get version information about the OPAE library. fpga_result fpgaGetOPAECVersionString (char * version_str, size_t len) Get version information about the OPAE library as a string."},{"location":"opae-code/version_8h/#macros","title":"Macros","text":"Type Name define FPGA_BUILD_STR_MAX 41 define FPGA_VERSION_STR_MAX 10"},{"location":"opae-code/version_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/version_8h/#function-fpgagetopaecbuildstring","title":"function fpgaGetOPAECBuildString","text":"Get build information about the OPAE library as a string.
fpga_result fpgaGetOPAECBuildString (\nchar * build_str,\nsize_t len\n)
Retrieve the build identifier of the OPAE library.
Parameters:
build_str String to copy build information into len Length of build_str
Returns:
FPGA_INVALID_PARAM if build_str is NULL, FPGA_EXCEPTION if the version string cannot be copied into build_str, FPGA_OK otherwise.
"},{"location":"opae-code/version_8h/#function-fpgagetopaecversion","title":"function fpgaGetOPAECVersion","text":"Get version information about the OPAE library.
fpga_result fpgaGetOPAECVersion (\nfpga_version * version\n)
Retrieve major version, minor version, and revision information about the OPAE library.
Parameters:
version FPGA version
Returns:
FPGA_INVALID_PARAM if any of the output parameters is NULL, FPGA_OK otherwise.
"},{"location":"opae-code/version_8h/#function-fpgagetopaecversionstring","title":"function fpgaGetOPAECVersionString","text":"Get version information about the OPAE library as a string.
fpga_result fpgaGetOPAECVersionString (\nchar * version_str,\nsize_t len\n)
Retrieve major version, minor version, and revision information about the OPAE library, encoded in a human-readable string (e.g. \"1.0.0\").
Parameters:
version_str String to copy version information into len Length of version_str
Returns:
FPGA_INVALID_PARAM if version_str is NULL, FPGA_EXCEPTION if the version string cannot be copied into version_str, FPGA_OK otherwise.
"},{"location":"opae-code/version_8h/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/version_8h/#define-fpga_build_str_max","title":"define FPGA_BUILD_STR_MAX","text":"#define FPGA_BUILD_STR_MAX 41\n
"},{"location":"opae-code/version_8h/#define-fpga_version_str_max","title":"define FPGA_VERSION_STR_MAX","text":"#define FPGA_VERSION_STR_MAX 10\n
The documentation for this class was generated from the following file docs/sw/include/opae/version.h
"},{"location":"opae-code/version_8h_source/","title":"File version.h","text":"File List > docs > sw > include > opae > version.h
Go to the documentation of this file.
// Copyright(c) 2017, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __FPGA_VERSION_H__\n#define __FPGA_VERSION_H__\n#include <opae/types.h>\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nfpga_result fpgaGetOPAECVersion(fpga_version *version);\nfpga_result fpgaGetOPAECVersionString(char *version_str, size_t len);\n#define FPGA_VERSION_STR_MAX 10\nfpga_result fpgaGetOPAECBuildString(char *build_str, size_t len);\n#define FPGA_BUILD_STR_MAX 41\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __FPGA_VERSION_H__\n
"},{"location":"opae-code/vfio_8h/","title":"File vfio.h","text":"FileList > docs > sw > include > opae > vfio.h
Go to the source code of this file.
APIs to manage a PCIe device via vfio-pci. More...
#include <stdio.h>#include <stdint.h>#include <pthread.h>#include <linux/vfio.h>#include <opae/mem_alloc.h>#include <opae/hash_map.h>
"},{"location":"opae-code/vfio_8h/#classes","title":"Classes","text":"Type Name struct opae_vfio OPAE VFIO device abstraction. struct opae_vfio_buffer System DMA buffer. struct opae_vfio_device VFIO device. struct opae_vfio_device_irq Interrupt info. struct opae_vfio_device_region MMIO region info. struct opae_vfio_group VFIO group. struct opae_vfio_iova_range IO Virtual Address Range. struct opae_vfio_sparse_info MMIO sparse-mappable region info."},{"location":"opae-code/vfio_8h/#public-types","title":"Public Types","text":"Type Name enum opae_vfio_buffer_flags Flags for opae_vfio_buffer_allocate_ex() ."},{"location":"opae-code/vfio_8h/#public-functions","title":"Public Functions","text":"Type Name int opae_vfio_buffer_allocate (struct opae_vfio * v, size_t * size, uint8_t ** buf, uint64_t * iova) Allocate and map system buffer. int opae_vfio_buffer_allocate_ex (struct opae_vfio * v, size_t * size, uint8_t ** buf, uint64_t * iova, int flags) Allocate and map system buffer (extended w/ flags) int opae_vfio_buffer_free (struct opae_vfio * v, uint8_t * buf) Unmap and free a system buffer. struct opae_vfio_buffer * opae_vfio_buffer_info (struct opae_vfio * v, uint8_t * vaddr) Extract the internal data structure pointer for the given vaddr. void opae_vfio_close (struct opae_vfio * v) Release and close a VFIO device. int opae_vfio_irq_disable (struct opae_vfio * v, uint32_t index, uint32_t subindex) Disable an IRQ. int opae_vfio_irq_enable (struct opae_vfio * v, uint32_t index, uint32_t subindex, int event_fd) Enable an IRQ. int opae_vfio_irq_mask (struct opae_vfio * v, uint32_t index, uint32_t subindex) Mask an IRQ. int opae_vfio_irq_unmask (struct opae_vfio * v, uint32_t index, uint32_t subindex) Unmask an IRQ. int opae_vfio_open (struct opae_vfio * v, const char * pciaddr) Open and populate a VFIO device. int opae_vfio_region_get (struct opae_vfio * v, uint32_t index, uint8_t ** ptr, size_t * size) Query device MMIO region. int opae_vfio_secure_open (struct opae_vfio * v, const char * pciaddr, const char * token) Open and populate a VFIO device."},{"location":"opae-code/vfio_8h/#detailed-description","title":"Detailed Description","text":"Presents a simple interface for interacting with a PCIe device that is bound to the vfio-pci driver. See https://kernel.org/doc/Documentation/vfio.txt for a description of vfio-pci.
Provides APIs for opening/closing the device, querying info about the MMIO regions of the device, and allocating/mapping and freeing/unmapping DMA buffers.
"},{"location":"opae-code/vfio_8h/#public-types-documentation","title":"Public Types Documentation","text":""},{"location":"opae-code/vfio_8h/#enum-opae_vfio_buffer_flags","title":"enum opae_vfio_buffer_flags","text":"enum opae_vfio_buffer_flags {\nOPAE_VFIO_BUF_PREALLOCATED = 1\n};\n
"},{"location":"opae-code/vfio_8h/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_allocate","title":"function opae_vfio_buffer_allocate","text":"Allocate and map system buffer.
int opae_vfio_buffer_allocate (\nstruct opae_vfio * v,\nsize_t * size,\nuint8_t ** buf,\nuint64_t * iova\n)
Allocate, map, and retrieve info for a system buffer capable of DMA. Saves an entry in the v->cont_buffers list. If the buffer is not explicitly freed by opae_vfio_buffer_free, it will be freed during opae_vfio_close.
mmap is used for the allocation. If the size is greater than 2MB, then the allocation request is fulfilled by a 1GB huge page. Else, if the size is greater than 4096, then the request is fulfilled by a 2MB huge page. Else, the request is fulfilled by the non-huge page pool.
Note:
Allocations from the huge page pool require that huge pages be configured on the system. Huge pages may be configured on the kernel boot command prompt. Example default_hugepagesz=1G hugepagesz=1G hugepages=2 hugepagesz=2M hugepages=8
Note:
Huge pages may also be configured at runtime. Example sudo sh -c 'echo 8 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages' sudo sh -c 'echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'
Note:
Be sure that the IOMMU is also enabled using the follow kernel boot command: intel_iommu=on
Parameters:
v The open OPAE VFIO device. size A pointer to the requested size. The size may be rounded to the next page size prior to return from the function. buf Optional pointer to receive the virtual address for the buffer. Pass NULL to ignore. iova Optional pointer to receive the IOVA address for the buffer. Pass NULL to ignore.
Returns:
Non-zero on error. Zero on success.
Example opae_vfio v;
size_t sz; uint8_t *buf_2m_virt = NULL; uint8_t *buf_1g_virt = NULL; uint64_t buf_2m_iova = 0; uint64_t buf_1g_iova = 0;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { sz = 2 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_2m_virt, &buf_2m_iova)) { // handle allocation error }
sz = 1024 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_1g_virt, &buf_1g_iova)) { // handle allocation error } }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_allocate_ex","title":"function opae_vfio_buffer_allocate_ex","text":"Allocate and map system buffer (extended w/ flags)
int opae_vfio_buffer_allocate_ex (\nstruct opae_vfio * v,\nsize_t * size,\nuint8_t ** buf,\nuint64_t * iova,\nint flags\n)
Allocate, map, and retrieve info for a system buffer capable of DMA. Saves an entry in the v->cont_buffers list. If the buffer is not explicitly freed by opae_vfio_buffer_free, it will be freed during opae_vfio_close, unless OPAE_VFIO_BUF_PREALLOCATED is used in which case the buffer is not freed by this library.
When not using OPAE_VFIO_BUF_PREALLOCATED, mmap is used for the allocation. If the size is greater than 2MB, then the allocation request is fulfilled by a 1GB huge page. Else, if the size is greater than 4096, then the request is fulfilled by a 2MB huge page. Else, the request is fulfilled by the non-huge page pool.
Parameters:
v The open OPAE VFIO device. size A pointer to the requested size. The size may be rounded to the next page size prior to return from the function. buf Optional pointer to receive the virtual address for the buffer/input buffer pointer when using OPAE_VFIO_BUF_PREALLOCATED. Pass NULL to ignore. iova Optional pointer to receive the IOVA address for the buffer. Pass NULL to ignore.
Returns:
Non-zero on error. Zero on success.
Example opae_vfio v;
size_t sz = MY_BUF_SIZE; uint8_t *prealloc_virt = NULL; uint64_t iova = 0;
prealloc_virt = allocate_my_buffer(sz);
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { if (opae_vfio_buffer_allocate_ex(&v, &sz, &prealloc_virt, &iova, OPAE_VFIO_BUF_PREALLOCATED)) { // handle allocation error } }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_free","title":"function opae_vfio_buffer_free","text":"Unmap and free a system buffer.
int opae_vfio_buffer_free (\nstruct opae_vfio * v,\nuint8_t * buf\n)
The buffer corresponding to buf must have been created by a previous call to opae_vfio_buffer_allocate.
Parameters:
v The open OPAE VFIO device. buf The virtual address corresponding to the buffer to be freed.
Returns:
Non-zero on error. Zero on success.
Example size_t sz; uint8_t *buf_2m_virt = NULL; uint64_t buf_2m_iova = 0;
sz = 2 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_2m_virt, &buf_2m_iova)) { // handle allocation error } else { // use the buffer
if (opae_vfio_buffer_free(&v, buf_2m_virt)) { // handle free error } }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_buffer_info","title":"function opae_vfio_buffer_info","text":"Extract the internal data structure pointer for the given vaddr.
struct opae_vfio_buffer * opae_vfio_buffer_info (\nstruct opae_vfio * v,\nuint8_t * vaddr\n)
The virtual address vaddr must correspond to a buffer previously allocated by opae_vfio_buffer_allocate() or opae_vfio_buffer_allocate_ex().
Parameters:
v The open OPAE VFIO device. vaddr The user virtual address of the desired buffer info struct.
Returns:
NULL on lookup error.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_close","title":"function opae_vfio_close","text":"Release and close a VFIO device.
void opae_vfio_close (\nstruct opae_vfio * v\n)
The given device pointer must have been previously initialized by opae_vfio_open. Releases all data structures, including any DMA buffer allocations that have not be explicitly freed by opae_vfio_buffer_free.
Parameters:
v Storage for the device info. May be stack-resident.
Example opae_vfio v;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { // interact with the device ... // free the device opae_vfio_close(&v); }
Example $ sudo opaevfio -r 0000:00:00.0
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_disable","title":"function opae_vfio_irq_disable","text":"Disable an IRQ.
int opae_vfio_irq_disable (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to disable.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_enable","title":"function opae_vfio_irq_enable","text":"Enable an IRQ.
int opae_vfio_irq_enable (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex,\nint event_fd\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to enable. event_fd The file descriptor, created by eventfd(). Interrupts will result in this event_fd being signaled.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_mask","title":"function opae_vfio_irq_mask","text":"Mask an IRQ.
int opae_vfio_irq_mask (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to mask.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_irq_unmask","title":"function opae_vfio_irq_unmask","text":"Unmask an IRQ.
int opae_vfio_irq_unmask (\nstruct opae_vfio * v,\nuint32_t index,\nuint32_t subindex\n)
Parameters:
v The open OPAE VFIO device. index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX. subindex The IRQ to unmask.
Returns:
Non-zero on error. Zero on success.
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_open","title":"function opae_vfio_open","text":"Open and populate a VFIO device.
int opae_vfio_open (\nstruct opae_vfio * v,\nconst char * pciaddr\n)
Opens the PCIe device corresponding to the address given in pciaddr. The device must be bound to the vfio-pci driver prior to opening it. The data structures corresponding to IOVA space, MMIO regions, and DMA buffers are initialized.
Parameters:
v Storage for the device info. May be stack-resident. pciaddr The PCIe address of the requested device.
Returns:
Non-zero on error. Zero on success.
Example $ sudo opaevfio -i 0000:00:00.0 -u user -g group
Example opae_vfio v;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_region_get","title":"function opae_vfio_region_get","text":"Query device MMIO region.
int opae_vfio_region_get (\nstruct opae_vfio * v,\nuint32_t index,\nuint8_t ** ptr,\nsize_t * size\n)
Retrieves info describing the MMIO region with the given region index. The device structure v must have been previously opened by a call to opae_vfio_open.
Parameters:
v The open OPAE VFIO device. index The zero-based index of the desired MMIO region. ptr Optional pointer to receive the virtual address for the region. Pass NULL to ignore. size Optional pointer to receive the size of the MMIO region. Pass NULL to ignore.
Returns:
Non-zero on error (including index out-of-range). Zero on success.
Example opae_vfio v;
uint8_t *fme_virt = NULL; uint8_t *port_virt = NULL; size_t fme_size = 0; size_t port_size = 0;
if (opae_vfio_open(&v, \"0000:00:00.0\")) { // handle error } else { opae_vfio_region_get(&v, 0, &fme_virt, &fme_size); opae_vfio_region_get(&v, 2, &port_virt, &port_size); }
"},{"location":"opae-code/vfio_8h/#function-opae_vfio_secure_open","title":"function opae_vfio_secure_open","text":"Open and populate a VFIO device.
int opae_vfio_secure_open (\nstruct opae_vfio * v,\nconst char * pciaddr,\nconst char * token\n)
Opens the PCIe device corresponding to the address given in pciaddr, using the VF token (GUID) given in token. The device must be bound to the vfio-pci driver prior to opening it. The data structures corresponding to IOVA space, MMIO regions, and DMA buffers are initialized.
Parameters:
v Storage for the device info. May be stack-resident. pciaddr The PCIe address of the requested device. token The GUID representing the VF token.
Returns:
Non-zero on error. Zero on success.
Example $ sudo opaevfio -i 0000:00:00.0 -u user -g group
Example opae_vfio v;
if (opae_vfio_secure_open(&v, \"0000:00:00.0\", \"00f5ad6b-2edd-422e-9d1e-34124c686fec\")) { // handle error }
The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h
"},{"location":"opae-code/vfio_8h_source/","title":"File vfio.h","text":"File List > docs > sw > include > opae > vfio.h
Go to the documentation of this file.
// Copyright(c) 2020-2023, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifndef __OPAE_VFIO_H__\n#define __OPAE_VFIO_H__\n#include <stdio.h>\n#include <stdint.h>\n#include <pthread.h>\n#include <linux/vfio.h>\n#include <opae/mem_alloc.h>\n#include <opae/hash_map.h>\nstruct opae_vfio_iova_range {\nuint64_t start; uint64_t end; struct opae_vfio_iova_range *next; };\nstruct opae_vfio_group {\nchar *group_device; int group_fd; };\nstruct opae_vfio_sparse_info {\nuint32_t index; uint32_t offset; uint32_t size; uint8_t *ptr; struct opae_vfio_sparse_info *next; };\nstruct opae_vfio_device_region {\nuint32_t region_index; uint8_t *region_ptr; size_t region_size; struct opae_vfio_sparse_info *region_sparse; struct opae_vfio_device_region *next; };\nstruct opae_vfio_device_irq {\nuint32_t flags; uint32_t index; uint32_t count; int32_t *event_fds; int32_t *masks; struct opae_vfio_device_irq *next; };\nstruct opae_vfio_device {\nint device_fd; uint64_t device_config_offset; uint32_t device_num_regions; struct opae_vfio_device_region *regions; uint32_t device_num_irqs; struct opae_vfio_device_irq *irqs; };\nstruct opae_vfio_buffer {\nuint8_t *buffer_ptr; size_t buffer_size; uint64_t buffer_iova; int flags; };\nstruct opae_vfio {\npthread_mutex_t lock; char *cont_device; char *cont_pciaddr; int cont_fd; struct opae_vfio_iova_range *cont_ranges; struct mem_alloc iova_alloc; struct opae_vfio_group group; struct opae_vfio_device device; opae_hash_map cont_buffers; };\n#ifdef __cplusplus\nextern \"C\" {\n#endif\nint opae_vfio_open(struct opae_vfio *v,\nconst char *pciaddr);\nint opae_vfio_secure_open(struct opae_vfio *v,\nconst char *pciaddr,\nconst char *token);\nint opae_vfio_region_get(struct opae_vfio *v,\nuint32_t index,\nuint8_t **ptr,\nsize_t *size);\nint opae_vfio_buffer_allocate(struct opae_vfio *v,\nsize_t *size,\nuint8_t **buf,\nuint64_t *iova);\nenum opae_vfio_buffer_flags {\nOPAE_VFIO_BUF_PREALLOCATED = 1, };\nint opae_vfio_buffer_allocate_ex(struct opae_vfio *v,\nsize_t *size,\nuint8_t **buf,\nuint64_t *iova,\nint flags);\nstruct opae_vfio_buffer *opae_vfio_buffer_info(struct opae_vfio *v,\nuint8_t *vaddr);\nint opae_vfio_buffer_free(struct opae_vfio *v,\nuint8_t *buf);\nint opae_vfio_irq_enable(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex,\nint event_fd);\nint opae_vfio_irq_unmask(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex);\nint opae_vfio_irq_mask(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex);\nint opae_vfio_irq_disable(struct opae_vfio *v,\nuint32_t index,\nuint32_t subindex);\nvoid opae_vfio_close(struct opae_vfio *v);\n#ifdef __cplusplus\n} // extern \"C\"\n#endif // __cplusplus\n#endif // __OPAE_VFIO_H__\n
"},{"location":"opae-code/dir_9a6968a8846ef48cff617fcd6355d7b4/","title":"Dir docs/sw/samples","text":"FileList > docs > sw > samples
"},{"location":"opae-code/dir_9a6968a8846ef48cff617fcd6355d7b4/#directories","title":"Directories","text":"Type Name dir hello_events dir hello_fpga The documentation for this class was generated from the following file docs/sw/samples/
"},{"location":"opae-code/dir_d66a8e4b979fa79493bebe26e2602d2b/","title":"Dir docs/sw/samples/hello_events","text":"FileList > docs > sw > samples > hello_events
"},{"location":"opae-code/dir_d66a8e4b979fa79493bebe26e2602d2b/#files","title":"Files","text":"Type Name file hello_events.c A code sample of using OPAE event API. The documentation for this class was generated from the following file docs/sw/samples/hello_events/
"},{"location":"opae-code/hello__events_8c/","title":"File hello_events.c","text":"FileList > docs > sw > samples > hello_events > hello_events.c
Go to the source code of this file.
A code sample of using OPAE event API. More...
#include <stdio.h>#include <stdlib.h>#include <string.h>#include <unistd.h>#include <getopt.h>#include <poll.h>#include <errno.h>#include <sys/stat.h>#include <pthread.h>#include <opae/fpga.h>#include <argsfilter.h>#include \"mock/opae_std.h\"
"},{"location":"opae-code/hello__events_8c/#classes","title":"Classes","text":"Type Name struct ras_inject_error"},{"location":"opae-code/hello__events_8c/#public-functions","title":"Public Functions","text":"Type Name void * error_thread (void * arg) fpga_result find_fpga (fpga_properties device_filter, fpga_token * fpga, uint32_t * num_matches) void help (void) fpga_result inject_ras_fatal_error (fpga_token fme_token, uint8_t err) int main (int argc, char * argv) fpga_result parse_args (int argc, char * argv) void print_err (const char * s, fpga_result res) int usleep (unsigned)"},{"location":"opae-code/hello__events_8c/#macros","title":"Macros","text":"Type Name define FME_SYSFS_INJECT_ERROR \"errors/inject_errors\" define GETOPT_STRING \"hv\" define ON_ERR_GOTO (res, label, desc)"},{"location":"opae-code/hello__events_8c/#detailed-description","title":"Detailed Description","text":"This sample starts two processes. One process injects an artificial fatal error to sysfs; while the other tries to asynchronously capture and handle the event. This sample code exercises all major functions of the event API, including creating and destroying event handles, register and unregister events, polling on event file descriptor, and getting the OS object associated with an event. For a full discussion of OPAE event API, refer to event.h.
"},{"location":"opae-code/hello__events_8c/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/hello__events_8c/#function-error_thread","title":"function error_thread","text":"void * error_thread (\nvoid * arg\n)
"},{"location":"opae-code/hello__events_8c/#function-find_fpga","title":"function find_fpga","text":"fpga_result find_fpga (\nfpga_properties device_filter,\nfpga_token * fpga,\nuint32_t * num_matches\n)
"},{"location":"opae-code/hello__events_8c/#function-help","title":"function help","text":"void help (\nvoid\n)
"},{"location":"opae-code/hello__events_8c/#function-inject_ras_fatal_error","title":"function inject_ras_fatal_error","text":"fpga_result inject_ras_fatal_error (\nfpga_token fme_token,\nuint8_t err\n)
"},{"location":"opae-code/hello__events_8c/#function-main","title":"function main","text":"int main (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__events_8c/#function-parse_args","title":"function parse_args","text":"fpga_result parse_args (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__events_8c/#function-print_err","title":"function print_err","text":"void print_err (\nconst char * s,\nfpga_result res\n)
"},{"location":"opae-code/hello__events_8c/#function-usleep","title":"function usleep","text":"int usleep (\nunsigned\n)
"},{"location":"opae-code/hello__events_8c/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/hello__events_8c/#define-fme_sysfs_inject_error","title":"define FME_SYSFS_INJECT_ERROR","text":"#define FME_SYSFS_INJECT_ERROR \"errors/inject_errors\"\n
"},{"location":"opae-code/hello__events_8c/#define-getopt_string","title":"define GETOPT_STRING","text":"#define GETOPT_STRING \"hv\"\n
"},{"location":"opae-code/hello__events_8c/#define-on_err_goto","title":"define ON_ERR_GOTO","text":"#define ON_ERR_GOTO (\nres,\nlabel,\ndesc\n) do { \\\n if ((res) != FPGA_OK ) { \\ print_err ((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\n
The documentation for this class was generated from the following file docs/sw/samples/hello_events/hello_events.c
"},{"location":"opae-code/hello__events_8c_source/","title":"File hello_events.c","text":"File List > docs > sw > samples > hello_events > hello_events.c
Go to the documentation of this file.
// Copyright(c) 2017-2021, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifdef HAVE_CONFIG_H\n#include <config.h>\n#endif // HAVE_CONFIG_H\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n#include <unistd.h>\n#include <getopt.h>\n#include <poll.h>\n#include <errno.h>\n#include <sys/stat.h>\n#include <pthread.h>\n#include <opae/fpga.h>\n#include <argsfilter.h>\n#include \"mock/opae_std.h\"\nint usleep(unsigned);\n#define FME_SYSFS_INJECT_ERROR \"errors/inject_errors\"\n#define ON_ERR_GOTO(res, label, desc) \\\n do { \\\n if ((res) != FPGA_OK) { \\\n print_err((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\nvoid print_err(const char *s, fpga_result res)\n{\nfprintf(stderr, \"Error %s: %s\\n\", s, fpgaErrStr(res));\n}\n// RAS Error Inject CSR\nstruct ras_inject_error {\nunion {\nuint64_t csr;\nstruct {\n/* Catastrophic error */\nuint64_t catastrophicr_error : 1;\n/* Fatal error */\nuint64_t fatal_error : 1;\n/* Non-fatal error */\nuint64_t nonfatal_error : 1;\n/* Reserved */\nuint64_t rsvd : 61;\n};\n};\n};\nfpga_result inject_ras_fatal_error(fpga_token fme_token, uint8_t err)\n{\nfpga_result res1 = FPGA_OK;\nfpga_result res2 = FPGA_OK;\nfpga_handle fme_handle = NULL;\nstruct ras_inject_error inj_error = { {0} };\nfpga_object inj_err_object;\nres1 = fpgaOpen(fme_token, &fme_handle, FPGA_OPEN_SHARED);\nif (res1 != FPGA_OK) {\nOPAE_ERR(\"Failed to open FPGA\");\nreturn res1;\n}\nres1 = fpgaHandleGetObject(fme_handle, FME_SYSFS_INJECT_ERROR, &inj_err_object, 0);\nON_ERR_GOTO(res1, out_close, \"Failed to get Handle Object\");\n// Inject fatal error\ninj_error.fatal_error = err;\nres1 = fpgaObjectWrite64(inj_err_object, inj_error.csr, 0);\nON_ERR_GOTO(res1, out_destroy_obj, \"Failed to Read Object\");\nout_destroy_obj:\nres2 = fpgaDestroyObject(&inj_err_object);\nON_ERR_GOTO(res2, out_close, \"Failed to Destroy Object\");\nout_close:\nres2 = fpgaClose(fme_handle);\nif (res2 != FPGA_OK) {\nOPAE_ERR(\"Failed to close FPGA\");\n}\nreturn res1 != FPGA_OK ? res1 : res2;\n}\nvoid *error_thread(void *arg)\n{\nfpga_token token = (fpga_token) arg;\nfpga_result res;\nusleep(5000000);\nprintf(\"injecting error\\n\");\nres = inject_ras_fatal_error(token, 1);\nif (res != FPGA_OK)\nprint_err(\"setting inject error register\", res);\nusleep(5000000);\nprintf(\"clearing error\\n\");\nres = inject_ras_fatal_error(token, 0);\nif (res != FPGA_OK)\nprint_err(\"clearing inject error register\", res);\nreturn NULL;\n}\nvoid help(void)\n{\nprintf(\"\\n\"\n\"hello_events\\n\"\n\"OPAE Events API sample\\n\"\n\"\\n\"\n\"Usage:\\n\"\n\" hello_events [-hv] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\\n\"\n\"\\n\"\n\" -h,--help Print this help\\n\"\n\" -v,--version Print version info and exit\\n\"\n\"\\n\");\n}\n/*\n * Parse command line arguments\n */\n#define GETOPT_STRING \"hv\"\nfpga_result parse_args(int argc, char *argv[])\n{\nstruct option longopts[] = {\n{ \"help\", no_argument, NULL, 'h' },\n{ \"version\", no_argument, NULL, 'v' },\n{ NULL, 0, NULL, 0 }\n};\nint getopt_ret;\nint option_index;\nwhile (-1 != (getopt_ret = getopt_long(argc, argv, GETOPT_STRING, longopts, &option_index))) {\nconst char *tmp_optarg = optarg;\n/* checks to see if optarg is null and if not goes to value of optarg */\nif ((optarg) && ('=' == *tmp_optarg)) {\n++tmp_optarg;\n}\nswitch (getopt_ret) {\ncase 'h': /* help */\nhelp();\nreturn -1;\ncase 'v': /* version */\nprintf(\"hello_events %s %s%s\\n\",\nOPAE_VERSION,\nOPAE_GIT_COMMIT_HASH,\nOPAE_GIT_SRC_TREE_DIRTY ? \"*\":\"\");\nreturn -1;\ndefault: /* invalid option */\nfprintf(stderr, \"Invalid cmdline options\\n\");\nreturn FPGA_EXCEPTION;\n}\n}\nreturn FPGA_OK;\n}\nfpga_result find_fpga(fpga_properties device_filter,\nfpga_token *fpga,\nuint32_t *num_matches)\n{\nfpga_properties filter = NULL;\nfpga_result res = FPGA_OK;\nfpga_result dres = FPGA_OK;\nres = fpgaCloneProperties(device_filter, &filter);\nON_ERR_GOTO(res, out, \"cloning properties object\");\nres = fpgaPropertiesSetObjectType(filter, FPGA_DEVICE);\nON_ERR_GOTO(res, out_destroy, \"setting interface ID\");\nres = fpgaEnumerate(&filter, 1, fpga, 1, num_matches);\nON_ERR_GOTO(res, out_destroy, \"enumerating FPGAs\");\nout_destroy:\ndres = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(dres, out, \"destroying properties object\");\nout:\nreturn (res == FPGA_OK) ? dres : res;\n}\nint main(int argc, char *argv[])\n{\nfpga_token fpga_device_token = NULL;\nfpga_handle fpga_device_handle = NULL;\nuint32_t num_matches = 1;\nfpga_result res1 = FPGA_OK;\nfpga_result res2 = FPGA_OK;\nfpga_event_handle eh;\nuint64_t count = 0;\nint res;\nstruct pollfd pfd;\nint timeout = 10000;\nint poll_ret = 0;\nssize_t bytes_read = 0;\npthread_t errthr;\nfpga_properties device_filter = NULL;\nres1 = fpgaGetProperties(NULL, &device_filter);\nif (res1) {\nprint_err(\"failed to alloc properties\", res1);\nreturn res1;\n}\nif (opae_set_properties_from_args(device_filter,\n&res1,\n&argc,\nargv)) {\nprint_err(\"failed arg parse\", res1);\nres1 = FPGA_EXCEPTION;\ngoto out_exit;\n} else if (res1) {\nprint_err(\"failed to set properties\", res1);\ngoto out_exit;\n}\nres1 = parse_args(argc, argv);\nif ((int)res1 < 0)\ngoto out_exit;\nON_ERR_GOTO(res1, out_exit, \"parsing arguments\");\nres1 = find_fpga(device_filter, &fpga_device_token, &num_matches);\nON_ERR_GOTO(res1, out_exit, \"finding FPGA accelerator\");\nif (num_matches < 1) {\nfprintf(stderr, \"No matches for address provided.\\n\");\nres1 = FPGA_NOT_FOUND;\ngoto out_exit;\n}\nif (num_matches > 1) {\nfprintf(stderr, \"Found more than one suitable slot.\\n\");\n}\nres1 = fpgaOpen(fpga_device_token, &fpga_device_handle, FPGA_OPEN_SHARED);\nON_ERR_GOTO(res1, out_destroy_tok, \"opening accelerator\");\nres1 = fpgaCreateEventHandle(&eh);\nON_ERR_GOTO(res1, out_close, \"creating event handle\");\nres1 = fpgaRegisterEvent(fpga_device_handle, FPGA_EVENT_ERROR, eh, 0);\nON_ERR_GOTO(res1, out_destroy_eh, \"registering an FME event\");\nprintf(\"Waiting for interrupts now...\\n\");\nres = pthread_create(&errthr, NULL, error_thread, fpga_device_token);\nif (res) {\nprintf(\"Failed to create error_thread.\\n\");\nres1 = FPGA_EXCEPTION;\ngoto out_destroy_eh;\n}\nres1 = fpgaGetOSObjectFromEventHandle(eh, &pfd.fd);\nON_ERR_GOTO(res1, out_join, \"getting file descriptor\");\npfd.events = POLLIN;\npoll_ret = poll(&pfd, 1, timeout);\nif (poll_ret < 0) {\nprintf(\"Poll error errno = %s\\n\", strerror(errno));\nres1 = FPGA_EXCEPTION;\ngoto out_join;\n} else if (poll_ret == 0) {\nprintf(\"Poll timeout occurred\\n\");\nres1 = FPGA_EXCEPTION;\ngoto out_join;\n} else {\nprintf(\"FME Interrupt occurred\\n\");\nbytes_read = opae_read(pfd.fd, &count, sizeof(count));\nif (bytes_read <= 0)\nprintf(\"WARNING: error reading from poll fd: %s\\n\",\nbytes_read < 0 ? strerror(errno) : \"zero bytes read\");\n}\nres1 = fpgaUnregisterEvent(fpga_device_handle, FPGA_EVENT_ERROR, eh);\nON_ERR_GOTO(res1, out_join, \"unregistering an FME event\");\nprintf(\"Successfully tested Register/Unregister for FME events!\\n\");\nout_join:\npthread_join(errthr, NULL);\nout_destroy_eh:\nres2 = fpgaDestroyEventHandle(&eh);\nON_ERR_GOTO(res2, out_close, \"deleting event handle\");\nout_close:\nres2 = fpgaClose(fpga_device_handle);\nON_ERR_GOTO(res2, out_destroy_tok, \"closing accelerator\");\nout_destroy_tok:\nres2 = fpgaDestroyToken(&fpga_device_token);\nON_ERR_GOTO(res2, out_exit, \"destroying token\");\nout_exit:\nfpgaDestroyProperties(&device_filter);\nreturn res1 != FPGA_OK ? res1 : res2;\n}\n
"},{"location":"opae-code/dir_a3c160366dc832de1042e5d4d49ef034/","title":"Dir docs/sw/samples/hello_fpga","text":"FileList > docs > sw > samples > hello_fpga
"},{"location":"opae-code/dir_a3c160366dc832de1042e5d4d49ef034/#files","title":"Files","text":"Type Name file hello_fpga.c A code sample illustrates the basic usage of the OPAE C API. The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/
"},{"location":"opae-code/hello__fpga_8c/","title":"File hello_fpga.c","text":"FileList > docs > sw > samples > hello_fpga > hello_fpga.c
Go to the source code of this file.
A code sample illustrates the basic usage of the OPAE C API. More...
#include <stdio.h>#include <stdlib.h>#include <string.h>#include <stdint.h>#include <getopt.h>#include <unistd.h>#include <uuid/uuid.h>#include <opae/fpga.h>#include <argsfilter.h>#include \"mock/opae_std.h\"
"},{"location":"opae-code/hello__fpga_8c/#classes","title":"Classes","text":"Type Name struct cache_line struct config"},{"location":"opae-code/hello__fpga_8c/#public-attributes","title":"Public Attributes","text":"Type Name struct config config = = { .open_flags = 0, .run_n3000 = 0 }"},{"location":"opae-code/hello__fpga_8c/#public-functions","title":"Public Functions","text":"Type Name fpga_result find_fpga (fpga_properties device_filter, fpga_guid afu_guid, fpga_token * accelerator_token, uint32_t * num_matches_accelerators) fpga_result find_nlb_n3000 (fpga_handle accelerator_handle, uint64_t * afu_baddr) void help (void) int main (int argc, char * argv) fpga_result parse_args (int argc, char * argv) void print_err (const char * s, fpga_result res) bool probe_for_ase (void) int usleep (unsigned)"},{"location":"opae-code/hello__fpga_8c/#macros","title":"Macros","text":"Type Name define CACHELINE_ALIGNED_ADDR (p) ((p) >> LOG2_CL) define CL (x) ((x) * 64) define CSR_AFU_DSM_BASEL 0x0110 define CSR_CFG 0x0140 define CSR_CTL 0x0138 define CSR_DST_ADDR 0x0128 define CSR_NUM_LINES 0x0130 define CSR_SRC_ADDR 0x0120 define CSR_STATUS1 0x0168 define DSM_STATUS_TEST_COMPLETE 0x40 define FPGA_NLB0_UUID_H 0xd8424dc4a4a3c413 define FPGA_NLB0_UUID_L 0xf89e433683f9040b define GETOPT_STRING \"hscv\" define LOG2_CL 6 define LPBK1_BUFFER_ALLOCATION_SIZE MB(2) define LPBK1_BUFFER_SIZE MB(1) define LPBK1_DSM_SIZE MB(2) define MB (x) ((x) * 1024 * 1024) define N3000_AFUID \"9AEFFE5F-8457-0612-C000-C9660D824272\" define NLB0_AFUID \"D8424DC4-A4A3-C413-F89E-433683F9040B\" define ON_ERR_GOTO (res, label, desc) define TEST_TIMEOUT 30000"},{"location":"opae-code/hello__fpga_8c/#detailed-description","title":"Detailed Description","text":"The sample is a host application that demonstrates the basic steps of interacting with FPGA using the OPAE library. These steps include:
- FPGA enumeration
- Resource acquiring and releasing
- Managing shared memory buffer
- MMIO read and write
The sample also demonstrates OPAE's object model, such as tokens, handles, and properties.
The sample requires a native loopback mode (NLB) test image to be loaded on the FPGA. Refer to Quick Start Guide for full instructions on building, configuring, and running this code sample.
"},{"location":"opae-code/hello__fpga_8c/#public-attributes-documentation","title":"Public Attributes Documentation","text":""},{"location":"opae-code/hello__fpga_8c/#variable-config","title":"variable config","text":"struct config config;\n
"},{"location":"opae-code/hello__fpga_8c/#public-functions-documentation","title":"Public Functions Documentation","text":""},{"location":"opae-code/hello__fpga_8c/#function-find_fpga","title":"function find_fpga","text":"fpga_result find_fpga (\nfpga_properties device_filter,\nfpga_guid afu_guid,\nfpga_token * accelerator_token,\nuint32_t * num_matches_accelerators\n)
"},{"location":"opae-code/hello__fpga_8c/#function-find_nlb_n3000","title":"function find_nlb_n3000","text":"fpga_result find_nlb_n3000 (\nfpga_handle accelerator_handle,\nuint64_t * afu_baddr\n)
"},{"location":"opae-code/hello__fpga_8c/#function-help","title":"function help","text":"void help (\nvoid\n)
"},{"location":"opae-code/hello__fpga_8c/#function-main","title":"function main","text":"int main (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__fpga_8c/#function-parse_args","title":"function parse_args","text":"fpga_result parse_args (\nint argc,\nchar * argv\n)
"},{"location":"opae-code/hello__fpga_8c/#function-print_err","title":"function print_err","text":"void print_err (\nconst char * s,\nfpga_result res\n)
"},{"location":"opae-code/hello__fpga_8c/#function-probe_for_ase","title":"function probe_for_ase","text":"bool probe_for_ase (\nvoid\n)
"},{"location":"opae-code/hello__fpga_8c/#function-usleep","title":"function usleep","text":"int usleep (\nunsigned\n)
"},{"location":"opae-code/hello__fpga_8c/#macro-definition-documentation","title":"Macro Definition Documentation","text":""},{"location":"opae-code/hello__fpga_8c/#define-cacheline_aligned_addr","title":"define CACHELINE_ALIGNED_ADDR","text":"#define CACHELINE_ALIGNED_ADDR (\np\n) ((p) >> LOG2_CL )\n
"},{"location":"opae-code/hello__fpga_8c/#define-cl","title":"define CL","text":"#define CL (\nx\n) ((x) * 64)\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_afu_dsm_basel","title":"define CSR_AFU_DSM_BASEL","text":"#define CSR_AFU_DSM_BASEL 0x0110\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_cfg","title":"define CSR_CFG","text":"#define CSR_CFG 0x0140\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_ctl","title":"define CSR_CTL","text":"#define CSR_CTL 0x0138\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_dst_addr","title":"define CSR_DST_ADDR","text":"#define CSR_DST_ADDR 0x0128\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_num_lines","title":"define CSR_NUM_LINES","text":"#define CSR_NUM_LINES 0x0130\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_src_addr","title":"define CSR_SRC_ADDR","text":"#define CSR_SRC_ADDR 0x0120\n
"},{"location":"opae-code/hello__fpga_8c/#define-csr_status1","title":"define CSR_STATUS1","text":"#define CSR_STATUS1 0x0168\n
"},{"location":"opae-code/hello__fpga_8c/#define-dsm_status_test_complete","title":"define DSM_STATUS_TEST_COMPLETE","text":"#define DSM_STATUS_TEST_COMPLETE 0x40\n
"},{"location":"opae-code/hello__fpga_8c/#define-fpga_nlb0_uuid_h","title":"define FPGA_NLB0_UUID_H","text":"#define FPGA_NLB0_UUID_H 0xd8424dc4a4a3c413\n
"},{"location":"opae-code/hello__fpga_8c/#define-fpga_nlb0_uuid_l","title":"define FPGA_NLB0_UUID_L","text":"#define FPGA_NLB0_UUID_L 0xf89e433683f9040b\n
"},{"location":"opae-code/hello__fpga_8c/#define-getopt_string","title":"define GETOPT_STRING","text":"#define GETOPT_STRING \"hscv\"\n
"},{"location":"opae-code/hello__fpga_8c/#define-log2_cl","title":"define LOG2_CL","text":"#define LOG2_CL 6\n
"},{"location":"opae-code/hello__fpga_8c/#define-lpbk1_buffer_allocation_size","title":"define LPBK1_BUFFER_ALLOCATION_SIZE","text":"#define LPBK1_BUFFER_ALLOCATION_SIZE MB (2)\n
"},{"location":"opae-code/hello__fpga_8c/#define-lpbk1_buffer_size","title":"define LPBK1_BUFFER_SIZE","text":"#define LPBK1_BUFFER_SIZE MB (1)\n
"},{"location":"opae-code/hello__fpga_8c/#define-lpbk1_dsm_size","title":"define LPBK1_DSM_SIZE","text":"#define LPBK1_DSM_SIZE MB (2)\n
"},{"location":"opae-code/hello__fpga_8c/#define-mb","title":"define MB","text":"#define MB (\nx\n) ((x) * 1024 * 1024)\n
"},{"location":"opae-code/hello__fpga_8c/#define-n3000_afuid","title":"define N3000_AFUID","text":"#define N3000_AFUID \"9AEFFE5F-8457-0612-C000-C9660D824272\"\n
"},{"location":"opae-code/hello__fpga_8c/#define-nlb0_afuid","title":"define NLB0_AFUID","text":"#define NLB0_AFUID \"D8424DC4-A4A3-C413-F89E-433683F9040B\"\n
"},{"location":"opae-code/hello__fpga_8c/#define-on_err_goto","title":"define ON_ERR_GOTO","text":"#define ON_ERR_GOTO (\nres,\nlabel,\ndesc\n) do { \\\n if ((res) != FPGA_OK ) { \\ print_err ((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\n
"},{"location":"opae-code/hello__fpga_8c/#define-test_timeout","title":"define TEST_TIMEOUT","text":"#define TEST_TIMEOUT 30000\n
The documentation for this class was generated from the following file docs/sw/samples/hello_fpga/hello_fpga.c
"},{"location":"opae-code/hello__fpga_8c_source/","title":"File hello_fpga.c","text":"File List > docs > sw > samples > hello_fpga > hello_fpga.c
Go to the documentation of this file.
// Copyright(c) 2017-2022, Intel Corporation\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are met:\n//\n// * Redistributions of source code must retain the above copyright notice,\n// this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above copyright notice,\n// this list of conditions and the following disclaimer in the documentation\n// and/or other materials provided with the distribution.\n// * Neither the name of Intel Corporation nor the names of its contributors\n// may be used to endorse or promote products derived from this software\n// without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\n// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n// POSSIBILITY OF SUCH DAMAGE.\n#ifdef HAVE_CONFIG_H\n#include <config.h>\n#endif // HAVE_CONFIG_H\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n#include <stdint.h>\n#include <getopt.h>\n#include <unistd.h>\n#include <uuid/uuid.h>\n#include <opae/fpga.h>\n#include <argsfilter.h>\n#include \"mock/opae_std.h\"\nint usleep(unsigned);\n#ifndef TEST_TIMEOUT\n#define TEST_TIMEOUT 30000\n#endif // TEST_TIMEOUT\n#ifndef CL\n# define CL(x) ((x) * 64)\n#endif // CL\n#ifndef LOG2_CL\n# define LOG2_CL 6\n#endif // LOG2_CL\n#ifndef MB\n# define MB(x) ((x) * 1024 * 1024)\n#endif // MB\n#define CACHELINE_ALIGNED_ADDR(p) ((p) >> LOG2_CL)\n#define LPBK1_BUFFER_SIZE MB(1)\n#define LPBK1_BUFFER_ALLOCATION_SIZE MB(2)\n#define LPBK1_DSM_SIZE MB(2)\n#define CSR_SRC_ADDR 0x0120\n#define CSR_DST_ADDR 0x0128\n#define CSR_CTL 0x0138\n#define CSR_STATUS1 0x0168\n#define CSR_CFG 0x0140\n#define CSR_NUM_LINES 0x0130\n#define DSM_STATUS_TEST_COMPLETE 0x40\n#define CSR_AFU_DSM_BASEL 0x0110\n/* NLB0 AFU_ID */\n#define NLB0_AFUID \"D8424DC4-A4A3-C413-F89E-433683F9040B\"\n/* NLB0 AFU_ID for N3000 */\n#define N3000_AFUID \"9AEFFE5F-8457-0612-C000-C9660D824272\"\n#define FPGA_NLB0_UUID_H 0xd8424dc4a4a3c413\n#define FPGA_NLB0_UUID_L 0xf89e433683f9040b\n/*\n * macro to check return codes, print error message, and goto cleanup label\n * NOTE: this changes the program flow (uses goto)!\n */\n#define ON_ERR_GOTO(res, label, desc) \\\n do { \\\n if ((res) != FPGA_OK) { \\\n print_err((desc), (res)); \\\n goto label; \\\n } \\\n } while (0)\n/* Type definitions */\ntypedef struct {\nuint32_t uint[16];\n} cache_line;\nvoid print_err(const char *s, fpga_result res)\n{\nfprintf(stderr, \"Error %s: %s\\n\", s, fpgaErrStr(res));\n}\n/*\n * Global configuration of bus, set during parse_args()\n * */\nstruct config {\nint open_flags;\nint run_n3000;\n}\nconfig = {\n.open_flags = 0,\n.run_n3000 = 0\n};\nvoid help(void)\n{\nprintf(\"\\n\"\n\"hello_fpga\\n\"\n\"OPAE Native Loopback 0 (NLB0) sample\\n\"\n\"\\n\"\n\"Usage:\\n\"\n\" hello_fpga [-schv] [-S <segment>] [-B <bus>] [-D <device>] [-F <function>] [PCI_ADDR]\\n\"\n\"\\n\"\n\" -s,--shared Open accelerator in shared mode\\n\"\n\" -c,--n3000 Assume N3000 MMIO layout\\n\"\n\" -h,--help Print this help\\n\"\n\" -v,--version Print version info and exit\\n\"\n\"\\n\");\n}\n#define GETOPT_STRING \"hscv\"\nfpga_result parse_args(int argc, char *argv[])\n{\nstruct option longopts[] = {\n{ \"help\", no_argument, NULL, 'h' },\n{ \"shared\", no_argument, NULL, 's' },\n{ \"n3000\", no_argument, NULL, 'c' },\n{ \"version\", no_argument, NULL, 'v' },\n{ NULL, 0, NULL, 0 }\n};\nint getopt_ret;\nint option_index;\nchar version[32];\nchar build[32];\nwhile (-1 != (getopt_ret = getopt_long(argc, argv, GETOPT_STRING,\nlongopts, &option_index))) {\nconst char *tmp_optarg = optarg;\n/* Checks to see if optarg is null and if not it goes to value of optarg */\nif ((optarg) && ('=' == *tmp_optarg)) {\n++tmp_optarg;\n}\nswitch (getopt_ret) {\ncase 'h':\nhelp();\nreturn -1;\ncase 's':\nconfig.open_flags |= FPGA_OPEN_SHARED;\nbreak;\ncase 'c':\nconfig.run_n3000 = 1;\nbreak;\ncase 'v':\nfpgaGetOPAECVersionString(version, sizeof(version));\nfpgaGetOPAECBuildString(build, sizeof(build));\nprintf(\"hello_fpga %s %s\\n\",\nversion, build);\nreturn -1;\ndefault: /* invalid option */\nfprintf(stderr, \"Invalid cmdline option \\n\");\nreturn FPGA_EXCEPTION;\n}\n}\nreturn FPGA_OK;\n}\nfpga_result find_fpga(fpga_properties device_filter,\nfpga_guid afu_guid,\nfpga_token *accelerator_token,\nuint32_t *num_matches_accelerators)\n{\nfpga_properties filter = NULL;\nfpga_result res1;\nfpga_result res2 = FPGA_OK;\nres1 = fpgaCloneProperties(device_filter, &filter);\nON_ERR_GOTO(res1, out, \"cloning properties object\");\nres1 = fpgaPropertiesSetObjectType(filter, FPGA_ACCELERATOR);\nON_ERR_GOTO(res1, out_destroy, \"setting object type\");\nres1 = fpgaPropertiesSetGUID(filter, afu_guid);\nON_ERR_GOTO(res1, out_destroy, \"setting GUID\");\nres1 = fpgaEnumerate(&filter, 1, accelerator_token, 1, num_matches_accelerators);\nON_ERR_GOTO(res1, out_destroy, \"enumerating accelerators\");\nout_destroy:\nres2 = fpgaDestroyProperties(&filter);\nON_ERR_GOTO(res2, out, \"destroying properties object\");\nout:\nreturn res1 != FPGA_OK ? res1 : res2;\n}\n/* Is the FPGA simulated with ASE? */\nbool probe_for_ase(void)\n{\nfpga_result r = FPGA_OK;\nuint16_t device_id = 0;\nfpga_properties filter = NULL;\nuint32_t num_matches = 1;\nfpga_token fme_token;\n/* Connect to the FPGA management engine */\nfpgaGetProperties(NULL, &filter);\nfpgaPropertiesSetObjectType(filter, FPGA_DEVICE);\n/* Connecting to one is sufficient to find ASE */\nfpgaEnumerate(&filter, 1, &fme_token, 1, &num_matches);\nif (0 != num_matches) {\n/* Retrieve the device ID of the FME */\nfpgaDestroyProperties(&filter);\nfpgaGetProperties(fme_token, &filter);\nr = fpgaPropertiesGetDeviceID(filter, &device_id);\nfpgaDestroyToken(&fme_token);\n}\nfpgaDestroyProperties(&filter);\n/* ASE's device ID is 0xa5e */\nreturn ((FPGA_OK == r) && (0xa5e == device_id));\n}\nfpga_result find_nlb_n3000(fpga_handle accelerator_handle,\nuint64_t *afu_baddr)\n{\nfpga_result res1 = FPGA_OK;\nint end_of_list = 0;\nint nlb0_found = 0;\nuint64_t header = 0;\nuint64_t uuid_hi = 0;\nuint64_t uuid_lo = 0;\nuint64_t next_offset = 0;\nuint64_t nlb0_offset = 0;\n/* find NLB0 in AFU */\ndo {\n// Read the next feature header\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb0_offset, &header);\nON_ERR_GOTO(res1, out_exit, \"fpgaReadMMIO64\");\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb0_offset+8, &uuid_lo);\nON_ERR_GOTO(res1, out_exit, \"fpgaReadMMIO64\");\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb0_offset+16, &uuid_hi);\nON_ERR_GOTO(res1, out_exit, \"fpgaReadMMIO64\");\n// printf(\"%zx: %zx %zx %zx\\n\", nlb0_offset, header, uuid_lo, uuid_hi);\nif ((((header >> 60) & 0xf) == 0x1) &&\n(uuid_lo == FPGA_NLB0_UUID_L) && (uuid_hi == FPGA_NLB0_UUID_H)) {\nnlb0_found = 1;\nbreak;\n}\n// End of the list flag\nend_of_list = (header >> 40) & 1;\n// Move to the next feature header\nnext_offset = (header >> 16) & 0xffffff;\nif ((next_offset == 0xffff) || (next_offset == 0)) {\nnlb0_found = 0;\nbreak;\n}\nnlb0_offset = nlb0_offset + next_offset;\n} while (!end_of_list);\nif (!nlb0_found) {\nprintf(\"AFU NLB0 not found\\n\");\nreturn FPGA_EXCEPTION;\n}\nprintf(\"AFU NLB0 found @ %zx\\n\", nlb0_offset);\n*afu_baddr = nlb0_offset;\nreturn FPGA_OK;\nout_exit:\nreturn FPGA_EXCEPTION;\n}\nint main(int argc, char *argv[])\n{\nfpga_token accelerator_token;\nfpga_handle accelerator_handle;\nfpga_guid guid;\nuint32_t num_matches_accelerators = 0;\nuint32_t use_ase;\nvolatile uint64_t *dsm_ptr = NULL;\nvolatile uint64_t *status_ptr = NULL;\nvolatile uint64_t *input_ptr = NULL;\nvolatile uint64_t *output_ptr = NULL;\nuint64_t dsm_wsid;\nuint64_t input_wsid;\nuint64_t output_wsid;\nuint32_t i;\nuint32_t timeout;\nfpga_result res1 = FPGA_OK;\nfpga_result res2 = FPGA_OK;\nfpga_properties device_filter = NULL;\nres1 = fpgaGetProperties(NULL, &device_filter);\nif (res1 != FPGA_OK) {\nprint_err(\"failed to allocate properties.\\n\", res1);\nreturn 1;\n}\nif (opae_set_properties_from_args(device_filter,\n&res1,\n&argc,\nargv)) {\nprint_err(\"failed arg parse.\\n\", res1);\nres1 = FPGA_EXCEPTION;\ngoto out_exit;\n} else if (res1) {\nprint_err(\"failed to set properties.\\n\", res1);\ngoto out_exit;\n}\nres1 = parse_args(argc, argv);\nif ((int)res1 < 0)\ngoto out_exit;\nON_ERR_GOTO(res1, out_exit, \"parsing arguments\");\nif (config.run_n3000) {\nif (uuid_parse(N3000_AFUID, guid) < 0)\nres1 = FPGA_EXCEPTION;\n} else {\nif (uuid_parse(NLB0_AFUID, guid) < 0)\nres1 = FPGA_EXCEPTION;\n}\nON_ERR_GOTO(res1, out_exit, \"parsing guid\");\nuse_ase = probe_for_ase();\nif (use_ase) {\nprintf(\"Running in ASE mode\\n\");\n}\n/* Look for accelerator with NLB0_AFUID */\nres1 = find_fpga(device_filter,\nguid,\n&accelerator_token,\n&num_matches_accelerators);\nON_ERR_GOTO(res1, out_exit, \"finding FPGA accelerator\");\nif (num_matches_accelerators == 0) {\nres1 = FPGA_NOT_FOUND;\n}\nON_ERR_GOTO(res1, out_exit, \"no matching accelerator\");\nif (num_matches_accelerators > 1) {\nprintf(\"Found more than one suitable accelerator. \");\n}\n/* Open accelerator and map MMIO */\nres1 = fpgaOpen(accelerator_token, &accelerator_handle, config.open_flags);\nON_ERR_GOTO(res1, out_destroy_tok, \"opening accelerator\");\nres1 = fpgaMapMMIO(accelerator_handle, 0, NULL);\nON_ERR_GOTO(res1, out_close, \"mapping MMIO space\");\n/* Allocate buffers */\nres1 = fpgaPrepareBuffer(accelerator_handle, LPBK1_DSM_SIZE,\n(void **)&dsm_ptr, &dsm_wsid, 0);\nON_ERR_GOTO(res1, out_close, \"allocating DSM buffer\");\nres1 = fpgaPrepareBuffer(accelerator_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&input_ptr, &input_wsid, 0);\nON_ERR_GOTO(res1, out_free_dsm, \"allocating input buffer\");\nres1 = fpgaPrepareBuffer(accelerator_handle, LPBK1_BUFFER_ALLOCATION_SIZE,\n(void **)&output_ptr, &output_wsid, 0);\nON_ERR_GOTO(res1, out_free_input, \"allocating output buffer\");\nprintf(\"Running Test\\n\");\n/* Initialize buffers */\nmemset((void *)dsm_ptr, 0, LPBK1_DSM_SIZE);\nmemset((void *)input_ptr, 0xAF, LPBK1_BUFFER_SIZE);\nmemset((void *)output_ptr, 0xBE, LPBK1_BUFFER_SIZE);\ncache_line *cl_ptr = (cache_line *)input_ptr;\nfor (i = 0; i < LPBK1_BUFFER_SIZE / CL(1); ++i) {\ncl_ptr[i].uint[15] = i+1; /* set the last uint in every cacheline */\n}\n/* Reset accelerator */\nres1 = fpgaReset(accelerator_handle);\nON_ERR_GOTO(res1, out_free_output, \"resetting accelerator\");\nuint64_t nlb_base_addr = 0;\nif (config.run_n3000) {\nres1 = find_nlb_n3000(accelerator_handle, &nlb_base_addr);\nON_ERR_GOTO(res1, out_free_output, \"finding nlb in AFU\");\n}\n/* Program DMA addresses */\nuint64_t iova = 0;\nres1 = fpgaGetIOAddress(accelerator_handle, dsm_wsid, &iova);\nON_ERR_GOTO(res1, out_free_output, \"getting DSM IOVA\");\nres1 = fpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_AFU_DSM_BASEL, iova);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_AFU_DSM_BASEL\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 0);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 1);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\nres1 = fpgaGetIOAddress(accelerator_handle, input_wsid, &iova);\nON_ERR_GOTO(res1, out_free_output, \"getting input IOVA\");\nres1 = fpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_SRC_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_SRC_ADDR\");\nres1 = fpgaGetIOAddress(accelerator_handle, output_wsid, &iova);\nON_ERR_GOTO(res1, out_free_output, \"getting output IOVA\");\nres1 = fpgaWriteMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_DST_ADDR, CACHELINE_ALIGNED_ADDR(iova));\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_DST_ADDR\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_NUM_LINES, LPBK1_BUFFER_SIZE / CL(1));\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_NUM_LINES\");\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CFG, 0x42000);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\nstatus_ptr = dsm_ptr + DSM_STATUS_TEST_COMPLETE/sizeof(uint64_t);\n/* Start the test */\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 3);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\n/* Wait for test completion */\ntimeout = TEST_TIMEOUT;\nwhile (0 == ((*status_ptr) & 0x1)) {\nusleep(100);\nif (!use_ase && (--timeout == 0)) {\nres1 = FPGA_EXCEPTION;\nON_ERR_GOTO(res1, out_free_output, \"test timed out\");\n}\n}\n/* Stop the device */\nres1 = fpgaWriteMMIO32(accelerator_handle, 0, nlb_base_addr + CSR_CTL, 7);\nON_ERR_GOTO(res1, out_free_output, \"writing CSR_CFG\");\n/* Wait for the AFU's read/write traffic to complete */\nuint32_t afu_traffic_trips = 0;\nwhile (afu_traffic_trips < 100) {\n/*\n * CSR_STATUS1 holds two 32 bit values: num pending reads and writes.\n * Wait for it to be 0.\n */\nuint64_t s1;\nres1 = fpgaReadMMIO64(accelerator_handle, 0, nlb_base_addr + CSR_STATUS1, &s1);\nON_ERR_GOTO(res1, out_free_output, \"reading CSR_STATUS1\");\nif (s1 == 0) {\nbreak;\n}\nafu_traffic_trips += 1;\nusleep(1000);\n}\n/* Check output buffer contents */\nfor (i = 0; i < LPBK1_BUFFER_SIZE; i++) {\nif (((uint8_t *)output_ptr)[i] != ((uint8_t *)input_ptr)[i]) {\nfprintf(stderr, \"Output does NOT match input \"\n\"at offset %i!\\n\", i);\nbreak;\n}\n}\nprintf(\"Done Running Test\\n\");\n/* Release buffers */\nout_free_output:\nres2 = fpgaReleaseBuffer(accelerator_handle, output_wsid);\nON_ERR_GOTO(res2, out_free_input, \"releasing output buffer\");\nout_free_input:\nres2 = fpgaReleaseBuffer(accelerator_handle, input_wsid);\nON_ERR_GOTO(res2, out_free_dsm, \"releasing input buffer\");\nout_free_dsm:\nres2 = fpgaReleaseBuffer(accelerator_handle, dsm_wsid);\nON_ERR_GOTO(res2, out_unmap, \"releasing DSM buffer\");\n/* Unmap MMIO space */\nout_unmap:\nres2 = fpgaUnmapMMIO(accelerator_handle, 0);\nON_ERR_GOTO(res2, out_close, \"unmapping MMIO space\");\n/* Release accelerator */\nout_close:\nres2 = fpgaClose(accelerator_handle);\nON_ERR_GOTO(res2, out_destroy_tok, \"closing accelerator\");\n/* Destroy token */\nout_destroy_tok:\nres2 = fpgaDestroyToken(&accelerator_token);\nON_ERR_GOTO(res2, out_exit, \"destroying token\");\nout_exit:\nfpgaDestroyProperties(&device_filter);\nreturn res1 != FPGA_OK ? res1 : res2;\n}\n
"},{"location":"opae-code/namespaces/","title":"Namespace List","text":"Here is a list of all namespaces with brief descriptions:
- namespace opae
- namespace fpga
- namespace types
- namespace detail
- namespace std
"},{"location":"opae-code/classes/","title":"Class Index","text":""},{"location":"opae-code/classes/#b","title":"b","text":" - busy (opae::fpga::types)
"},{"location":"opae-code/classes/#c","title":"c","text":" - cache_line
- config
"},{"location":"opae-code/classes/#e","title":"e","text":" - error (opae::fpga::types)
- event (opae::fpga::types)
- except (opae::fpga::types)
- exception (opae::fpga::types)
"},{"location":"opae-code/classes/#f","title":"f","text":" - fpga_error_info
- fpga_metric
- fpga_metric_info
- fpga_version
"},{"location":"opae-code/classes/#g","title":"g","text":" - guid_t (opae::fpga::types)
"},{"location":"opae-code/classes/#h","title":"h","text":" - handle (opae::fpga::types)
"},{"location":"opae-code/classes/#i","title":"i","text":" - invalid_param (opae::fpga::types)
"},{"location":"opae-code/classes/#m","title":"m","text":" - mem_alloc
- mem_link
- metric_threshold
"},{"location":"opae-code/classes/#n","title":"n","text":" - no_access (opae::fpga::types)
- no_daemon (opae::fpga::types)
- no_driver (opae::fpga::types)
- no_memory (opae::fpga::types)
- not_found (opae::fpga::types)
- not_supported (opae::fpga::types)
"},{"location":"opae-code/classes/#o","title":"o","text":" - opae_uio
- opae_uio_device_region
- opae_vfio
- opae_vfio_buffer
- opae_vfio_device
- opae_vfio_device_irq
- opae_vfio_device_region
- opae_vfio_group
- opae_vfio_iova_range
- opae_vfio_sparse_info
"},{"location":"opae-code/classes/#p","title":"p","text":" - properties (opae::fpga::types)
- pvalue (opae::fpga::types)
"},{"location":"opae-code/classes/#r","title":"r","text":" - reconf_error (opae::fpga::types)
- ras_inject_error
"},{"location":"opae-code/classes/#s","title":"s","text":" - shared_buffer (opae::fpga::types)
- src_location (opae::fpga::types)
- sysobject (opae::fpga::types)
"},{"location":"opae-code/classes/#t","title":"t","text":" - token (opae::fpga::types)
- type_t (opae::fpga::types::event)
- threshold
"},{"location":"opae-code/classes/#v","title":"v","text":" - version (opae::fpga::types)
## \\
- _fpga_token_header
- _opae_hash_map
- _opae_hash_map_item
"},{"location":"opae-code/hierarchy/","title":"Class Hierarchy","text":"This inheritance list is sorted roughly, but not completely, alphabetically:
- class opae::fpga::types::error An error object represents an error register for a resource.
- class opae::fpga::types::event Wraps fpga event routines in OPAE C.
- class opae::fpga::types::handle An allocated accelerator resource.
- class opae::fpga::types::properties Wraps an OPAE fpga_properties object.
- class opae::fpga::types::shared_buffer Host/AFU shared memory blocks.
- class opae::fpga::types::src_location Identify a particular line in a source file.
- class opae::fpga::types::sysobject Wraps the OPAE fpga_object primitive.
- class opae::fpga::types::token Wraps the OPAE fpga_token primitive.
- class opae::fpga::types::version
- struct _fpga_token_header Internal token type header.
- struct _opae_hash_map Hash map object.
- struct _opae_hash_map_item List link item.
- struct cache_line
- struct config
- struct fpga_error_info
- struct fpga_metric Metric struct.
- struct fpga_metric_info Metric info struct.
- struct fpga_version Semantic version.
- struct mem_alloc
- struct mem_link Provides an API for allocating/freeing a logical address space.
- struct metric_threshold
- struct opae::fpga::types::event::type_t C++ struct that is interchangeable with fpga_event_type enum.
- struct opae::fpga::types::guid_t Representation of the guid member of a properties object.
- struct opae::fpga::types::pvalue Wraps OPAE properties defined in the OPAE C API by associating an
fpga_properties reference with the getters and setters defined for a property. - struct opae_uio OPAE UIO device abstraction.
- struct opae_uio_device_region MMIO region info.
- struct opae_vfio OPAE VFIO device abstraction.
- struct opae_vfio_buffer System DMA buffer.
- struct opae_vfio_device VFIO device.
- struct opae_vfio_device_irq Interrupt info.
- struct opae_vfio_device_region MMIO region info.
- struct opae_vfio_group VFIO group.
- struct opae_vfio_iova_range IO Virtual Address Range.
- struct opae_vfio_sparse_info MMIO sparse-mappable region info.
- struct ras_inject_error
- struct threshold Threshold struct.
- class std::exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
- class opae::fpga::types::except Generic OPAE exception.
- class opae::fpga::types::busy busy exception
- class opae::fpga::types::exception exception exception
- class opae::fpga::types::invalid_param invalid_param exception
- class opae::fpga::types::no_access no_access exception
- class opae::fpga::types::no_daemon no_daemon exception
- class opae::fpga::types::no_driver no_driver exception
- class opae::fpga::types::no_memory no_memory exception
- class opae::fpga::types::not_found not_found exception
- class opae::fpga::types::not_supported not_supported exception
- class opae::fpga::types::reconf_error reconf_error exception
"},{"location":"opae-code/modules/","title":"Modules","text":"Here is a list of all modules:
"},{"location":"opae-code/todo/","title":"Todo List","text":""},{"location":"opae-code/todo/#global-fpgaregisterevent-fpga_handle-handle-fpga_event_type-event_type-fpga_event_handle-event_handle-uint32_t-flags","title":"Global fpgaRegisterEvent (fpga_handle handle, fpga_event_type event_type, fpga_event_handle event_handle, uint32_t flags)","text":"define if calling fpgaRegisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored. define if calling fpgaUnregisterEvent multiple times with the same event_handle is an error condition or if it is silently ignored.
"},{"location":"opae-code/pages/","title":"Class List","text":"Here are the classes, structs, unions and interfaces with brief descriptions:
"},{"location":"opae-code/class_members/","title":"Class Members","text":""},{"location":"opae-code/class_members/#a","title":"a","text":" - allocated (mem_alloc)
- address (mem_link)
- accelerator_state (opae::fpga::types::properties)
- allocate (opae::fpga::types::shared_buffer)
- attach (opae::fpga::types::shared_buffer)
- as_string (opae::fpga::types::version)
- as_struct (opae::fpga::types::version)
"},{"location":"opae-code/class_members/#b","title":"b","text":" - bus (_fpga_token_header, opae::fpga::types::properties)
- buckets (_opae_hash_map)
- busy (opae::fpga::types::busy)
- buf_ (opae::fpga::types::except)
- bbs_id (opae::fpga::types::properties)
- bbs_version (opae::fpga::types::properties)
- bytes (opae::fpga::types::sysobject)
- build (opae::fpga::types::version)
- buffer_iova (opae_vfio_buffer)
- buffer_ptr (opae_vfio_buffer)
- buffer_size (opae_vfio_buffer)
"},{"location":"opae-code/class_members/#c","title":"c","text":" - cleanup_context (_opae_hash_map)
- can_clear (fpga_error_info, opae::fpga::types::error)
- c_type (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
- close (opae::fpga::types::handle)
- capabilities (opae::fpga::types::properties)
- copy_ (opae::fpga::types::pvalue)
- copy_t (opae::fpga::types::pvalue)
- compare (opae::fpga::types::shared_buffer)
- cont_buffers (opae_vfio)
- cont_device (opae_vfio)
- cont_fd (opae_vfio)
- cont_pciaddr (opae_vfio)
- cont_ranges (opae_vfio)
- count (opae_vfio_device_irq)
- catastrophicr_error (ras_inject_error)
- csr (ras_inject_error)
"},{"location":"opae-code/class_members/#d","title":"d","text":" - device (_fpga_token_header, opae::fpga::types::properties, opae_vfio)
- device_id (_fpga_token_header, opae::fpga::types::properties)
- data_ (opae::fpga::types::guid_t)
- device_fd (opae_uio, opae_vfio_device)
- device_path (opae_uio)
- device_config_offset (opae_vfio_device)
- device_num_irqs (opae_vfio_device)
- device_num_regions (opae_vfio_device)
"},{"location":"opae-code/class_members/#e","title":"e","text":" - error (opae::fpga::types::error, opae::fpga::types::event::type_t)
- error_info_ (opae::fpga::types::error)
- error_num_ (opae::fpga::types::error)
- event (opae::fpga::types::event)
- event_handle_ (opae::fpga::types::event)
- except (opae::fpga::types::except)
- exception (opae::fpga::types::exception)
- enumerate (opae::fpga::types::token)
- event_fds (opae_vfio_device_irq)
- end (opae_vfio_iova_range)
"},{"location":"opae-code/class_members/#f","title":"f","text":" - function (_fpga_token_header, opae::fpga::types::properties)
- flags (_opae_hash_map, opae_vfio_buffer, opae_vfio_device_irq)
- free (mem_alloc)
- fill (opae::fpga::types::shared_buffer)
- file (opae::fpga::types::src_location)
- file_ (opae::fpga::types::src_location)
- fn (opae::fpga::types::src_location)
- fn_ (opae::fpga::types::src_location)
- fatal_error (ras_inject_error)
"},{"location":"opae-code/class_members/#g","title":"g","text":" - guid (_fpga_token_header, opae::fpga::types::properties)
- group_name (fpga_metric_info)
- get (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::properties, opae::fpga::types::sysobject)
- guid_t (opae::fpga::types::guid_t)
- get_token (opae::fpga::types::handle)
- get_ (opae::fpga::types::pvalue)
- get_value (opae::fpga::types::pvalue)
- getter_t (opae::fpga::types::pvalue)
- get_parent (opae::fpga::types::token)
- group (opae_vfio)
- group_device (opae_vfio_group)
- group_fd (opae_vfio_group)
"},{"location":"opae-code/class_members/#h","title":"h","text":" - hash_seed (_opae_hash_map)
- hysteresis (metric_threshold)
- handle_ (opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject)
- handle (opae::fpga::types::handle)
"},{"location":"opae-code/class_members/#i","title":"i","text":" - interface (_fpga_token_header, opae::fpga::types::properties)
- isvalid (fpga_metric)
- interrupt (opae::fpga::types::event::type_t)
- invalidate (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- is_set (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- is_set_ (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- invalid_param (opae::fpga::types::invalid_param)
- io_address (opae::fpga::types::shared_buffer)
- io_address_ (opae::fpga::types::shared_buffer)
- iova_alloc (opae_vfio)
- irqs (opae_vfio_device)
- index (opae_vfio_device_irq, opae_vfio_sparse_info)
- is_valid (threshold)
"},{"location":"opae-code/class_members/#k","title":"k","text":" - key_cleanup (_opae_hash_map)
- key_compare (_opae_hash_map)
- key_hash (_opae_hash_map)
- key (_opae_hash_map_item)
"},{"location":"opae-code/class_members/#l","title":"l","text":" - lower_c_threshold (metric_threshold)
- lower_nc_threshold (metric_threshold)
- lower_nr_threshold (metric_threshold)
- loc_ (opae::fpga::types::except)
- local_memory_size (opae::fpga::types::properties)
- len_ (opae::fpga::types::shared_buffer)
- line (opae::fpga::types::src_location)
- line_ (opae::fpga::types::src_location)
- lock (opae_vfio)
"},{"location":"opae-code/class_members/#m","title":"m","text":" - magic (_fpga_token_header)
- metric_num (fpga_metric, fpga_metric_info)
- metric_datatype (fpga_metric_info)
- metric_guid (fpga_metric_info)
- metric_name (fpga_metric_info, metric_threshold)
- metric_type (fpga_metric_info)
- metric_units (fpga_metric_info)
- major (fpga_version)
- minor (fpga_version)
- MAX_EXCEPT (opae::fpga::types::except)
- msg_ (opae::fpga::types::except)
- mmio_ptr (opae::fpga::types::handle)
- model (opae::fpga::types::properties)
- masks (opae_vfio_device_irq)
"},{"location":"opae-code/class_members/#n","title":"n","text":" - num_buckets (_opae_hash_map)
- next (_opae_hash_map_item, mem_link, opae_uio_device_region, opae_vfio_device_irq, opae_vfio_device_region, opae_vfio_iova_range, opae_vfio_sparse_info)
- name (fpga_error_info, opae::fpga::types::error)
- no_access (opae::fpga::types::no_access)
- no_daemon (opae::fpga::types::no_daemon)
- no_driver (opae::fpga::types::no_driver)
- no_memory (opae::fpga::types::no_memory)
- not_found (opae::fpga::types::not_found)
- not_supported (opae::fpga::types::not_supported)
- none (opae::fpga::types::properties)
- num_errors (opae::fpga::types::properties)
- num_interrupts (opae::fpga::types::properties)
- num_mmio (opae::fpga::types::properties)
- num_slots (opae::fpga::types::properties)
- nonfatal_error (ras_inject_error)
"},{"location":"opae-code/class_members/#o","title":"o","text":" - object_id (_fpga_token_header, opae::fpga::types::properties)
- objtype (_fpga_token_header)
- open_flags (config)
- operator= (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::pvalue, opae::fpga::types::shared_buffer, opae::fpga::types::src_location, opae::fpga::types::sysobject)
- operator fpga_event_type (opae::fpga::types::event::type_t)
- operator fpga_event_handle (opae::fpga::types::event)
- os_object (opae::fpga::types::event)
- os_object_ (opae::fpga::types::event)
- operator fpga_result (opae::fpga::types::except)
- operator uint8_t * (opae::fpga::types::guid_t)
- operator== (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- open (opae::fpga::types::handle)
- operator fpga_handle (opae::fpga::types::handle)
- operator fpga_properties (opae::fpga::types::properties)
- operator copy_t (opae::fpga::types::pvalue)
- owner (opae::fpga::types::shared_buffer)
- operator fpga_object (opae::fpga::types::sysobject)
- operator fpga_token (opae::fpga::types::token)
- offset (opae_vfio_sparse_info)
"},{"location":"opae-code/class_members/#p","title":"p","text":" - patch (fpga_version)
- prev (mem_link)
- ptr_t (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
- power_thermal (opae::fpga::types::event::type_t)
- parse (opae::fpga::types::guid_t)
- props_ (opae::fpga::types::guid_t, opae::fpga::types::properties, opae::fpga::types::pvalue)
- parent (opae::fpga::types::properties)
- properties (opae::fpga::types::properties)
- pvalue (opae::fpga::types::pvalue)
- ptr (opae_vfio_sparse_info)
"},{"location":"opae-code/class_members/#q","title":"q","text":" - qualifier_name (fpga_metric_info)
"},{"location":"opae-code/class_members/#r","title":"r","text":" - run_n3000 (config)
- read_value (opae::fpga::types::error)
- register_event (opae::fpga::types::event)
- res_ (opae::fpga::types::except)
- read_csr32 (opae::fpga::types::handle)
- read_csr64 (opae::fpga::types::handle)
- reconfigure (opae::fpga::types::handle)
- reset (opae::fpga::types::handle)
- reconf_error (opae::fpga::types::reconf_error)
- read (opae::fpga::types::shared_buffer)
- release (opae::fpga::types::shared_buffer)
- read64 (opae::fpga::types::sysobject)
- regions (opae_uio, opae_vfio_device)
- region_index (opae_uio_device_region, opae_vfio_device_region)
- region_page_offset (opae_uio_device_region)
- region_ptr (opae_uio_device_region, opae_vfio_device_region)
- region_size (opae_uio_device_region, opae_vfio_device_region)
- region_sparse (opae_vfio_device_region)
- rsvd (ras_inject_error)
"},{"location":"opae-code/class_members/#s","title":"s","text":" - segment (_fpga_token_header, opae::fpga::types::properties)
- subsystem_device_id (_fpga_token_header, opae::fpga::types::properties)
- subsystem_vendor_id (_fpga_token_header, opae::fpga::types::properties)
- size (mem_link, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae_vfio_sparse_info)
- socket_id (opae::fpga::types::properties)
- set_ (opae::fpga::types::pvalue)
- setter_t (opae::fpga::types::pvalue)
- shared_buffer (opae::fpga::types::shared_buffer)
- size_t (opae::fpga::types::shared_buffer)
- src_location (opae::fpga::types::src_location)
- sysobject (opae::fpga::types::sysobject)
- sysobject_ (opae::fpga::types::sysobject)
- start (opae_vfio_iova_range)
"},{"location":"opae-code/class_members/#t","title":"t","text":" - token_ (opae::fpga::types::error, opae::fpga::types::handle, opae::fpga::types::sysobject, opae::fpga::types::token)
- type_ (opae::fpga::types::event::type_t, opae::fpga::types::event)
- type_t (opae::fpga::types::event::type_t)
- type (opae::fpga::types::properties, opae::fpga::types::sysobject)
- token (opae::fpga::types::token)
- threshold_name (threshold)
"},{"location":"opae-code/class_members/#u","title":"u","text":" - uint (cache_line)
- upper_c_threshold (metric_threshold)
- upper_nc_threshold (metric_threshold)
- upper_nr_threshold (metric_threshold)
- update (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
"},{"location":"opae-code/class_members/#v","title":"v","text":" - vendor_id (_fpga_token_header, opae::fpga::types::properties)
- value_cleanup (_opae_hash_map)
- value (_opae_hash_map_item, fpga_metric, threshold)
- virt_ (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_members/#w","title":"w","text":" - what (opae::fpga::types::except)
- write_csr32 (opae::fpga::types::handle)
- write_csr512 (opae::fpga::types::handle)
- write_csr64 (opae::fpga::types::handle)
- write (opae::fpga::types::shared_buffer)
- wsid (opae::fpga::types::shared_buffer)
- wsid_ (opae::fpga::types::shared_buffer)
- write64 (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_members/#_1","title":"~","text":" - ~error (opae::fpga::types::error)
- ~event (opae::fpga::types::event)
- ~handle (opae::fpga::types::handle)
- ~properties (opae::fpga::types::properties)
- ~shared_buffer (opae::fpga::types::shared_buffer)
- ~sysobject (opae::fpga::types::sysobject)
- ~token (opae::fpga::types::token)
"},{"location":"opae-code/class_members/#_2","title":"@","text":" - @1 (ras_inject_error)
"},{"location":"opae-code/class_member_functions/","title":"Class Member Functions","text":""},{"location":"opae-code/class_member_functions/#a","title":"a","text":" - allocate (opae::fpga::types::shared_buffer)
- attach (opae::fpga::types::shared_buffer)
- as_string (opae::fpga::types::version)
- as_struct (opae::fpga::types::version)
"},{"location":"opae-code/class_member_functions/#b","title":"b","text":" - busy (opae::fpga::types::busy)
- bytes (opae::fpga::types::sysobject)
- build (opae::fpga::types::version)
"},{"location":"opae-code/class_member_functions/#c","title":"c","text":" - c_type (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
- can_clear (opae::fpga::types::error)
- close (opae::fpga::types::handle)
- compare (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_functions/#e","title":"e","text":" - error (opae::fpga::types::error)
- event (opae::fpga::types::event)
- except (opae::fpga::types::except)
- exception (opae::fpga::types::exception)
- enumerate (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#f","title":"f","text":" - fill (opae::fpga::types::shared_buffer)
- file (opae::fpga::types::src_location)
- fn (opae::fpga::types::src_location)
"},{"location":"opae-code/class_member_functions/#g","title":"g","text":" - get (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::properties, opae::fpga::types::sysobject)
- guid_t (opae::fpga::types::guid_t)
- get_token (opae::fpga::types::handle)
- get_value (opae::fpga::types::pvalue)
- get_parent (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#h","title":"h","text":" - handle (opae::fpga::types::handle)
"},{"location":"opae-code/class_member_functions/#i","title":"i","text":" - invalidate (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- is_set (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- invalid_param (opae::fpga::types::invalid_param)
- io_address (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_functions/#l","title":"l","text":" - line (opae::fpga::types::src_location)
"},{"location":"opae-code/class_member_functions/#m","title":"m","text":" - mmio_ptr (opae::fpga::types::handle)
"},{"location":"opae-code/class_member_functions/#n","title":"n","text":" - name (opae::fpga::types::error)
- no_access (opae::fpga::types::no_access)
- no_daemon (opae::fpga::types::no_daemon)
- no_driver (opae::fpga::types::no_driver)
- no_memory (opae::fpga::types::no_memory)
- not_found (opae::fpga::types::not_found)
- not_supported (opae::fpga::types::not_supported)
"},{"location":"opae-code/class_member_functions/#o","title":"o","text":" - operator= (opae::fpga::types::error, opae::fpga::types::guid_t, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::pvalue, opae::fpga::types::shared_buffer, opae::fpga::types::src_location, opae::fpga::types::sysobject)
- operator fpga_event_type (opae::fpga::types::event::type_t)
- operator fpga_event_handle (opae::fpga::types::event)
- os_object (opae::fpga::types::event)
- operator fpga_result (opae::fpga::types::except)
- operator uint8_t * (opae::fpga::types::guid_t)
- operator== (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- open (opae::fpga::types::handle)
- operator fpga_handle (opae::fpga::types::handle)
- operator fpga_properties (opae::fpga::types::properties)
- operator copy_t (opae::fpga::types::pvalue)
- owner (opae::fpga::types::shared_buffer)
- operator fpga_object (opae::fpga::types::sysobject)
- operator fpga_token (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#p","title":"p","text":" - parse (opae::fpga::types::guid_t)
- properties (opae::fpga::types::properties)
- pvalue (opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_functions/#r","title":"r","text":" - read_value (opae::fpga::types::error)
- register_event (opae::fpga::types::event)
- read_csr32 (opae::fpga::types::handle)
- read_csr64 (opae::fpga::types::handle)
- reconfigure (opae::fpga::types::handle)
- reset (opae::fpga::types::handle)
- reconf_error (opae::fpga::types::reconf_error)
- read (opae::fpga::types::shared_buffer)
- release (opae::fpga::types::shared_buffer)
- read64 (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_functions/#s","title":"s","text":" - shared_buffer (opae::fpga::types::shared_buffer)
- size (opae::fpga::types::shared_buffer, opae::fpga::types::sysobject)
- src_location (opae::fpga::types::src_location)
- sysobject (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_functions/#t","title":"t","text":" - type_t (opae::fpga::types::event::type_t)
- type (opae::fpga::types::sysobject)
- token (opae::fpga::types::token)
"},{"location":"opae-code/class_member_functions/#u","title":"u","text":" - update (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_functions/#w","title":"w","text":" - what (opae::fpga::types::except)
- write_csr32 (opae::fpga::types::handle)
- write_csr512 (opae::fpga::types::handle)
- write_csr64 (opae::fpga::types::handle)
- write (opae::fpga::types::shared_buffer)
- wsid (opae::fpga::types::shared_buffer)
- write64 (opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_functions/#_1","title":"~","text":" - ~error (opae::fpga::types::error)
- ~event (opae::fpga::types::event)
- ~handle (opae::fpga::types::handle)
- ~properties (opae::fpga::types::properties)
- ~shared_buffer (opae::fpga::types::shared_buffer)
- ~sysobject (opae::fpga::types::sysobject)
- ~token (opae::fpga::types::token)
"},{"location":"opae-code/class_member_variables/","title":"Class Member Variables","text":""},{"location":"opae-code/class_member_variables/#a","title":"a","text":" - allocated (mem_alloc)
- address (mem_link)
- accelerator_state (opae::fpga::types::properties)
"},{"location":"opae-code/class_member_variables/#b","title":"b","text":" - bus (_fpga_token_header, opae::fpga::types::properties)
- buckets (_opae_hash_map)
- buf_ (opae::fpga::types::except)
- bbs_id (opae::fpga::types::properties)
- bbs_version (opae::fpga::types::properties)
- buffer_iova (opae_vfio_buffer)
- buffer_ptr (opae_vfio_buffer)
- buffer_size (opae_vfio_buffer)
"},{"location":"opae-code/class_member_variables/#c","title":"c","text":" - cleanup_context (_opae_hash_map)
- can_clear (fpga_error_info)
- capabilities (opae::fpga::types::properties)
- copy_ (opae::fpga::types::pvalue)
- cont_buffers (opae_vfio)
- cont_device (opae_vfio)
- cont_fd (opae_vfio)
- cont_pciaddr (opae_vfio)
- cont_ranges (opae_vfio)
- count (opae_vfio_device_irq)
- catastrophicr_error (ras_inject_error)
- csr (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#d","title":"d","text":" - device (_fpga_token_header, opae::fpga::types::properties, opae_vfio)
- device_id (_fpga_token_header, opae::fpga::types::properties)
- data_ (opae::fpga::types::guid_t)
- device_fd (opae_uio, opae_vfio_device)
- device_path (opae_uio)
- device_config_offset (opae_vfio_device)
- device_num_irqs (opae_vfio_device)
- device_num_regions (opae_vfio_device)
"},{"location":"opae-code/class_member_variables/#e","title":"e","text":" - error_info_ (opae::fpga::types::error)
- error_num_ (opae::fpga::types::error)
- event_handle_ (opae::fpga::types::event)
- error (opae::fpga::types::event::type_t)
- event_fds (opae_vfio_device_irq)
- end (opae_vfio_iova_range)
"},{"location":"opae-code/class_member_variables/#f","title":"f","text":" - function (_fpga_token_header, opae::fpga::types::properties)
- flags (_opae_hash_map, opae_vfio_buffer, opae_vfio_device_irq)
- free (mem_alloc)
- file_ (opae::fpga::types::src_location)
- fn_ (opae::fpga::types::src_location)
- fatal_error (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#g","title":"g","text":" - guid (_fpga_token_header, opae::fpga::types::properties)
- group_name (fpga_metric_info)
- get_ (opae::fpga::types::pvalue)
- group (opae_vfio)
- group_device (opae_vfio_group)
- group_fd (opae_vfio_group)
"},{"location":"opae-code/class_member_variables/#h","title":"h","text":" - hash_seed (_opae_hash_map)
- hysteresis (metric_threshold)
- handle_ (opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject)
"},{"location":"opae-code/class_member_variables/#i","title":"i","text":" - interface (_fpga_token_header, opae::fpga::types::properties)
- isvalid (fpga_metric)
- interrupt (opae::fpga::types::event::type_t)
- is_set_ (opae::fpga::types::guid_t, opae::fpga::types::pvalue)
- io_address_ (opae::fpga::types::shared_buffer)
- iova_alloc (opae_vfio)
- irqs (opae_vfio_device)
- index (opae_vfio_device_irq, opae_vfio_sparse_info)
- is_valid (threshold)
"},{"location":"opae-code/class_member_variables/#k","title":"k","text":" - key_cleanup (_opae_hash_map)
- key_compare (_opae_hash_map)
- key_hash (_opae_hash_map)
- key (_opae_hash_map_item)
"},{"location":"opae-code/class_member_variables/#l","title":"l","text":" - lower_c_threshold (metric_threshold)
- lower_nc_threshold (metric_threshold)
- lower_nr_threshold (metric_threshold)
- loc_ (opae::fpga::types::except)
- local_memory_size (opae::fpga::types::properties)
- len_ (opae::fpga::types::shared_buffer)
- line_ (opae::fpga::types::src_location)
- lock (opae_vfio)
"},{"location":"opae-code/class_member_variables/#m","title":"m","text":" - magic (_fpga_token_header)
- metric_num (fpga_metric, fpga_metric_info)
- metric_datatype (fpga_metric_info)
- metric_guid (fpga_metric_info)
- metric_name (fpga_metric_info, metric_threshold)
- metric_type (fpga_metric_info)
- metric_units (fpga_metric_info)
- major (fpga_version)
- minor (fpga_version)
- MAX_EXCEPT (opae::fpga::types::except)
- msg_ (opae::fpga::types::except)
- model (opae::fpga::types::properties)
- masks (opae_vfio_device_irq)
"},{"location":"opae-code/class_member_variables/#n","title":"n","text":" - num_buckets (_opae_hash_map)
- next (_opae_hash_map_item, mem_link, opae_uio_device_region, opae_vfio_device_irq, opae_vfio_device_region, opae_vfio_iova_range, opae_vfio_sparse_info)
- name (fpga_error_info)
- none (opae::fpga::types::properties)
- num_errors (opae::fpga::types::properties)
- num_interrupts (opae::fpga::types::properties)
- num_mmio (opae::fpga::types::properties)
- num_slots (opae::fpga::types::properties)
- nonfatal_error (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#o","title":"o","text":" - object_id (_fpga_token_header, opae::fpga::types::properties)
- objtype (_fpga_token_header)
- open_flags (config)
- os_object_ (opae::fpga::types::event)
- offset (opae_vfio_sparse_info)
"},{"location":"opae-code/class_member_variables/#p","title":"p","text":" - patch (fpga_version)
- prev (mem_link)
- power_thermal (opae::fpga::types::event::type_t)
- props_ (opae::fpga::types::guid_t, opae::fpga::types::properties, opae::fpga::types::pvalue)
- parent (opae::fpga::types::properties)
- ptr (opae_vfio_sparse_info)
"},{"location":"opae-code/class_member_variables/#q","title":"q","text":" - qualifier_name (fpga_metric_info)
"},{"location":"opae-code/class_member_variables/#r","title":"r","text":" - run_n3000 (config)
- res_ (opae::fpga::types::except)
- regions (opae_uio, opae_vfio_device)
- region_index (opae_uio_device_region, opae_vfio_device_region)
- region_page_offset (opae_uio_device_region)
- region_ptr (opae_uio_device_region, opae_vfio_device_region)
- region_size (opae_uio_device_region, opae_vfio_device_region)
- region_sparse (opae_vfio_device_region)
- rsvd (ras_inject_error)
"},{"location":"opae-code/class_member_variables/#s","title":"s","text":" - segment (_fpga_token_header, opae::fpga::types::properties)
- subsystem_device_id (_fpga_token_header, opae::fpga::types::properties)
- subsystem_vendor_id (_fpga_token_header, opae::fpga::types::properties)
- size (mem_link, opae_vfio_sparse_info)
- socket_id (opae::fpga::types::properties)
- set_ (opae::fpga::types::pvalue)
- sysobject_ (opae::fpga::types::sysobject)
- start (opae_vfio_iova_range)
"},{"location":"opae-code/class_member_variables/#t","title":"t","text":" - token_ (opae::fpga::types::error, opae::fpga::types::handle, opae::fpga::types::sysobject, opae::fpga::types::token)
- type_ (opae::fpga::types::event::type_t, opae::fpga::types::event)
- type (opae::fpga::types::properties)
- threshold_name (threshold)
"},{"location":"opae-code/class_member_variables/#u","title":"u","text":" - uint (cache_line)
- upper_c_threshold (metric_threshold)
- upper_nc_threshold (metric_threshold)
- upper_nr_threshold (metric_threshold)
"},{"location":"opae-code/class_member_variables/#v","title":"v","text":" - vendor_id (_fpga_token_header, opae::fpga::types::properties)
- value_cleanup (_opae_hash_map)
- value (_opae_hash_map_item, fpga_metric, threshold)
- virt_ (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_variables/#w","title":"w","text":" - wsid_ (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_variables/#_1","title":"@","text":" - @1 (ras_inject_error)
"},{"location":"opae-code/class_member_typedefs/","title":"Class Member Typedefs","text":""},{"location":"opae-code/class_member_typedefs/#c","title":"c","text":" - copy_t (opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_typedefs/#g","title":"g","text":" - getter_t (opae::fpga::types::pvalue)
"},{"location":"opae-code/class_member_typedefs/#p","title":"p","text":" - ptr_t (opae::fpga::types::error, opae::fpga::types::event, opae::fpga::types::handle, opae::fpga::types::properties, opae::fpga::types::shared_buffer, opae::fpga::types::sysobject, opae::fpga::types::token)
"},{"location":"opae-code/class_member_typedefs/#s","title":"s","text":" - setter_t (opae::fpga::types::pvalue)
- size_t (opae::fpga::types::shared_buffer)
"},{"location":"opae-code/class_member_enums/","title":"Class Member Enums","text":""},{"location":"opae-code/namespace_members/","title":"Namespace Members","text":""},{"location":"opae-code/namespace_members/#a","title":"a","text":" - assert_fpga_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_members/#e","title":"e","text":" - exception_fn (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_members/#i","title":"i","text":" - is_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_members/#o","title":"o","text":" - opae_exceptions (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_functions/","title":"Namespace Member Functions","text":""},{"location":"opae-code/namespace_member_functions/#a","title":"a","text":" - assert_fpga_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_functions/#i","title":"i","text":" - is_ok (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_variables/","title":"Namespace Member Variables","text":""},{"location":"opae-code/namespace_member_variables/#o","title":"o","text":" - opae_exceptions (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_typedefs/","title":"Namespace Member Typedefs","text":""},{"location":"opae-code/namespace_member_typedefs/#e","title":"e","text":" - exception_fn (opae::fpga::types::detail)
"},{"location":"opae-code/namespace_member_enums/","title":"Namespace Member Enums","text":""},{"location":"opae-code/functions/","title":"Functions","text":""},{"location":"opae-code/functions/#e","title":"e","text":" - error_thread (hello_events.c)
"},{"location":"opae-code/functions/#f","title":"f","text":" - fpgaClose (access.h)
- fpgaOpen (access.h)
- fpgaReset (access.h)
- fpgaGetIOAddress (buffer.h)
- fpgaPrepareBuffer (buffer.h)
- fpgaReleaseBuffer (buffer.h)
- fpgaCloneToken (enum.h)
- fpgaDestroyToken (enum.h)
- fpgaEnumerate (enum.h)
- fpgaClearAllErrors (error.h)
- fpgaClearError (error.h)
- fpgaGetErrorInfo (error.h)
- fpgaReadError (error.h)
- fpgaCreateEventHandle (event.h)
- fpgaDestroyEventHandle (event.h)
- fpgaGetOSObjectFromEventHandle (event.h)
- fpgaRegisterEvent (event.h)
- fpgaUnregisterEvent (event.h)
- fpgaFinalize (init.h)
- fpgaInitialize (init.h)
- fpgaAssignPortToInterface (manage.h)
- fpgaAssignToInterface (manage.h)
- fpgaReconfigureSlot (manage.h)
- fpgaReleaseFromInterface (manage.h)
- fpgaGetMetricsByIndex (metrics.h)
- fpgaGetMetricsByName (metrics.h)
- fpgaGetMetricsInfo (metrics.h)
- fpgaGetMetricsThresholdInfo (metrics.h)
- fpgaGetNumMetrics (metrics.h)
- fpgaMapMMIO (mmio.h)
- fpgaReadMMIO32 (mmio.h)
- fpgaReadMMIO64 (mmio.h)
- fpgaUnmapMMIO (mmio.h)
- fpgaWriteMMIO32 (mmio.h)
- fpgaWriteMMIO512 (mmio.h)
- fpgaWriteMMIO64 (mmio.h)
- fpgaClearProperties (properties.h)
- fpgaCloneProperties (properties.h)
- fpgaDestroyProperties (properties.h)
- fpgaGetProperties (properties.h)
- fpgaGetPropertiesFromHandle (properties.h)
- fpgaPropertiesGetAcceleratorState (properties.h)
- fpgaPropertiesGetBBSID (properties.h)
- fpgaPropertiesGetBBSVersion (properties.h)
- fpgaPropertiesGetBus (properties.h)
- fpgaPropertiesGetCapabilities (properties.h)
- fpgaPropertiesGetDevice (properties.h)
- fpgaPropertiesGetDeviceID (properties.h)
- fpgaPropertiesGetFunction (properties.h)
- fpgaPropertiesGetGUID (properties.h)
- fpgaPropertiesGetInterface (properties.h)
- fpgaPropertiesGetLocalMemorySize (properties.h)
- fpgaPropertiesGetModel (properties.h)
- fpgaPropertiesGetNumErrors (properties.h)
- fpgaPropertiesGetNumInterrupts (properties.h)
- fpgaPropertiesGetNumMMIO (properties.h)
- fpgaPropertiesGetNumSlots (properties.h)
- fpgaPropertiesGetObjectID (properties.h)
- fpgaPropertiesGetObjectType (properties.h)
- fpgaPropertiesGetParent (properties.h)
- fpgaPropertiesGetSegment (properties.h)
- fpgaPropertiesGetSocketID (properties.h)
- fpgaPropertiesGetSubsystemDeviceID (properties.h)
- fpgaPropertiesGetSubsystemVendorID (properties.h)
- fpgaPropertiesGetVendorID (properties.h)
- fpgaPropertiesSetAcceleratorState (properties.h)
- fpgaPropertiesSetBBSID (properties.h)
- fpgaPropertiesSetBBSVersion (properties.h)
- fpgaPropertiesSetBus (properties.h)
- fpgaPropertiesSetCapabilities (properties.h)
- fpgaPropertiesSetDevice (properties.h)
- fpgaPropertiesSetDeviceID (properties.h)
- fpgaPropertiesSetFunction (properties.h)
- fpgaPropertiesSetGUID (properties.h)
- fpgaPropertiesSetInterface (properties.h)
- fpgaPropertiesSetLocalMemorySize (properties.h)
- fpgaPropertiesSetModel (properties.h)
- fpgaPropertiesSetNumErrors (properties.h)
- fpgaPropertiesSetNumInterrupts (properties.h)
- fpgaPropertiesSetNumMMIO (properties.h)
- fpgaPropertiesSetNumSlots (properties.h)
- fpgaPropertiesSetObjectID (properties.h)
- fpgaPropertiesSetObjectType (properties.h)
- fpgaPropertiesSetParent (properties.h)
- fpgaPropertiesSetSegment (properties.h)
- fpgaPropertiesSetSocketID (properties.h)
- fpgaPropertiesSetSubsystemDeviceID (properties.h)
- fpgaPropertiesSetSubsystemVendorID (properties.h)
- fpgaPropertiesSetVendorID (properties.h)
- fpgaUpdateProperties (properties.h)
- fpgaDestroyObject (sysobject.h)
- fpgaHandleGetObject (sysobject.h)
- fpgaObjectGetObject (sysobject.h)
- fpgaObjectGetObjectAt (sysobject.h)
- fpgaObjectGetSize (sysobject.h)
- fpgaObjectGetType (sysobject.h)
- fpgaObjectRead (sysobject.h)
- fpgaObjectRead64 (sysobject.h)
- fpgaObjectWrite64 (sysobject.h)
- fpgaTokenGetObject (sysobject.h)
- fpgaGetNumUmsg (umsg.h)
- fpgaGetUmsgPtr (umsg.h)
- fpgaSetUmsgAttributes (umsg.h)
- fpgaTriggerUmsg (umsg.h)
- fpgaGetUserClock (userclk.h)
- fpgaSetUserClock (userclk.h)
- fpgaErrStr (utils.h)
- fpgaGetOPAECBuildString (version.h)
- fpgaGetOPAECVersion (version.h)
- fpgaGetOPAECVersionString (version.h)
- find_fpga (hello_events.c, hello_fpga.c)
- find_nlb_n3000 (hello_fpga.c)
"},{"location":"opae-code/functions/#h","title":"h","text":" - help (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/functions/#i","title":"i","text":" - inject_ras_fatal_error (hello_events.c)
"},{"location":"opae-code/functions/#m","title":"m","text":" - mem_alloc_add_free (mem_alloc.h)
- mem_alloc_destroy (mem_alloc.h)
- mem_alloc_get (mem_alloc.h)
- mem_alloc_init (mem_alloc.h)
- mem_alloc_put (mem_alloc.h)
- main (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/functions/#o","title":"o","text":" - opae_hash_map_add (hash_map.h)
- opae_hash_map_destroy (hash_map.h)
- opae_hash_map_find (hash_map.h)
- opae_hash_map_init (hash_map.h)
- opae_hash_map_is_empty (hash_map.h)
- opae_hash_map_remove (hash_map.h)
- opae_u64_key_compare (hash_map.h)
- opae_u64_key_hash (hash_map.h)
- opae_print (log.h)
- opae_uio_close (uio.h)
- opae_uio_open (uio.h)
- opae_uio_region_get (uio.h)
- opae_vfio_buffer_allocate (vfio.h)
- opae_vfio_buffer_allocate_ex (vfio.h)
- opae_vfio_buffer_free (vfio.h)
- opae_vfio_buffer_info (vfio.h)
- opae_vfio_close (vfio.h)
- opae_vfio_irq_disable (vfio.h)
- opae_vfio_irq_enable (vfio.h)
- opae_vfio_irq_mask (vfio.h)
- opae_vfio_irq_unmask (vfio.h)
- opae_vfio_open (vfio.h)
- opae_vfio_region_get (vfio.h)
- opae_vfio_secure_open (vfio.h)
"},{"location":"opae-code/functions/#p","title":"p","text":" - parse_args (hello_events.c, hello_fpga.c)
- print_err (hello_events.c, hello_fpga.c)
- probe_for_ase (hello_fpga.c)
"},{"location":"opae-code/functions/#u","title":"u","text":" - usleep (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/macros/","title":"Macros","text":""},{"location":"opae-code/macros/#a","title":"a","text":" - ASSERT_FPGA_OK (except.h)
"},{"location":"opae-code/macros/#c","title":"c","text":" - CACHELINE_ALIGNED_ADDR (hello_fpga.c)
- CL (hello_fpga.c)
- CSR_AFU_DSM_BASEL (hello_fpga.c)
- CSR_CFG (hello_fpga.c)
- CSR_CTL (hello_fpga.c)
- CSR_DST_ADDR (hello_fpga.c)
- CSR_NUM_LINES (hello_fpga.c)
- CSR_SRC_ADDR (hello_fpga.c)
- CSR_STATUS1 (hello_fpga.c)
"},{"location":"opae-code/macros/#d","title":"d","text":" - DSM_STATUS_TEST_COMPLETE (hello_fpga.c)
"},{"location":"opae-code/macros/#f","title":"f","text":" - FPGA_ERROR_NAME_MAX (types.h)
- FPGA_METRIC_STR_SIZE (types.h)
- fpga_is_parent_child (types.h)
- FPGA_BUILD_STR_MAX (version.h)
- FPGA_VERSION_STR_MAX (version.h)
- FME_SYSFS_INJECT_ERROR (hello_events.c)
- FPGA_NLB0_UUID_H (hello_fpga.c)
- FPGA_NLB0_UUID_L (hello_fpga.c)
"},{"location":"opae-code/macros/#g","title":"g","text":" - GETOPT_STRING (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/macros/#l","title":"l","text":" - LOG2_CL (hello_fpga.c)
- LPBK1_BUFFER_ALLOCATION_SIZE (hello_fpga.c)
- LPBK1_BUFFER_SIZE (hello_fpga.c)
- LPBK1_DSM_SIZE (hello_fpga.c)
"},{"location":"opae-code/macros/#m","title":"m","text":" - MB (hello_fpga.c)
"},{"location":"opae-code/macros/#n","title":"n","text":" - N3000_AFUID (hello_fpga.c)
- NLB0_AFUID (hello_fpga.c)
"},{"location":"opae-code/macros/#o","title":"o","text":" - OPAECXX_HERE (except.h)
- OPAE_DBG (log.h)
- OPAE_DEFAULT_LOGLEVEL (log.h)
- OPAE_ERR (log.h)
- OPAE_MSG (log.h)
- OPAE_UIO_PATH_MAX (uio.h)
- ON_ERR_GOTO (hello_events.c, hello_fpga.c)
"},{"location":"opae-code/macros/#t","title":"t","text":" - TEST_TIMEOUT (hello_fpga.c)
"},{"location":"opae-code/macros/#_","title":"_","text":" - __SHORT_FILE__ (log.h)
"},{"location":"opae-code/variables/","title":"Variables","text":""},{"location":"opae-code/variables/#c","title":"c","text":" - config (hello_fpga.c)
"},{"location":"opae-code/variables/#f","title":"f","text":" - fpga_event_handle (types.h)
- fpga_guid (types.h)
- fpga_handle (types.h)
- fpga_metric (types.h)
- fpga_metric_info (types.h)
- fpga_object (types.h)
- fpga_properties (types.h)
- fpga_token (types.h)
- fpga_token_header (types.h)
- fpga_accelerator_state (types_enum.h)
- fpga_buffer_flags (types_enum.h)
- fpga_event_type (types_enum.h)
- fpga_interface (types_enum.h)
- fpga_metric_datatype (types_enum.h)
- fpga_metric_type (types_enum.h)
- fpga_objtype (types_enum.h)
- fpga_open_flags (types_enum.h)
- fpga_reconf_flags (types_enum.h)
- fpga_result (types_enum.h)
- fpga_sysobject_flags (types_enum.h)
- fpga_sysobject_type (types_enum.h)
"},{"location":"opae-code/variables/#m","title":"m","text":" - metric_threshold (types.h)
"},{"location":"opae-code/variables/#o","title":"o","text":" - opae_hash_map (hash_map.h)
- opae_hash_map_flags (hash_map.h)
- opae_hash_map_item (hash_map.h)
- opae_loglevel (log.h)
- opae_vfio_buffer_flags (vfio.h)
"},{"location":"opae-code/variables/#t","title":"t","text":" - threshold (types.h)
"},{"location":"opae-code/variables/#_","title":"_","text":" - _opae_hash_map_flags (hash_map.h)
"},{"location":"opae-code/links/","title":"Links","text":"* Related Pages * Todo List * Modules * Class List * struct _fpga_token_header * struct _opae_hash_map * struct _opae_hash_map_item * struct cache_line * struct config * struct fpga_error_info * struct fpga_metric * struct fpga_metric_info * struct fpga_version * struct mem_alloc * struct mem_link * struct metric_threshold * union metric_value * namespace opae * namespace opae::fpga * namespace opae::fpga::types * class opae::fpga::types::busy * namespace opae::fpga::types::detail * class opae::fpga::types::error * class opae::fpga::types::event * struct opae::fpga::types::event::type_t * class opae::fpga::types::except * class opae::fpga::types::exception * struct opae::fpga::types::guid_t * class opae::fpga::types::handle * class opae::fpga::types::invalid_param * class opae::fpga::types::no_access * class opae::fpga::types::no_daemon * class opae::fpga::types::no_driver * class opae::fpga::types::no_memory * class opae::fpga::types::not_found * class opae::fpga::types::not_supported * class opae::fpga::types::properties * struct opae::fpga::types::pvalue * class opae::fpga::types::reconf_error * class opae::fpga::types::shared_buffer * class opae::fpga::types::src_location * class opae::fpga::types::sysobject * class opae::fpga::types::token * class opae::fpga::types::version * struct opae_uio * struct opae_uio_device_region * struct opae_vfio * struct opae_vfio_buffer * struct opae_vfio_device * struct opae_vfio_device_irq * struct opae_vfio_device_region * struct opae_vfio_group * struct opae_vfio_iova_range * struct opae_vfio_sparse_info * struct ras_inject_error * namespace std * struct threshold * Namespace ListNamespace List * Namespace Members * Namespace Member Functions * Namespace Member Variables * Namespace Member Typedefs * Namespace Member Enumerations * Class Index * Class Hierarchy * Class Members * Class Member Functions * Class Member Variables * Class Member Typedefs * Class Member Enumerations * Files * docs * docs/sw * docs/sw/include * docs/sw/include/opae * access.h * access.h source * buffer.h * buffer.h source * docs/sw/include/opae/cxx * core.h * core.h source * docs/sw/include/opae/cxx/core * errors.h * errors.h source * events.h * events.h source * except.h * except.h source * handle.h * handle.h source * properties.h * properties.h source * pvalue.h * pvalue.h source * shared_buffer.h * shared_buffer.h source * sysobject.h * sysobject.h source * token.h * token.h source * version.h * version.h source * enum.h * enum.h source * error.h * error.h source * event.h * event.h source * fpga.h * fpga.h source * hash_map.h * hash_map.h source * init.h * init.h source * log.h * log.h source * manage.h * manage.h source * mem_alloc.h * mem_alloc.h source * metrics.h * metrics.h source * mmio.h * mmio.h source * properties.h * properties.h source * sysobject.h * sysobject.h source * types.h * types.h source * types_enum.h * types_enum.h source * uio.h * uio.h source * umsg.h * umsg.h source * userclk.h * userclk.h source * utils.h * utils.h source * version.h * version.h source * vfio.h * vfio.h source * docs/sw/samples * docs/sw/samples/hello_events * hello_events.c * hello_events.c source * docs/sw/samples/hello_fpga * hello_fpga.c * hello_fpga.c source * File Variables * File Functions * File Macros
"}]}
\ No newline at end of file
diff --git a/ofs-2024.2-1/sitemap.xml.gz b/ofs-2024.2-1/sitemap.xml.gz
index 1d572097..75ccd2b7 100644
Binary files a/ofs-2024.2-1/sitemap.xml.gz and b/ofs-2024.2-1/sitemap.xml.gz differ